You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by cr...@apache.org on 2018/01/03 17:48:01 UTC
[01/35] incubator-airflow-site git commit: 1.9.0
Repository: incubator-airflow-site
Updated Branches:
refs/heads/asf-site 69cff4922 -> 28a3eb600
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/security.html
----------------------------------------------------------------------
diff --git a/security.html b/security.html
index d0156b7..9bde2ce 100644
--- a/security.html
+++ b/security.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -104,7 +109,11 @@
<li class="toctree-l2"><a class="reference internal" href="#multi-tenancy">Multi-tenancy</a></li>
<li class="toctree-l2"><a class="reference internal" href="#kerberos">Kerberos</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#limitations">Limitations</a></li>
-<li class="toctree-l3"><a class="reference internal" href="#enabling-kerberos">Enabling kerberos</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#enabling-kerberos">Enabling kerberos</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#airflow">Airflow</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#hadoop">Hadoop</a></li>
+</ul>
+</li>
<li class="toctree-l3"><a class="reference internal" href="#using-kerberos-authentication">Using kerberos authentication</a></li>
</ul>
</li>
@@ -119,8 +128,9 @@
</li>
</ul>
</li>
-<li class="toctree-l2"><a class="reference internal" href="#ssl">SSL</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="#impersonation">Impersonation</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#ssl">SSL</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#impersonation">Impersonation</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#default-impersonation">Default Impersonation</a></li>
</ul>
</li>
</ul>
@@ -198,7 +208,8 @@
to the web application is to do it at the network level, or by using
SSH tunnels.</p>
<p>It is however possible to switch on authentication by either using one of the supplied
-backends or create your own.</p>
+backends or creating your own.</p>
+<p>Be sure to checkout <a class="reference internal" href="api.html"><span class="doc">Experimental Rest API</span></a> for securing the API.</p>
<div class="section" id="web-authentication">
<h2>Web Authentication<a class="headerlink" href="#web-authentication" title="Permalink to this headline">¶</a></h2>
<div class="section" id="password">
@@ -217,7 +228,7 @@ attack. Creating a new user has to be done via a Python REPL on the same machine
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># navigate to the airflow installation directory</span>
$ <span class="nb">cd</span> ~/airflow
$ python
-Python 2.7.9 <span class="o">(</span>default, Feb <span class="m">10</span> 2015, 03:28:08<span class="o">)</span>
+Python <span class="m">2</span>.7.9 <span class="o">(</span>default, Feb <span class="m">10</span> <span class="m">2015</span>, <span class="m">03</span>:28:08<span class="o">)</span>
Type <span class="s2">"help"</span>, <span class="s2">"copyright"</span>, <span class="s2">"credits"</span> or <span class="s2">"license"</span> <span class="k">for</span> more information.
>>> import airflow
>>> from airflow import models, settings
@@ -240,7 +251,7 @@ Type <span class="s2">"help"</span>, <span class="s2">"copyright&
an encrypted connection to the ldap server as you probably do not want passwords be readable on the network level.
It is however possible to configure without encryption if you really want to.</p>
<p>Additionally, if you are using Active Directory, and are not explicitly specifying an OU that your users are in,
-you will need to change <code class="docutils literal"><span class="pre">search_scope</span></code> to “SUBTREE”.</p>
+you will need to change <code class="docutils literal"><span class="pre">search_scope</span></code> to “SUBTREE”.</p>
<p>Valid search_scope options can be found in the <a class="reference external" href="http://ldap3.readthedocs.org/searches.html?highlight=search_scope">ldap3 Documentation</a></p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
@@ -252,6 +263,11 @@ you will need to change <code class="docutils literal"><span class="pre">search_
<span class="nv">user_filter</span> <span class="o">=</span> <span class="nv">objectClass</span><span class="o">=</span>*
<span class="c1"># in case of Active Directory you would use: user_name_attr = sAMAccountName</span>
<span class="nv">user_name_attr</span> <span class="o">=</span> uid
+<span class="c1"># group_member_attr should be set accordingly with *_filter</span>
+<span class="c1"># eg :</span>
+<span class="c1"># group_member_attr = groupMembership</span>
+<span class="c1"># superuser_filter = groupMembership=CN=airflow-super-users...</span>
+<span class="nv">group_member_attr</span> <span class="o">=</span> memberOf
<span class="nv">superuser_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-super-users,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
<span class="nv">data_profiler_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-data-profilers,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
<span class="nv">bind_user</span> <span class="o">=</span> <span class="nv">cn</span><span class="o">=</span>Manager,dc<span class="o">=</span>example,dc<span class="o">=</span>com
@@ -269,7 +285,7 @@ you will need to change <code class="docutils literal"><span class="pre">search_
<h3>Roll your own<a class="headerlink" href="#roll-your-own" title="Permalink to this headline">¶</a></h3>
<p>Airflow uses <code class="docutils literal"><span class="pre">flask_login</span></code> and
exposes a set of hooks in the <code class="docutils literal"><span class="pre">airflow.default_login</span></code> module. You can
-alter the content and make it part of the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code> and configure it as a backend in <code class="docutils literal"><span class="pre">airflow.cfg`</span></code>.</p>
+alter the content and make it part of the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code> and configure it as a backend in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> mypackage.auth
@@ -279,12 +295,13 @@ alter the content and make it part of the <code class="docutils literal"><span c
</div>
<div class="section" id="multi-tenancy">
<h2>Multi-tenancy<a class="headerlink" href="#multi-tenancy" title="Permalink to this headline">¶</a></h2>
-<p>You can filter the list of dags in webserver by owner name, when authentication
-is turned on, by setting webserver.filter_by_owner as true in your <code class="docutils literal"><span class="pre">airflow.cfg</span></code>
-With this, when a user authenticates and logs into webserver, it will see only the dags
-which it is owner of. A super_user, will be able to see all the dags although.
-This makes the web UI a multi-tenant UI, where a user will only be able to see dags
-created by itself.</p>
+<p>You can filter the list of dags in webserver by owner name when authentication
+is turned on by setting <code class="docutils literal"><span class="pre">webserver:filter_by_owner</span></code> in your config. With this, a user will see
+only the dags which it is owner of, unless it is a superuser.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">filter_by_owner</span> <span class="o">=</span> True
+</pre></div>
+</div>
</div>
<div class="section" id="kerberos">
<h2>Kerberos<a class="headerlink" href="#kerberos" title="Permalink to this headline">¶</a></h2>
@@ -293,15 +310,16 @@ tickets for itself and store it in the ticket cache. The hooks and dags can make
to authenticate against kerberized services.</p>
<div class="section" id="limitations">
<h3>Limitations<a class="headerlink" href="#limitations" title="Permalink to this headline">¶</a></h3>
-<p>Please note that at this time not all hooks have been adjusted to make use of this functionality yet.
+<p>Please note that at this time, not all hooks have been adjusted to make use of this functionality.
Also it does not integrate kerberos into the web interface and you will have to rely on network
level security for now to make sure your service remains secure.</p>
-<p>Celery integration has not been tried and tested yet. However if you generate a key tab for every host
-and launch a ticket renewer next to every worker it will most likely work.</p>
+<p>Celery integration has not been tried and tested yet. However, if you generate a key tab for every
+host and launch a ticket renewer next to every worker it will most likely work.</p>
</div>
<div class="section" id="enabling-kerberos">
<h3>Enabling kerberos<a class="headerlink" href="#enabling-kerberos" title="Permalink to this headline">¶</a></h3>
-<p>#### Airflow</p>
+<div class="section" id="airflow">
+<h4>Airflow<a class="headerlink" href="#airflow" title="Permalink to this headline">¶</a></h4>
<p>To enable kerberos you will need to generate a (service) key tab.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># in the kadmin.local or kadmin shell, create the airflow principal</span>
kadmin: addprinc -randkey airflow/fully.qualified.domain.name@YOUR-REALM.COM
@@ -317,7 +335,7 @@ your <code class="docutils literal"><span class="pre">airflow.cfg</span></code><
<span class="o">[</span>kerberos<span class="o">]</span>
<span class="nv">keytab</span> <span class="o">=</span> /etc/airflow/airflow.keytab
-<span class="nv">reinit_frequency</span> <span class="o">=</span> 3600
+<span class="nv">reinit_frequency</span> <span class="o">=</span> <span class="m">3600</span>
<span class="nv">principal</span> <span class="o">=</span> airflow
</pre></div>
</div>
@@ -326,7 +344,9 @@ your <code class="docutils literal"><span class="pre">airflow.cfg</span></code><
airflow kerberos
</pre></div>
</div>
-<p>#### Hadoop</p>
+</div>
+<div class="section" id="hadoop">
+<h4>Hadoop<a class="headerlink" href="#hadoop" title="Permalink to this headline">¶</a></h4>
<p>If want to use impersonation this needs to be enabled in <code class="docutils literal"><span class="pre">core-site.xml</span></code> of your hadoop config.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><property>
<name>hadoop.proxyuser.airflow.groups</name>
@@ -346,17 +366,18 @@ airflow kerberos
</div>
<p>Of course if you need to tighten your security replace the asterisk with something more appropriate.</p>
</div>
+</div>
<div class="section" id="using-kerberos-authentication">
<h3>Using kerberos authentication<a class="headerlink" href="#using-kerberos-authentication" title="Permalink to this headline">¶</a></h3>
-<p>The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to use it simply
-update the connection details with, for example:</p>
+<p>The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to
+use it, simply update the connection details with, for example:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span><span class="o">}</span>
</pre></div>
</div>
<p>Adjust the principal to your settings. The _HOST part will be replaced by the fully qualified domain name of
the server.</p>
<p>You can specify if you would like to use the dag owner as the user for the connection or the user specified in the login
-section of the connection. For the login user specify the following as extra:</p>
+section of the connection. For the login user, specify the following as extra:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span>, <span class="s2">"proxy_user"</span>: <span class="s2">"login"</span><span class="o">}</span>
</pre></div>
</div>
@@ -364,7 +385,7 @@ section of the connection. For the login user specify the following as extra:</p
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span>, <span class="s2">"proxy_user"</span>: <span class="s2">"owner"</span><span class="o">}</span>
</pre></div>
</div>
-<p>and in your DAG, when initializing the HiveOperator, specify</p>
+<p>and in your DAG, when initializing the HiveOperator, specify:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">run_as_owner</span><span class="o">=</span>True
</pre></div>
</div>
@@ -378,8 +399,6 @@ section of the connection. For the login user specify the following as extra:</p
against an installation of GitHub Enterprise using OAuth2. You can optionally
specify a team whitelist (composed of slug cased team names) to restrict login
to only members of those teams.</p>
-<p><em>NOTE</em> If you do not specify a team whitelist, anyone with a valid account on
-your GHE installation will be able to login to Airflow.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">authenticate</span> <span class="o">=</span> True
<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.github_enterprise_auth
@@ -389,21 +408,26 @@ your GHE installation will be able to login to Airflow.</p>
<span class="nv">client_id</span> <span class="o">=</span> oauth_key_from_github_enterprise
<span class="nv">client_secret</span> <span class="o">=</span> oauth_secret_from_github_enterprise
<span class="nv">oauth_callback_route</span> <span class="o">=</span> /example/ghe_oauth/callback
-<span class="nv">allowed_teams</span> <span class="o">=</span> 1, 345, 23
+<span class="nv">allowed_teams</span> <span class="o">=</span> <span class="m">1</span>, <span class="m">345</span>, <span class="m">23</span>
</pre></div>
</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">If you do not specify a team whitelist, anyone with a valid account on
+your GHE installation will be able to login to Airflow.</p>
+</div>
<div class="section" id="setting-up-ghe-authentication">
<h4>Setting up GHE Authentication<a class="headerlink" href="#setting-up-ghe-authentication" title="Permalink to this headline">¶</a></h4>
<p>An application must be setup in GHE before you can use the GHE authentication
backend. In order to setup an application:</p>
<ol class="arabic simple">
<li>Navigate to your GHE profile</li>
-<li>Select ‘Applications’ from the left hand nav</li>
-<li>Select the ‘Developer Applications’ tab</li>
-<li>Click ‘Register new application’</li>
-<li>Fill in the required information (the ‘Authorization callback URL’ must be fully qualifed e.g. <a class="reference external" href="http://airflow.example.com/example/ghe_oauth/callback">http://airflow.example.com/example/ghe_oauth/callback</a>)</li>
-<li>Click ‘Register application’</li>
-<li>Copy ‘Client ID’, ‘Client Secret’, and your callback route to your airflow.cfg according to the above example</li>
+<li>Select ‘Applications’ from the left hand nav</li>
+<li>Select the ‘Developer Applications’ tab</li>
+<li>Click ‘Register new application’</li>
+<li>Fill in the required information (the ‘Authorization callback URL’ must be fully qualifed e.g. <a class="reference external" href="http://airflow.example.com/example/ghe_oauth/callback">http://airflow.example.com/example/ghe_oauth/callback</a>)</li>
+<li>Click ‘Register application’</li>
+<li>Copy ‘Client ID’, ‘Client Secret’, and your callback route to your airflow.cfg according to the above example</li>
</ol>
</div>
</div>
@@ -429,47 +453,67 @@ to only members of that domain.</p>
backend. In order to setup an application:</p>
<ol class="arabic simple">
<li>Navigate to <a class="reference external" href="https://console.developers.google.com/apis/">https://console.developers.google.com/apis/</a></li>
-<li>Select ‘Credentials’ from the left hand nav</li>
-<li>Click ‘Create credentials’ and choose ‘OAuth client ID’</li>
-<li>Choose ‘Web application’</li>
-<li>Fill in the required information (the ‘Authorized redirect URIs’ must be fully qualifed e.g. <a class="reference external" href="http://airflow.example.com/oauth2callback">http://airflow.example.com/oauth2callback</a>)</li>
-<li>Click ‘Create’</li>
-<li>Copy ‘Client ID’, ‘Client Secret’, and your redirect URI to your airflow.cfg according to the above example</li>
</ol>
+<p>2. Select ‘Credentials’ from the left hand nav
+2. Select ‘Credentials’ from the left hand nav
+3. Click ‘Create credentials’ and choose ‘OAuth client ID’
+4. Choose ‘Web application’
+5. Fill in the required information (the ‘Authorized redirect URIs’ must be fully qualifed e.g. <a class="reference external" href="http://airflow.example.com/oauth2callback">http://airflow.example.com/oauth2callback</a>)
+6. Click ‘Create’
+7. Copy ‘Client ID’, ‘Client Secret’, and your redirect URI to your airflow.cfg according to the above example</p>
</div>
</div>
</div>
<div class="section" id="ssl">
<h2>SSL<a class="headerlink" href="#ssl" title="Permalink to this headline">¶</a></h2>
<p>SSL can be enabled by providing a certificate and key. Once enabled, be sure to use
-“<a class="reference external" href="https://">https://</a>” in your browser.</p>
+“<a class="reference external" href="https://">https://</a>” in your browser.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
<span class="nv">web_server_ssl_cert</span> <span class="o">=</span> <path to cert>
<span class="nv">web_server_ssl_key</span> <span class="o">=</span> <path to key>
</pre></div>
</div>
<p>Enabling SSL will not automatically change the web server port. If you want to use the
-standard port 443, you’ll need to configure that too. Be aware that super user privileges
+standard port 443, you’ll need to configure that too. Be aware that super user privileges
(or cap_net_bind_service on Linux) are required to listen on port 443.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Optionally, set the server to listen on the standard SSL port.</span>
-<span class="nv">web_server_port</span> <span class="o">=</span> 443
+<span class="nv">web_server_port</span> <span class="o">=</span> <span class="m">443</span>
<span class="nv">base_url</span> <span class="o">=</span> http://<hostname or IP>:443
</pre></div>
</div>
+<p>Enable CeleryExecutor with SSL. Ensure you properly generate client and server
+certs and keys.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>celery<span class="o">]</span>
+<span class="nv">CELERY_SSL_ACTIVE</span> <span class="o">=</span> True
+<span class="nv">CELERY_SSL_KEY</span> <span class="o">=</span> <path to key>
+<span class="nv">CELERY_SSL_CERT</span> <span class="o">=</span> <path to cert>
+<span class="nv">CELERY_SSL_CACERT</span> <span class="o">=</span> <path to cacert>
+</pre></div>
+</div>
+</div>
<div class="section" id="impersonation">
-<h3>Impersonation<a class="headerlink" href="#impersonation" title="Permalink to this headline">¶</a></h3>
+<h2>Impersonation<a class="headerlink" href="#impersonation" title="Permalink to this headline">¶</a></h2>
<p>Airflow has the ability to impersonate a unix user while running task
-instances based on the task’s <code class="docutils literal"><span class="pre">run_as_user</span></code> parameter, which takes a user’s name.</p>
-<p><em>NOTE</em> For impersonations to work, Airflow must be run with <cite>sudo</cite> as subtasks are run
+instances based on the task’s <code class="docutils literal"><span class="pre">run_as_user</span></code> parameter, which takes a user’s name.</p>
+<p><strong>NOTE:</strong> For impersonations to work, Airflow must be run with <cite>sudo</cite> as subtasks are run
with <cite>sudo -u</cite> and permissions of files are changed. Furthermore, the unix user needs to
exist on the worker. Here is what a simple sudoers file entry could look like to achieve
this, assuming as airflow is running as the <cite>airflow</cite> user. Note that this means that
the airflow user must be trusted and treated the same way as the root user.</p>
+<div class="highlight-none"><div class="highlight"><pre><span></span>airflow ALL=(ALL) NOPASSWD: ALL
+</pre></div>
+</div>
<p>Subtasks with impersonation will still log to the same folder, except that the files they
log to will have permissions changed such that only the unix user can write to it.</p>
-<p><em>Default impersonation</em> To prevent tasks that don’t use impersonation to be run with
-<cite>sudo</cite> privileges, you can set the <cite>default_impersonation</cite> config in <cite>core</cite> which sets a
-default user impersonate if <cite>run_as_user</cite> is not set.</p>
+<div class="section" id="default-impersonation">
+<h3>Default Impersonation<a class="headerlink" href="#default-impersonation" title="Permalink to this headline">¶</a></h3>
+<p>To prevent tasks that don’t use impersonation to be run with <cite>sudo</cite> privileges, you can set the
+<code class="docutils literal"><span class="pre">core:default_impersonation</span></code> config which sets a default user impersonate if <cite>run_as_user</cite> is
+not set.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
+<span class="nv">default_impersonation</span> <span class="o">=</span> airflow
+</pre></div>
+</div>
</div>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/start.html
----------------------------------------------------------------------
diff --git a/start.html b/start.html
index 511a95b..d1aa874 100644
--- a/start.html
+++ b/start.html
@@ -13,6 +13,8 @@
+
+
@@ -81,11 +83,14 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Quick Start</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="#what-s-next">What’s Next?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#what-s-next">What’s Next?</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
@@ -174,17 +179,17 @@
<span class="nb">export</span> <span class="nv">AIRFLOW_HOME</span><span class="o">=</span>~/airflow
<span class="c1"># install from pypi using pip</span>
-pip install airflow
+pip install apache-airflow
<span class="c1"># initialize the database</span>
airflow initdb
<span class="c1"># start the web server, default port is 8080</span>
-airflow webserver -p 8080
+airflow webserver -p <span class="m">8080</span>
</pre></div>
</div>
<p>Upon running these commands, Airflow will create the <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> folder
-and lay an “airflow.cfg” file with defaults that get you going fast. You can
+and lay an “airflow.cfg” file with defaults that get you going fast. You can
inspect the file either in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>, or through the UI in
the <code class="docutils literal"><span class="pre">Admin->Configuration</span></code> menu. The PID file for the webserver will be stored
in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow-webserver.pid</span></code> or in <code class="docutils literal"><span class="pre">/run/airflow/webserver.pid</span></code>
@@ -199,14 +204,14 @@ command line utilities.</p>
be able to see the status of the jobs change in the <code class="docutils literal"><span class="pre">example1</span></code> DAG as you
run the commands below.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># run your first task instance</span>
-airflow run example_bash_operator runme_0 2015-01-01
+airflow run example_bash_operator runme_0 <span class="m">2015</span>-01-01
<span class="c1"># run a backfill over 2 days</span>
-airflow backfill example_bash_operator -s 2015-01-01 -e 2015-01-02
+airflow backfill example_bash_operator -s <span class="m">2015</span>-01-01 -e <span class="m">2015</span>-01-02
</pre></div>
</div>
<div class="section" id="what-s-next">
-<h2>What’s Next?<a class="headerlink" href="#what-s-next" title="Permalink to this headline">¶</a></h2>
-<p>From this point, you can head to the <a class="reference internal" href="tutorial.html"><span class="doc">Tutorial</span></a> section for further examples or the <a class="reference internal" href="configuration.html"><span class="doc">Configuration</span></a> section if you’re ready to get your hands dirty.</p>
+<h2>What’s Next?<a class="headerlink" href="#what-s-next" title="Permalink to this headline">¶</a></h2>
+<p>From this point, you can head to the <a class="reference internal" href="tutorial.html"><span class="doc">Tutorial</span></a> section for further examples or the <a class="reference internal" href="configuration.html"><span class="doc">Configuration</span></a> section if you’re ready to get your hands dirty.</p>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/tutorial.html
----------------------------------------------------------------------
diff --git a/tutorial.html b/tutorial.html
index 2a55053..ea3af9b 100644
--- a/tutorial.html
+++ b/tutorial.html
@@ -13,6 +13,8 @@
+
+
@@ -81,14 +83,17 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Tutorial</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#example-pipeline-definition">Example Pipeline definition</a></li>
-<li class="toctree-l2"><a class="reference internal" href="#it-s-a-dag-definition-file">It’s a DAG definition file</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#it-s-a-dag-definition-file">It’s a DAG definition file</a></li>
<li class="toctree-l2"><a class="reference internal" href="#importing-modules">Importing Modules</a></li>
<li class="toctree-l2"><a class="reference internal" href="#default-arguments">Default Arguments</a></li>
<li class="toctree-l2"><a class="reference internal" href="#instantiate-a-dag">Instantiate a DAG</a></li>
@@ -103,7 +108,7 @@
<li class="toctree-l3"><a class="reference internal" href="#backfill">Backfill</a></li>
</ul>
</li>
-<li class="toctree-l2"><a class="reference internal" href="#what-s-next">What’s Next?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#what-s-next">What’s Next?</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
@@ -202,7 +207,7 @@ complicated, a line by line explanation follows below.</p>
<span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
<span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
- <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@airflow.com'</span><span class="p">],</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@example.com'</span><span class="p">],</span>
<span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
@@ -247,10 +252,10 @@ complicated, a line by line explanation follows below.</p>
</div>
</div>
<div class="section" id="it-s-a-dag-definition-file">
-<h2>It’s a DAG definition file<a class="headerlink" href="#it-s-a-dag-definition-file" title="Permalink to this headline">¶</a></h2>
+<h2>It’s a DAG definition file<a class="headerlink" href="#it-s-a-dag-definition-file" title="Permalink to this headline">¶</a></h2>
<p>One thing to wrap your head around (it may not be very intuitive for everyone
at first) is that this Airflow Python script is really
-just a configuration file specifying the DAG’s structure as code.
+just a configuration file specifying the DAG’s structure as code.
The actual tasks defined here will run in a different context from
the context of this script. Different tasks run on different workers
at different points in time, which means that this script cannot be used
@@ -258,14 +263,14 @@ to cross communicate between tasks. Note that for this
purpose we have a more advanced feature called <code class="docutils literal"><span class="pre">XCom</span></code>.</p>
<p>People sometimes think of the DAG definition file as a place where they
can do some actual data processing - that is not the case at all!
-The script’s purpose is to define a DAG object. It needs to evaluate
+The script’s purpose is to define a DAG object. It needs to evaluate
quickly (seconds, not minutes) since the scheduler will execute it
periodically to reflect the changes if any.</p>
</div>
<div class="section" id="importing-modules">
<h2>Importing Modules<a class="headerlink" href="#importing-modules" title="Permalink to this headline">¶</a></h2>
<p>An Airflow pipeline is just a Python script that happens to define an
-Airflow DAG object. Let’s start by importing the libraries we will need.</p>
+Airflow DAG object. Let’s start by importing the libraries we will need.</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># The DAG object; we'll need this to instantiate a DAG</span>
<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">DAG</span>
@@ -276,8 +281,8 @@ Airflow DAG object. Let’s start by importing the libraries we will need.</
</div>
<div class="section" id="default-arguments">
<h2>Default Arguments<a class="headerlink" href="#default-arguments" title="Permalink to this headline">¶</a></h2>
-<p>We’re about to create a DAG and some tasks, and we have the choice to
-explicitly pass a set of arguments to each task’s constructor
+<p>We’re about to create a DAG and some tasks, and we have the choice to
+explicitly pass a set of arguments to each task’s constructor
(which would become redundant), or (better!) we can define a dictionary
of default parameters that we can use when creating tasks.</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
@@ -286,7 +291,7 @@ of default parameters that we can use when creating tasks.</p>
<span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
<span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
- <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@airflow.com'</span><span class="p">],</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@example.com'</span><span class="p">],</span>
<span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
@@ -298,7 +303,7 @@ of default parameters that we can use when creating tasks.</p>
<span class="p">}</span>
</pre></div>
</div>
-<p>For more information about the BaseOperator’s parameters and what they do,
+<p>For more information about the BaseOperator’s parameters and what they do,
refer to the :py:class:<code class="docutils literal"><span class="pre">airflow.models.BaseOperator</span></code> documentation.</p>
<p>Also, note that you could easily define different sets of arguments that
would serve different purposes. An example of that would be to have
@@ -306,7 +311,7 @@ different settings between a production and development environment.</p>
</div>
<div class="section" id="instantiate-a-dag">
<h2>Instantiate a DAG<a class="headerlink" href="#instantiate-a-dag" title="Permalink to this headline">¶</a></h2>
-<p>We’ll need a DAG object to nest our tasks into. Here we pass a string
+<p>We’ll need a DAG object to nest our tasks into. Here we pass a string
that defines the <code class="docutils literal"><span class="pre">dag_id</span></code>, which serves as a unique identifier for your DAG.
We also pass the default argument dictionary that we just defined and
define a <code class="docutils literal"><span class="pre">schedule_interval</span></code> of 1 day for the DAG.</p>
@@ -334,14 +339,14 @@ instantiated from an operator is called a constructor. The first argument
</div>
<p>Notice how we pass a mix of operator specific arguments (<code class="docutils literal"><span class="pre">bash_command</span></code>) and
an argument common to all operators (<code class="docutils literal"><span class="pre">retries</span></code>) inherited
-from BaseOperator to the operator’s constructor. This is simpler than
+from BaseOperator to the operator’s constructor. This is simpler than
passing every argument for every constructor call. Also, notice that in
the second task we override the <code class="docutils literal"><span class="pre">retries</span></code> parameter with <code class="docutils literal"><span class="pre">3</span></code>.</p>
<p>The precedence rules for a task are as follows:</p>
<ol class="arabic simple">
<li>Explicitly passed arguments</li>
<li>Values that exist in the <code class="docutils literal"><span class="pre">default_args</span></code> dictionary</li>
-<li>The operator’s default value, if one exists</li>
+<li>The operator’s default value, if one exists</li>
</ol>
<p>A task must include or inherit the arguments <code class="docutils literal"><span class="pre">task_id</span></code> and <code class="docutils literal"><span class="pre">owner</span></code>,
otherwise Airflow will raise an exception.</p>
@@ -357,7 +362,8 @@ templates.</p>
<p>This tutorial barely scratches the surface of what you can do with
templating in Airflow, but the goal of this section is to let you know
this feature exists, get you familiar with double curly brackets, and
-point to the most common template variable: <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code>.</p>
+point to the most common template variable: <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> (today’s “date
+stamp”).</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">templated_command</span> <span class="o">=</span> <span class="s2">"""</span>
<span class="s2"> {</span><span class="si">% f</span><span class="s2">or i in range(5) %}</span>
<span class="s2"> echo "{{ ds }}"</span>
@@ -383,17 +389,26 @@ to understand how the parameter <code class="docutils literal"><span class="pre"
<p>Files can also be passed to the <code class="docutils literal"><span class="pre">bash_command</span></code> argument, like
<code class="docutils literal"><span class="pre">bash_command='templated_command.sh'</span></code>, where the file location is relative to
the directory containing the pipeline file (<code class="docutils literal"><span class="pre">tutorial.py</span></code> in this case). This
-may be desirable for many reasons, like separating your script’s logic and
+may be desirable for many reasons, like separating your script’s logic and
pipeline code, allowing for proper code highlighting in files composed in
different languages, and general flexibility in structuring pipelines. It is
also possible to define your <code class="docutils literal"><span class="pre">template_searchpath</span></code> as pointing to any folder
locations in the DAG constructor call.</p>
+<p>Using that same DAG constructor call, it is possible to define
+<code class="docutils literal"><span class="pre">user_defined_macros</span></code> which allow you to specify your own variables.
+For example, passing <code class="docutils literal"><span class="pre">dict(foo='bar')</span></code> to this argument allows you
+to use <code class="docutils literal"><span class="pre">{{</span> <span class="pre">foo</span> <span class="pre">}}</span></code> in your templates. Moreover, specifying
+<code class="docutils literal"><span class="pre">user_defined_filters</span></code> allow you to register you own filters. For example,
+passing <code class="docutils literal"><span class="pre">dict(hello=lambda</span> <span class="pre">name:</span> <span class="pre">'Hello</span> <span class="pre">%s'</span> <span class="pre">%</span> <span class="pre">name)</span></code> to this argument allows
+you to use <code class="docutils literal"><span class="pre">{{</span> <span class="pre">'world'</span> <span class="pre">|</span> <span class="pre">hello</span> <span class="pre">}}</span></code> in your templates. For more information
+regarding custom filters have a look at the
+<a class="reference external" href="http://jinja.pocoo.org/docs/dev/api/#writing-filters">Jinja Documentation</a></p>
<p>For more information on the variables and macros that can be referenced
in templates, make sure to read through the <a class="reference internal" href="code.html#macros"><span class="std std-ref">Macros</span></a> section</p>
</div>
<div class="section" id="setting-up-dependencies">
<h2>Setting up Dependencies<a class="headerlink" href="#setting-up-dependencies" title="Permalink to this headline">¶</a></h2>
-<p>We have two simple tasks that do not depend on each other. Here’s a few ways
+<p>We have two simple tasks that do not depend on each other. Here’s a few ways
you can define dependencies between them:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">t2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
@@ -430,7 +445,7 @@ something like this:</p>
<span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
<span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
- <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@airflow.com'</span><span class="p">],</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@example.com'</span><span class="p">],</span>
<span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
<span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
@@ -479,20 +494,20 @@ something like this:</p>
<h2>Testing<a class="headerlink" href="#testing" title="Permalink to this headline">¶</a></h2>
<div class="section" id="running-the-script">
<h3>Running the Script<a class="headerlink" href="#running-the-script" title="Permalink to this headline">¶</a></h3>
-<p>Time to run some tests. First let’s make sure that the pipeline
-parses. Let’s assume we’re saving the code from the previous step in
+<p>Time to run some tests. First let’s make sure that the pipeline
+parses. Let’s assume we’re saving the code from the previous step in
<code class="docutils literal"><span class="pre">tutorial.py</span></code> in the DAGs folder referenced in your <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.
The default location for your DAGs is <code class="docutils literal"><span class="pre">~/airflow/dags</span></code>.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span>python ~/airflow/dags/tutorial.py
</pre></div>
</div>
-<p>If the script does not raise an exception it means that you haven’t done
+<p>If the script does not raise an exception it means that you haven’t done
anything horribly wrong, and that your Airflow environment is somewhat
sound.</p>
</div>
<div class="section" id="command-line-metadata-validation">
<h3>Command Line Metadata Validation<a class="headerlink" href="#command-line-metadata-validation" title="Permalink to this headline">¶</a></h3>
-<p>Let’s run a few commands to validate this script further.</p>
+<p>Let’s run a few commands to validate this script further.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># print the list of active DAGs</span>
airflow list_dags
@@ -506,36 +521,36 @@ airflow list_tasks tutorial --tree
</div>
<div class="section" id="id1">
<h3>Testing<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
-<p>Let’s test by running the actual task instances on a specific date. The
+<p>Let’s test by running the actual task instances on a specific date. The
date specified in this context is an <code class="docutils literal"><span class="pre">execution_date</span></code>, which simulates the
scheduler running your task or dag at a specific date + time:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># command layout: command subcommand dag_id task_id date</span>
<span class="c1"># testing print_date</span>
-airflow <span class="nb">test</span> tutorial print_date 2015-06-01
+airflow <span class="nb">test</span> tutorial print_date <span class="m">2015</span>-06-01
<span class="c1"># testing sleep</span>
-airflow <span class="nb">test</span> tutorial sleep 2015-06-01
+airflow <span class="nb">test</span> tutorial sleep <span class="m">2015</span>-06-01
</pre></div>
</div>
<p>Now remember what we did with templating earlier? See how this template
gets rendered and executed by running this command:</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># testing templated</span>
-airflow <span class="nb">test</span> tutorial templated 2015-06-01
+airflow <span class="nb">test</span> tutorial templated <span class="m">2015</span>-06-01
</pre></div>
</div>
<p>This should result in displaying a verbose log of events and ultimately
running your bash command and printing the result.</p>
<p>Note that the <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">test</span></code> command runs task instances locally, outputs
-their log to stdout (on screen), doesn’t bother with dependencies, and
-doesn’t communicate state (running, success, failed, ...) to the database.
+their log to stdout (on screen), doesn’t bother with dependencies, and
+doesn’t communicate state (running, success, failed, …) to the database.
It simply allows testing a single task instance.</p>
</div>
<div class="section" id="backfill">
<h3>Backfill<a class="headerlink" href="#backfill" title="Permalink to this headline">¶</a></h3>
-<p>Everything looks like it’s running fine so let’s run a backfill.
+<p>Everything looks like it’s running fine so let’s run a backfill.
<code class="docutils literal"><span class="pre">backfill</span></code> will respect your dependencies, emit logs into files and talk to
-the database to record status. If you do have a webserver up, you’ll be able
+the database to record status. If you do have a webserver up, you’ll be able
to track the progress. <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">webserver</span></code> will start a web server if you
are interested in tracking the progress visually as your backfill progresses.</p>
<p>Note that if you use <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, individual task instances
@@ -547,17 +562,17 @@ which are used to populate the run schedule with task instances from this dag.</
<span class="c1"># airflow webserver --debug &</span>
<span class="c1"># start your backfill on a date range</span>
-airflow backfill tutorial -s 2015-06-01 -e 2015-06-07
+airflow backfill tutorial -s <span class="m">2015</span>-06-01 -e <span class="m">2015</span>-06-07
</pre></div>
</div>
</div>
</div>
<div class="section" id="what-s-next">
-<h2>What’s Next?<a class="headerlink" href="#what-s-next" title="Permalink to this headline">¶</a></h2>
-<p>That’s it, you’ve written, tested and backfilled your very first Airflow
+<h2>What’s Next?<a class="headerlink" href="#what-s-next" title="Permalink to this headline">¶</a></h2>
+<p>That’s it, you’ve written, tested and backfilled your very first Airflow
pipeline. Merging your code into a code repository that has a master scheduler
running against it should get it to get triggered and run every day.</p>
-<p>Here’s a few things you might want to do next:</p>
+<p>Here’s a few things you might want to do next:</p>
<ul>
<li><p class="first">Take an in-depth tour of the UI - click all the things!</p>
</li>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/ui.html
----------------------------------------------------------------------
diff --git a/ui.html b/ui.html
index 5852c52..92fa47d 100644
--- a/ui.html
+++ b/ui.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -175,7 +180,7 @@
<div class="section" id="ui-screenshots">
<h1>UI / Screenshots<a class="headerlink" href="#ui-screenshots" title="Permalink to this headline">¶</a></h1>
<p>The Airflow UI make it easy to monitor and troubleshoot your data pipelines.
-Here’s a quick overview of some of the features and visualizations you
+Here’s a quick overview of some of the features and visualizations you
can find in the Airflow UI.</p>
<div class="section" id="dags-view">
<h2>DAGs View<a class="headerlink" href="#dags-view" title="Permalink to this headline">¶</a></h2>
@@ -197,7 +202,7 @@ the blocking ones.</p>
<hr class="docutils" />
<div class="section" id="graph-view">
<h2>Graph View<a class="headerlink" href="#graph-view" title="Permalink to this headline">¶</a></h2>
-<p>The graph view is perhaps the most comprehensive. Visualize your DAG’s
+<p>The graph view is perhaps the most comprehensive. Visualize your DAG’s
dependencies and their current status for a specific run.</p>
<hr class="docutils" />
<img alt="_images/graph.png" src="_images/graph.png" />
@@ -207,7 +212,7 @@ dependencies and their current status for a specific run.</p>
<h2>Variable View<a class="headerlink" href="#variable-view" title="Permalink to this headline">¶</a></h2>
<p>The variable view allows you to list, create, edit or delete the key-value pair
of a variable used during jobs. Value of a variable will be hidden if the key contains
-any words in (‘password’, ‘secret’, ‘passwd’, ‘authorization’, ‘api_key’, ‘apikey’, ‘access_token’)
+any words in (‘password’, ‘secret’, ‘passwd’, ‘authorization’, ‘api_key’, ‘apikey’, ‘access_token’)
by default, but can be configured to show in clear-text.</p>
<hr class="docutils" />
<img alt="_images/variable_hidden.png" src="_images/variable_hidden.png" />
@@ -242,7 +247,7 @@ provide yet more context.</p>
<hr class="docutils" />
<div class="section" id="task-instance-context-menu">
<h2>Task Instance Context Menu<a class="headerlink" href="#task-instance-context-menu" title="Permalink to this headline">¶</a></h2>
-<p>From the pages seen above (tree view, graph view, gantt, ...), it is always
+<p>From the pages seen above (tree view, graph view, gantt, …), it is always
possible to click on a task instance, and get to this rich context menu
that can take you to more detailed metadata, and perform some actions.</p>
<hr class="docutils" />
[06/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/concepts.html
----------------------------------------------------------------------
diff --git a/concepts.html b/concepts.html
index a329ae8..16d995c 100644
--- a/concepts.html
+++ b/concepts.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -208,23 +213,23 @@ workflows.</p>
<h2>Core Ideas<a class="headerlink" href="#core-ideas" title="Permalink to this headline">¶</a></h2>
<div class="section" id="dags">
<h3>DAGs<a class="headerlink" href="#dags" title="Permalink to this headline">¶</a></h3>
-<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> – or a Directed Acyclic Graph – is a collection of all
+<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> – or a Directed Acyclic Graph – is a collection of all
the tasks you want to run, organized in a way that reflects their relationships
and dependencies.</p>
<p>For example, a simple DAG could consist of three tasks: A, B, and C. It could
say that A has to run successfully before B can run, but C can run anytime. It
could say that task A times out after 5 minutes, and B can be restarted up to 5
times in case it fails. It might also say that the workflow will run every night
-at 10pm, but shouldn’t start until a certain date.</p>
+at 10pm, but shouldn’t start until a certain date.</p>
<p>In this way, a DAG describes <em>how</em> you want to carry out your workflow; but
-notice that we haven’t said anything about <em>what</em> we actually want to do! A, B,
+notice that we haven’t said anything about <em>what</em> we actually want to do! A, B,
and C could be anything. Maybe A prepares data for B to analyze while C sends an
email. Or perhaps A monitors your location so B can open your garage door while
-C turns on your house lights. The important thing is that the DAG isn’t
+C turns on your house lights. The important thing is that the DAG isn’t
concerned with what its constituent tasks do; its job is to make sure that
whatever they do happens at the right time, or in the right order, or with the
right handling of any unexpected issues.</p>
-<p>DAGs are defined in standard Python files that are placed in Airflow’s
+<p>DAGs are defined in standard Python files that are placed in Airflow’s
<code class="docutils literal"><span class="pre">DAG_FOLDER</span></code>. Airflow will execute the code in each file to dynamically build
the <code class="docutils literal"><span class="pre">DAG</span></code> objects. You can have as many DAGs as you want, each describing an
arbitrary number of tasks. In general, each one should correspond to a single
@@ -245,7 +250,7 @@ scope.</p>
</div>
<p>Sometimes this can be put to good use. For example, a common pattern with
<code class="docutils literal"><span class="pre">SubDagOperator</span></code> is to define the subdag inside a function so that Airflow
-doesn’t try to load it as a standalone DAG.</p>
+doesn’t try to load it as a standalone DAG.</p>
</div>
<div class="section" id="default-arguments">
<h4>Default Arguments<a class="headerlink" href="#default-arguments" title="Permalink to this headline">¶</a></h4>
@@ -278,13 +283,13 @@ any of its operators. This makes it easy to apply a common parameter to many ope
<p>While DAGs describe <em>how</em> to run a workflow, <code class="docutils literal"><span class="pre">Operators</span></code> determine what
actually gets done.</p>
<p>An operator describes a single task in a workflow. Operators are usually (but
-not always) atomic, meaning they can stand on their own and don’t need to share
+not always) atomic, meaning they can stand on their own and don’t need to share
resources with any other operators. The DAG will make sure that operators run in
the correct certain order; other than those dependencies, operators generally
run independently. In fact, they may run on two completely different machines.</p>
<p>This is a subtle but very important point: in general, if two operators need to
share information, like a filename or small amount of data, you should consider
-combining them into a single operator. If it absolutely can’t be avoided,
+combining them into a single operator. If it absolutely can’t be avoided,
Airflow does have a feature for operator cross-communication called XCom that is
described elsewhere in this document.</p>
<p>Airflow provides operators for many common tasks, including:</p>
@@ -293,14 +298,14 @@ described elsewhere in this document.</p>
<li><code class="docutils literal"><span class="pre">PythonOperator</span></code> - calls an arbitrary Python function</li>
<li><code class="docutils literal"><span class="pre">EmailOperator</span></code> - sends an email</li>
<li><code class="docutils literal"><span class="pre">HTTPOperator</span></code> - sends an HTTP request</li>
-<li><code class="docutils literal"><span class="pre">SqlOperator</span></code> - executes a SQL command</li>
-<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etc...</li>
+<li><code class="docutils literal"><span class="pre">MySqlOperator</span></code>, <code class="docutils literal"><span class="pre">SqliteOperator</span></code>, <code class="docutils literal"><span class="pre">PostgresOperator</span></code>, <code class="docutils literal"><span class="pre">MsSqlOperator</span></code>, <code class="docutils literal"><span class="pre">OracleOperator</span></code>, <code class="docutils literal"><span class="pre">JdbcOperator</span></code>, etc. - executes a SQL command</li>
+<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etc…</li>
</ul>
<p>In addition to these basic building blocks, there are many more specific
operators: <code class="docutils literal"><span class="pre">DockerOperator</span></code>, <code class="docutils literal"><span class="pre">HiveOperator</span></code>, <code class="docutils literal"><span class="pre">S3FileTransferOperator</span></code>,
-<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>... you get the idea!</p>
+<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>… you get the idea!</p>
<p>The <code class="docutils literal"><span class="pre">airflow/contrib/</span></code> directory contains yet more operators built by the
-community. These operators aren’t always as complete or well-tested as those in
+community. These operators aren’t always as complete or well-tested as those in
the main distribution, but allow users to more easily add new functionality to
the platform.</p>
<p>Operators are only loaded by Airflow if they are assigned to a DAG.</p>
@@ -344,7 +349,7 @@ functionally equivalent:</p>
<p>When using the bitshift to compose operators, the relationship is set in the
direction that the bitshift operator points. For example, <code class="docutils literal"><span class="pre">op1</span> <span class="pre">>></span> <span class="pre">op2</span></code> means
that <code class="docutils literal"><span class="pre">op1</span></code> runs first and <code class="docutils literal"><span class="pre">op2</span></code> runs second. Multiple operators can be
-composed – keep in mind the chain is executed left-to-right and the rightmost
+composed – keep in mind the chain is executed left-to-right and the rightmost
object is always returned. For example:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span> <span class="o">>></span> <span class="n">op2</span> <span class="o">>></span> <span class="n">op3</span> <span class="o"><<</span> <span class="n">op4</span>
</pre></div>
@@ -367,8 +372,7 @@ object is always returned. For example:</p>
<p>We can put this all together to build a simple pipeline:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
<span class="p">(</span>
- <span class="n">dag</span>
- <span class="o">>></span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
+ <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
<span class="o">>></span> <span class="n">BashOperator</span><span class="p">(</span>
<span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_1'</span><span class="p">,</span>
<span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "HELLO!"'</span><span class="p">)</span>
@@ -382,7 +386,7 @@ object is always returned. For example:</p>
</div>
<div class="section" id="tasks">
<h3>Tasks<a class="headerlink" href="#tasks" title="Permalink to this headline">¶</a></h3>
-<p>Once an operator is instantiated, it is referred to as a “task”. The
+<p>Once an operator is instantiated, it is referred to as a “task”. The
instantiation defines specific values when calling the abstract operator, and
the parameterized task becomes a node in a DAG.</p>
</div>
@@ -390,12 +394,12 @@ the parameterized task becomes a node in a DAG.</p>
<h3>Task Instances<a class="headerlink" href="#task-instances" title="Permalink to this headline">¶</a></h3>
<p>A task instance represents a specific run of a task and is characterized as the
combination of a dag, a task, and a point in time. Task instances also have an
-indicative state, which could be “running”, “success”, “failed”, “skipped”, “up
-for retry”, etc.</p>
+indicative state, which could be “running”, “success”, “failed”, “skipped”, “up
+for retry”, etc.</p>
</div>
<div class="section" id="workflows">
<h3>Workflows<a class="headerlink" href="#workflows" title="Permalink to this headline">¶</a></h3>
-<p>You’re now familiar with the core building blocks of Airflow.
+<p>You’re now familiar with the core building blocks of Airflow.
Some of the concepts may sound very similar, but the vocabulary can
be conceptualized like this:</p>
<ul class="simple">
@@ -457,8 +461,8 @@ gets prioritized accordingly.</p>
reached, runnable tasks get queued and their state will show as such in the
UI. As slots free up, queued tasks start running based on the
<code class="docutils literal"><span class="pre">priority_weight</span></code> (of the task and its descendants).</p>
-<p>Note that by default tasks aren’t assigned to any pool and their
-execution parallelism is only limited to the executor’s setting.</p>
+<p>Note that by default tasks aren’t assigned to any pool and their
+execution parallelism is only limited to the executor’s setting.</p>
</div>
<div class="section" id="connections">
<h3>Connections<a class="headerlink" href="#connections" title="Permalink to this headline">¶</a></h3>
@@ -489,7 +493,7 @@ variable to be in a URI format (e.g.
<p>When using the CeleryExecutor, the celery queues that tasks are sent to
can be specified. <code class="docutils literal"><span class="pre">queue</span></code> is an attribute of BaseOperator, so any
task can be assigned to any queue. The default queue for the environment
-is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>‘s <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines
+is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>’s <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines
the queue that tasks get assigned to when not specified, as well as which
queue Airflow workers listen to when started.</p>
<p>Workers can listen to one or multiple queues of tasks. When a worker is
@@ -505,16 +509,16 @@ itself because it needs a very specific environment and security rights).</p>
<div class="section" id="xcoms">
<h3>XComs<a class="headerlink" href="#xcoms" title="Permalink to this headline">¶</a></h3>
<p>XComs let tasks exchange messages, allowing more nuanced forms of control and
-shared state. The name is an abbreviation of “cross-communication”. XComs are
+shared state. The name is an abbreviation of “cross-communication”. XComs are
principally defined by a key, value, and timestamp, but also track attributes
like the task/DAG that created the XCom and when it should become visible. Any
object that can be pickled can be used as an XCom value, so users should make
sure to use objects of appropriate size.</p>
-<p>XComs can be “pushed” (sent) or “pulled” (received). When a task pushes an
+<p>XComs can be “pushed” (sent) or “pulled” (received). When a task pushes an
XCom, it makes it generally available to other tasks. Tasks can push XComs at
any time by calling the <code class="docutils literal"><span class="pre">xcom_push()</span></code> method. In addition, if a task returns
-a value (either from its Operator’s <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a
-PythonOperator’s <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that
+a value (either from its Operator’s <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a
+PythonOperator’s <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that
value is automatically pushed.</p>
<p>Tasks call <code class="docutils literal"><span class="pre">xcom_pull()</span></code> to retrieve XComs, optionally applying filters
based on criteria like <code class="docutils literal"><span class="pre">key</span></code>, source <code class="docutils literal"><span class="pre">task_ids</span></code>, and source <code class="docutils literal"><span class="pre">dag_id</span></code>. By
@@ -533,7 +537,7 @@ passed, then a correpsonding list of XCom values is returned.</p>
<span class="n">value</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task_instance'</span><span class="p">]</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'pushing_task'</span><span class="p">)</span>
</pre></div>
</div>
-<p>It is also possible to pull XCom directly in a template, here’s an example
+<p>It is also possible to pull XCom directly in a template, here’s an example
of what this may look like:</p>
<div class="code sql highlight-default"><div class="highlight"><pre><span></span><span class="n">SELECT</span> <span class="o">*</span> <span class="n">FROM</span> <span class="p">{{</span> <span class="n">task_instance</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="s1">'table_name'</span><span class="p">)</span> <span class="p">}}</span>
</pre></div>
@@ -575,9 +579,9 @@ directly downstream from the BranchPythonOperator task.</p>
will invariably lead to block tasks that depend on their past successes.
<code class="docutils literal"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
<code class="docutils literal"><span class="pre">skipped</span></code>.</p>
-<p>If you want to skip some tasks, keep in mind that you can’t have an empty
+<p>If you want to skip some tasks, keep in mind that you can’t have an empty
path, if so make a dummy task.</p>
-<p>like this, the dummy task “branch_false” is skipped</p>
+<p>like this, the dummy task “branch_false” is skipped</p>
<img alt="_images/branch_good.png" src="_images/branch_good.png" />
<p>Not like this, where the join task is skipped</p>
<img alt="_images/branch_bad.png" src="_images/branch_bad.png" />
@@ -649,11 +653,11 @@ the tasks contained within the SubDAG:</p>
<img alt="_images/subdag_zoom.png" src="_images/subdag_zoom.png" />
<p>Some other tips when using SubDAGs:</p>
<ul class="simple">
-<li>by convention, a SubDAG’s <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and
+<li>by convention, a SubDAG’s <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and
a dot. As in <code class="docutils literal"><span class="pre">parent.child</span></code></li>
<li>share arguments between the main DAG and the SubDAG by passing arguments to
the SubDAG operator (as demonstrated above)</li>
-<li>SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is
+<li>SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is
set to <code class="docutils literal"><span class="pre">None</span></code> or <code class="docutils literal"><span class="pre">@once</span></code>, the SubDAG will succeed without having done
anything</li>
<li>clearing a SubDagOperator also clears the state of the tasks within</li>
@@ -685,8 +689,8 @@ directly upstream tasks have succeeded, Airflow allows for more complex
dependency settings.</p>
<p>All operators have a <code class="docutils literal"><span class="pre">trigger_rule</span></code> argument which defines the rule by which
the generated task get triggered. The default value for <code class="docutils literal"><span class="pre">trigger_rule</span></code> is
-<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as “trigger this task when all directly
-upstream tasks have succeeded”. All other rules described here are based
+<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as “trigger this task when all directly
+upstream tasks have succeeded”. All other rules described here are based
on direct parent tasks and are values that can be passed to any operator
while creating tasks:</p>
<ul class="simple">
@@ -699,7 +703,7 @@ while creating tasks:</p>
</ul>
<p>Note that these can be used in conjunction with <code class="docutils literal"><span class="pre">depends_on_past</span></code> (boolean)
that, when set to <code class="docutils literal"><span class="pre">True</span></code>, keeps a task from getting triggered if the
-previous schedule for the task hasn’t succeeded.</p>
+previous schedule for the task hasn’t succeeded.</p>
</div>
<div class="section" id="latest-run-only">
<h3>Latest Run Only<a class="headerlink" href="#latest-run-only" title="Permalink to this headline">¶</a></h3>
@@ -772,16 +776,16 @@ and <code class="docutils literal"><span class="pre">task2</span></code> has suc
but sometimes unexpectedly.</p>
<p>Zombie tasks are characterized by the absence
of an heartbeat (emitted by the job periodically) and a <code class="docutils literal"><span class="pre">running</span></code> status
-in the database. They can occur when a worker node can’t reach the database,
+in the database. They can occur when a worker node can’t reach the database,
when Airflow processes are killed externally, or when a node gets rebooted
-for instance. Zombie killing is performed periodically by the scheduler’s
+for instance. Zombie killing is performed periodically by the scheduler’s
process.</p>
<p>Undead processes are characterized by the existence of a process and a matching
-heartbeat, but Airflow isn’t aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database.
+heartbeat, but Airflow isn’t aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database.
This mismatch typically occurs as the state of the database is altered,
-most likely by deleting rows in the “Task Instances” view in the UI.
+most likely by deleting rows in the “Task Instances” view in the UI.
Tasks are instructed to verify their state as part of the heartbeat routine,
-and terminate themselves upon figuring out that they are in this “undead”
+and terminate themselves upon figuring out that they are in this “undead”
state.</p>
</div>
<div class="section" id="cluster-policy">
@@ -792,10 +796,10 @@ attributes. It receives a single argument as a reference to task objects,
and is expected to alter its attributes.</p>
<p>For example, this function could apply a specific queue property when
using a specific operator, or enforce a task timeout policy, making sure
-that no tasks run for more than 48 hours. Here’s an example of what this
+that no tasks run for more than 48 hours. Here’s an example of what this
may look like inside your <code class="docutils literal"><span class="pre">airflow_settings.py</span></code>:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">policy</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span>
<span class="n">task</span><span class="o">.</span><span class="n">queue</span> <span class="o">=</span> <span class="s2">"sensor_queue"</span>
<span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">></span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">):</span>
<span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">)</span>
@@ -804,8 +808,8 @@ may look like inside your <code class="docutils literal"><span class="pre">airfl
</div>
<div class="section" id="documentation-notes">
<h3>Documentation & Notes<a class="headerlink" href="#documentation-notes" title="Permalink to this headline">¶</a></h3>
-<p>It’s possible to add documentation or notes to your dags & task objects that
-become visible in the web interface (“Graph View” for dags, “Task Details” for
+<p>It’s possible to add documentation or notes to your dags & task objects that
+become visible in the web interface (“Graph View” for dags, “Task Details” for
tasks). There are a set of special task attributes that get rendered as rich
content if defined:</p>
<table border="1" class="docutils">
@@ -845,7 +849,7 @@ to the related tasks in Airflow.</p>
<span class="sd">"""</span>
<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span>
-<span class="n">dag</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="n">__doc__</span>
+<span class="n">dag</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="vm">__doc__</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span><span class="s2">"foo"</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
<span class="n">t</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="s2">"""</span><span class="se">\</span>
@@ -854,11 +858,11 @@ to the related tasks in Airflow.</p>
<span class="s2">"""</span>
</pre></div>
</div>
-<p>This content will get rendered as markdown respectively in the “Graph View” and
-“Task Details” pages.</p>
+<p>This content will get rendered as markdown respectively in the “Graph View” and
+“Task Details” pages.</p>
</div>
<div class="section" id="jinja-templating">
-<h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">¶</a></h3>
+<span id="id1"></span><h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">¶</a></h3>
<p>Airflow leverages the power of
<a class="reference external" href="http://jinja.pocoo.org/docs/dev/">Jinja Templating</a> and this can be a
powerful tool to use in combination with macros (see the <a class="reference internal" href="code.html#macros"><span class="std std-ref">Macros</span></a> section).</p>
@@ -876,8 +880,9 @@ to a Bash script using the <code class="docutils literal"><span class="pre">Bash
<p>Here, <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> is a macro, and because the <code class="docutils literal"><span class="pre">env</span></code> parameter of the
<code class="docutils literal"><span class="pre">BashOperator</span></code> is templated with Jinja, the execution date will be available
as an environment variable named <code class="docutils literal"><span class="pre">EXECUTION_DATE</span></code> in your Bash script.</p>
-<p>You can use Jinja templating with every parameter that is marked as “templated”
-in the documentation.</p>
+<p>You can use Jinja templating with every parameter that is marked as “templated”
+in the documentation. Template substitution occurs just before the pre_execute
+function of your operator is called.</p>
</div>
</div>
<div class="section" id="packaged-dags">
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/configuration.html
----------------------------------------------------------------------
diff --git a/configuration.html b/configuration.html
index ea1d2e9..881716e 100644
--- a/configuration.html
+++ b/configuration.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -92,6 +97,7 @@
<li class="toctree-l2"><a class="reference internal" href="#setting-up-a-backend">Setting up a Backend</a></li>
<li class="toctree-l2"><a class="reference internal" href="#connections">Connections</a></li>
<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-celery">Scaling Out with Celery</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-dask">Scaling Out with Dask</a></li>
<li class="toctree-l2"><a class="reference internal" href="#logs">Logs</a></li>
<li class="toctree-l2"><a class="reference internal" href="#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li>
<li class="toctree-l2"><a class="reference internal" href="#integration-with-systemd">Integration with systemd</a></li>
@@ -180,7 +186,7 @@ building a production-grade environment requires a bit more work!</p>
<div class="section" id="setting-configuration-options">
<span id="setting-options"></span><h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">¶</a></h2>
<p>The first time you run Airflow, it will create a file called <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in
-your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you
+your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you
can edit it to change any of the settings. You can also set options with environment variables by using this format:
<code class="docutils literal"><span class="pre">$AIRFLOW__{SECTION}__{KEY}</span></code> (note the double underscores).</p>
<p>For example, the
@@ -222,10 +228,10 @@ specific schema in the Postgres connection URI, you may
want to set a default schema for your role with a
command similar to <code class="docutils literal"><span class="pre">ALTER</span> <span class="pre">ROLE</span> <span class="pre">username</span> <span class="pre">SET</span> <span class="pre">search_path</span> <span class="pre">=</span> <span class="pre">airflow,</span> <span class="pre">foobar;</span></code></p>
</div>
-<p>Once you’ve setup your database to host Airflow, you’ll need to alter the
+<p>Once you’ve setup your database to host Airflow, you’ll need to alter the
SqlAlchemy connection string located in your configuration file
-<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the “executor”
-setting to use “LocalExecutor”, an executor that can parallelize task
+<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the “executor”
+setting to use “LocalExecutor”, an executor that can parallelize task
instances locally.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># initialize the database</span>
airflow initdb
@@ -237,12 +243,35 @@ airflow initdb
<p>Airflow needs to know how to connect to your environment. Information
such as hostname, port, login and passwords to other systems and services is
handled in the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. The pipeline code you
-will author will reference the ‘conn_id’ of the Connection objects.</p>
+will author will reference the ‘conn_id’ of the Connection objects.</p>
<img alt="_images/connections.png" src="_images/connections.png" />
<p>By default, Airflow will save the passwords for the connection in plain text
within the metadata database. The <code class="docutils literal"><span class="pre">crypto</span></code> package is highly recommended
during installation. The <code class="docutils literal"><span class="pre">crypto</span></code> package does require that your operating
system have libffi-dev installed.</p>
+<p>If <code class="docutils literal"><span class="pre">crypto</span></code> package was not installed initially, you can still enable encryption for
+connections by following steps below:</p>
+<ol class="arabic simple">
+<li>Install crypto package <code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[crypto]</span></code></li>
+<li>Generate fernet_key, using this code snippet below. fernet_key must be a base64-encoded 32-byte key.</li>
+</ol>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cryptography.fernet</span> <span class="k">import</span> <span class="n">Fernet</span>
+<span class="n">fernet_key</span><span class="o">=</span> <span class="n">Fernet</span><span class="o">.</span><span class="n">generate_key</span><span class="p">()</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">fernet_key</span><span class="p">)</span> <span class="c1"># your fernet_key, keep it in secured place!</span>
+</pre></div>
+</div>
+<p>3. Replace <code class="docutils literal"><span class="pre">airflow.cfg</span></code> fernet_key value with the one from step 2.
+Alternatively, you can store your fernet_key in OS environment variable. You
+do not need to change <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in this case as AirFlow will use environment
+variable over the value in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Note the double underscores</span>
+EXPORT <span class="nv">AIRFLOW__CORE__FERNET_KEY</span> <span class="o">=</span> your_fernet_key
+</pre></div>
+</div>
+<ol class="arabic simple" start="4">
+<li>Restart AirFlow webserver.</li>
+<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it.</li>
+</ol>
<p>Connections in Airflow pipelines can be created using environment variables.
The environment variable needs to have a prefix of <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> for
Airflow with the value in a URI format to use the connection properly. Please
@@ -252,7 +281,7 @@ variables and connections.</p>
<div class="section" id="scaling-out-with-celery">
<h2>Scaling Out with Celery<a class="headerlink" href="#scaling-out-with-celery" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal"><span class="pre">CeleryExecutor</span></code> is one of the ways you can scale out the number of workers. For this
-to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, ...) and
+to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, …) and
change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point the executor parameter to
<code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and provide the related Celery settings.</p>
<p>For more information about setting up a Celery broker, refer to the
@@ -280,10 +309,37 @@ subcommand</p>
</div>
<p>Your worker should start picking up tasks as soon as they get fired in
its direction.</p>
-<p>Note that you can also run “Celery Flower”, a web UI built on top of Celery,
+<p>Note that you can also run “Celery Flower”, a web UI built on top of Celery,
to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code>
to start a Flower web server.</p>
</div>
+<div class="section" id="scaling-out-with-dask">
+<h2>Scaling Out with Dask<a class="headerlink" href="#scaling-out-with-dask" title="Permalink to this headline">¶</a></h2>
+<p><code class="docutils literal"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p>
+<p>Dask clusters can be run on a single machine or on remote networks. For complete
+details, consult the <a class="reference external" href="https://distributed.readthedocs.io/">Distributed documentation</a>.</p>
+<p>To create a cluster, first start a Scheduler:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span>
+<span class="nv">DASK_HOST</span><span class="o">=</span><span class="m">127</span>.0.0.1
+<span class="nv">DASK_PORT</span><span class="o">=</span><span class="m">8786</span>
+
+dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv">$DASK_PORT</span>
+</pre></div>
+</div>
+<p>Next start at least one Worker on any machine that can connect to the host:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span>
+</pre></div>
+</div>
+<p>Edit your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal"><span class="pre">DaskExecutor</span></code> and provide
+the Dask Scheduler address in the <code class="docutils literal"><span class="pre">[dask]</span></code> section.</p>
+<p>Please note:</p>
+<ul class="simple">
+<li>Each Dask worker must be able to import Airflow and any dependencies you
+require.</li>
+<li>Dask does not support queues. If an Airflow task was created with a queue, a
+warning will be raised but the task will be submitted to the cluster.</li>
+</ul>
+</div>
<div class="section" id="logs">
<h2>Logs<a class="headerlink" href="#logs" title="Permalink to this headline">¶</a></h2>
<p>Users can specify a logs folder in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>. By default, it is in
@@ -302,13 +358,14 @@ To enable this feature, <code class="docutils literal"><span class="pre">airflow
<span class="nv">encrypt_s3_logs</span> <span class="o">=</span> False
</pre></div>
</div>
-<p>Remote logging uses an existing Airflow connection to read/write logs. If you don’t
+<p>Remote logging uses an existing Airflow connection to read/write logs. If you don’t
have a connection properly setup, this will fail. In the above example, Airflow will
try to use <code class="docutils literal"><span class="pre">S3Hook('MyS3Conn')</span></code>.</p>
<p>In the Airflow Web UI, local logs take precedance over remote logs. If local logs
can not be found or accessed, the remote logs will be displayed. Note that logs
are only sent to remote storage once a task completes (including failure). In other
-words, remote logs for running tasks are unavailable.</p>
+words, remote logs for running tasks are unavailable. Logs are stored in the log
+folder as <code class="docutils literal"><span class="pre">{dag_id}/{task_id}/{execution_date}/{try_number}.log</span></code>.</p>
</div>
<div class="section" id="scaling-out-on-mesos-community-contributed">
<h2>Scaling Out on Mesos (community contributed)<a class="headerlink" href="#scaling-out-on-mesos-community-contributed" title="Permalink to this headline">¶</a></h2>
@@ -317,7 +374,7 @@ For this to work, you need a running mesos cluster and you must perform the foll
steps -</p>
<ol class="arabic simple">
<li>Install airflow on a machine where web server and scheduler will run,
-let’s refer to this as the “Airflow server”.</li>
+let’s refer to this as the “Airflow server”.</li>
<li>On the Airflow server, install mesos python eggs from <a class="reference external" href="http://open.mesosphere.com/downloads/mesos/">mesos downloads</a>.</li>
<li>On the Airflow server, use a database (such as mysql) which can be accessed from mesos
slave machines and add configuration in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</li>
@@ -374,7 +431,7 @@ integrated with upstart</p>
</div>
<div class="section" id="test-mode">
<h2>Test Mode<a class="headerlink" href="#test-mode" title="Permalink to this headline">¶</a></h2>
-<p>Airflow has a fixed set of “test mode” configuration options. You can load these
+<p>Airflow has a fixed set of “test mode” configuration options. You can load these
at any time by calling <code class="docutils literal"><span class="pre">airflow.configuration.load_test_config()</span></code> (note this
operation is not reversible!). However, some options (like the DAG_FOLDER) are
loaded before you have a chance to call load_test_config(). In order to eagerly load
@@ -383,7 +440,7 @@ the test configuration, set test_mode in airflow.cfg:</p>
<span class="nv">unit_test_mode</span> <span class="o">=</span> True
</pre></div>
</div>
-<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>),
+<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>),
you can also set the env var <code class="docutils literal"><span class="pre">AIRFLOW__CORE__UNIT_TEST_MODE</span></code> to temporarily overwrite
airflow.cfg.</p>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/faq.html
----------------------------------------------------------------------
diff --git a/faq.html b/faq.html
index 7b6c8ab..f223045 100644
--- a/faq.html
+++ b/faq.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -98,10 +103,10 @@
<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li>
<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">FAQ</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
-<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto">Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?</a></li>
-<li class="toctree-l2"><a class="reference internal" href="#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#how-can-i-create-dags-dynamically">How can I create DAGs dynamically?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#what-are-all-the-airflow-run-commands-in-my-process-list">What are all the <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span></code> commands in my process list?</a></li>
</ul>
@@ -173,11 +178,11 @@
<div class="section" id="faq">
<h1>FAQ<a class="headerlink" href="#faq" title="Permalink to this headline">¶</a></h1>
<div class="section" id="why-isn-t-my-task-getting-scheduled">
-<h2>Why isn’t my task getting scheduled?<a class="headerlink" href="#why-isn-t-my-task-getting-scheduled" title="Permalink to this headline">¶</a></h2>
+<h2>Why isn’t my task getting scheduled?<a class="headerlink" href="#why-isn-t-my-task-getting-scheduled" title="Permalink to this headline">¶</a></h2>
<p>There are very many reasons why your task might not be getting scheduled.
Here are some of the common causes:</p>
<ul class="simple">
-<li>Does your script “compile”, can the Airflow engine parse it and find your
+<li>Does your script “compile”, can the Airflow engine parse it and find your
DAG object. To test this, you can run <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_dags</span></code> and
confirm that your DAG shows up in the list. You can also run
<code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_tasks</span> <span class="pre">foo_dag_id</span> <span class="pre">--tree</span></code> and confirm that your task
@@ -189,9 +194,9 @@ task soon after the <code class="docutils literal"><span class="pre">start_date<
<li>Is your <code class="docutils literal"><span class="pre">schedule_interval</span></code> set properly? The default <code class="docutils literal"><span class="pre">schedule_interval</span></code>
is one day (<code class="docutils literal"><span class="pre">datetime.timedelta(1)</span></code>). You must specify a different <code class="docutils literal"><span class="pre">schedule_interval</span></code>
directly to the DAG object you instantiate, not as a <code class="docutils literal"><span class="pre">default_param</span></code>, as task instances
-do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li>
+do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li>
<li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> beyond where you can see it in the UI? If you
-set your it to some time say 3 months ago, you won’t be able to see
+set your <code class="docutils literal"><span class="pre">start_date</span></code> to some time say 3 months ago, you won’t be able to see
it in the main view in the UI, but you should be able to see it in the
<code class="docutils literal"><span class="pre">Menu</span> <span class="pre">-></span> <span class="pre">Browse</span> <span class="pre">->Task</span> <span class="pre">Instances</span></code>.</li>
<li>Are the dependencies for the task met. The task instances directly
@@ -203,7 +208,7 @@ what it means.
You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Instance</span> <span class="pre">Details</span></code>
page for your task.</li>
<li>Are the DagRuns you need created and active? A DagRun represents a specific
-execution of an entire DAG and has a state (running, success, failed, ...).
+execution of an entire DAG and has a state (running, success, failed, …).
The scheduler creates new DagRun as it moves forward, but never goes back
in time to create new ones. The scheduler only evaluates <code class="docutils literal"><span class="pre">running</span></code> DagRuns
to see what task instances it can trigger. Note that clearing tasks
@@ -220,19 +225,17 @@ how many <code class="docutils literal"><span class="pre">running</span></code>
sure you fully understand how it proceeds.</p>
</div>
<div class="section" id="how-do-i-trigger-tasks-based-on-another-task-s-failure">
-<h2>How do I trigger tasks based on another task’s failure?<a class="headerlink" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure" title="Permalink to this headline">¶</a></h2>
+<h2>How do I trigger tasks based on another task’s failure?<a class="headerlink" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure" title="Permalink to this headline">¶</a></h2>
<p>Check out the <code class="docutils literal"><span class="pre">Trigger</span> <span class="pre">Rule</span></code> section in the Concepts section of the
documentation</p>
</div>
<div class="section" id="why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto">
<h2>Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?<a class="headerlink" href="#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto" title="Permalink to this headline">¶</a></h2>
-<ul class="simple">
-<li>Verify that the <code class="docutils literal"><span class="pre">fernet_key</span></code> defined in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code> is a valid Fernet key. It must be a base64-encoded 32-byte key. You need to restart the webserver after you update the key</li>
-<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it</li>
-</ul>
+<p>Check out the <code class="docutils literal"><span class="pre">Connections</span></code> section in the Configuration section of the
+documentation</p>
</div>
<div class="section" id="what-s-the-deal-with-start-date">
-<h2>What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?<a class="headerlink" href="#what-s-the-deal-with-start-date" title="Permalink to this headline">¶</a></h2>
+<h2>What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?<a class="headerlink" href="#what-s-the-deal-with-start-date" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal"><span class="pre">start_date</span></code> is partly legacy from the pre-DagRun era, but it is still
relevant in many ways. When creating a new DAG, you probably want to set
a global <code class="docutils literal"><span class="pre">start_date</span></code> for your tasks using <code class="docutils literal"><span class="pre">default_args</span></code>. The first
@@ -241,7 +244,7 @@ task. From that point on, the scheduler creates new DagRuns based on
your <code class="docutils literal"><span class="pre">schedule_interval</span></code> and the corresponding task instances run as your
dependencies are met. When introducing new tasks to your DAG, you need to
pay special attention to <code class="docutils literal"><span class="pre">start_date</span></code>, and may want to reactivate
-inactive DagRuns to get the new task to get onboarded properly.</p>
+inactive DagRuns to get the new task onboarded properly.</p>
<p>We recommend against using dynamic values as <code class="docutils literal"><span class="pre">start_date</span></code>, especially
<code class="docutils literal"><span class="pre">datetime.now()</span></code> as it can be quite confusing. The task is triggered
once the period closes, and in theory an <code class="docutils literal"><span class="pre">@hourly</span></code> DAG would never get to
@@ -257,16 +260,16 @@ the execution of tasks within the schedule interval.
While <code class="docutils literal"><span class="pre">schedule_interval</span></code> does allow specifying a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code>
object, we recommend using the macros or cron expressions instead, as
it enforces this idea of rounded schedules.</p>
-<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> it’s important to pay special attention
+<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> it’s important to pay special attention
to <code class="docutils literal"><span class="pre">start_date</span></code> as the past dependency is not enforced only on the specific
-schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. It’ also
+schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. It’s also
important to watch DagRun activity status in time when introducing
new <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, unless you are planning on running a backfill
for the new task(s).</p>
<p>Also important to note is that the tasks <code class="docutils literal"><span class="pre">start_date</span></code>, in the context of a
-backfill CLI command, get overridden by the backfill’s command <code class="docutils literal"><span class="pre">start_date</span></code>.
+backfill CLI command, get overridden by the backfill’s command <code class="docutils literal"><span class="pre">start_date</span></code>.
This allows for a backfill on tasks that have <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> to
-actually start, if it wasn’t the case, the backfill just wouldn’t start.</p>
+actually start, if that wasn’t the case, the backfill just wouldn’t start.</p>
</div>
<div class="section" id="how-can-i-create-dags-dynamically">
<h2>How can I create DAGs dynamically?<a class="headerlink" href="#how-can-i-create-dags-dynamically" title="Permalink to this headline">¶</a></h2>
@@ -295,7 +298,7 @@ LocalExecutor, that translates into running it in a subprocess pool.</li>
command (described below) as a subprocess and is in charge of
emitting heartbeats, listening for external kill signals
and ensures some cleanup takes place if the subprocess fails</li>
-<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and
+<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and
performs the actual work</li>
</ul>
</div>
[02/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/searchindex.js
----------------------------------------------------------------------
diff --git a/searchindex.js b/searchindex.js
index 121f5a8..03bf994 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({docnames:["api","cli","code","concepts","configuration","faq","index","installation","integration","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],envversion:50,filenames:["api.rst","cli.rst","code.rst","concepts.rst","configuration.rst","faq.rst","index.rst","installation.rst","integration.rst","license.rst","plugins.rst","profiling.rst","project.rst","scheduler.rst","security.rst","start.rst","tutorial.rst","ui.rst"],objects:{"airflow.contrib":{hooks:[2,1,0,"-"],operators:[2,1,0,"-"]},"airflow.contrib.executors.mesos_executor":{MesosExecutor:[2,0,1,""]},"airflow.contrib.hooks":{FTPHook:[2,0,1,""],SSHHook:[2,0,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[2,2,1,""],create_directory:[2,2,1,""],delete_directory:[2,2,1,""],delete_file:[2,2,1,""],describe_directory:[2,2,1,""],get_conn:[2,2,1,""],list_directory:[2,2,1,""],rename:[2,2,1,""],retrieve_file:[2,2,1,""],store_file:[2,2,1,""]},"airflow.contrib.hooks.SSHHook":{Pop
en:[2,2,1,""],check_output:[2,2,1,""],tunnel:[2,2,1,""]},"airflow.contrib.operators":{SSHExecuteOperator:[2,0,1,""]},"airflow.contrib.operators.hipchat_operator":{HipChatAPIOperator:[2,0,1,""],HipChatAPISendRoomNotificationOperator:[2,0,1,""]},"airflow.executors":{LocalExecutor:[2,0,1,""],SequentialExecutor:[2,0,1,""]},"airflow.hooks":{DbApiHook:[2,0,1,""],HttpHook:[2,0,1,""],MySqlHook:[2,0,1,""],PrestoHook:[2,0,1,""],SqliteHook:[2,0,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[2,2,1,""],bulk_load:[2,2,1,""],get_conn:[2,2,1,""],get_cursor:[2,2,1,""],get_first:[2,2,1,""],get_pandas_df:[2,2,1,""],get_records:[2,2,1,""],insert_rows:[2,2,1,""],run:[2,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[2,2,1,""],run:[2,2,1,""],run_and_check:[2,2,1,""]},"airflow.hooks.MySqlHook":{bulk_load:[2,2,1,""],get_conn:[2,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[2,2,1,""],get_first:[2,2,1,""],get_pandas_df:[2,2,1,""],get_records:[2,2,1,""],run:[2,2,1,""]},"airflow.hooks.SqliteHook":{get_conn:[2,2,1,
""]},"airflow.macros":{ds_add:[2,3,1,""],ds_format:[2,3,1,""],hive:[2,1,0,"-"],random:[2,3,1,""]},"airflow.macros.hive":{closest_ds_partition:[2,3,1,""],max_partition:[2,3,1,""]},"airflow.models":{BaseOperator:[2,0,1,""],Connection:[2,0,1,""],DAG:[2,0,1,""],DagBag:[2,0,1,""],TaskInstance:[2,0,1,""]},"airflow.models.BaseOperator":{clear:[2,2,1,""],dag:[2,4,1,""],deps:[2,4,1,""],detect_downstream_cycle:[2,2,1,""],downstream_list:[2,4,1,""],execute:[2,2,1,""],get_direct_relatives:[2,2,1,""],get_flat_relatives:[2,2,1,""],get_task_instances:[2,2,1,""],has_dag:[2,2,1,""],on_kill:[2,2,1,""],post_execute:[2,2,1,""],pre_execute:[2,2,1,""],prepare_template:[2,2,1,""],render_template:[2,2,1,""],render_template_from_field:[2,2,1,""],run:[2,2,1,""],schedule_interval:[2,4,1,""],set_downstream:[2,2,1,""],set_upstream:[2,2,1,""],upstream_list:[2,4,1,""],xcom_pull:[2,2,1,""],xcom_push:[2,2,1,""]},"airflow.models.Connection":{extra_dejson:[2,4,1,""]},"airflow.models.DAG":{add_task:[2,2,1,""],add_task
s:[2,2,1,""],clear:[2,2,1,""],cli:[2,2,1,""],concurrency_reached:[2,4,1,""],crawl_for_tasks:[2,2,1,""],create_dagrun:[2,2,1,""],deactivate_stale_dags:[2,5,1,""],deactivate_unknown_dags:[2,5,1,""],filepath:[2,4,1,""],folder:[2,4,1,""],get_active_runs:[2,2,1,""],get_dagrun:[2,2,1,""],get_last_dagrun:[2,2,1,""],get_template_env:[2,2,1,""],is_paused:[2,4,1,""],latest_execution_date:[2,4,1,""],normalize_schedule:[2,2,1,""],run:[2,2,1,""],set_dependency:[2,2,1,""],sub_dag:[2,2,1,""],subdags:[2,4,1,""],sync_to_db:[2,5,1,""],topological_sort:[2,2,1,""],tree_view:[2,2,1,""]},"airflow.models.DagBag":{bag_dag:[2,2,1,""],collect_dags:[2,2,1,""],dagbag_report:[2,2,1,""],get_dag:[2,2,1,""],kill_zombies:[2,2,1,""],process_file:[2,2,1,""],size:[2,2,1,""]},"airflow.models.TaskInstance":{are_dependencies_met:[2,2,1,""],are_dependents_done:[2,2,1,""],clear_xcom_data:[2,2,1,""],command:[2,2,1,""],command_as_list:[2,2,1,""],current_state:[2,2,1,""],error:[2,2,1,""],generate_command:[2,5,1,""],get_dagrun
:[2,2,1,""],init_on_load:[2,2,1,""],is_premature:[2,4,1,""],key:[2,4,1,""],next_retry_datetime:[2,2,1,""],pool_full:[2,2,1,""],previous_ti:[2,4,1,""],ready_for_retry:[2,2,1,""],refresh_from_db:[2,2,1,""],run:[2,2,1,""],xcom_pull:[2,2,1,""],xcom_push:[2,2,1,""]},"airflow.operators":{BashOperator:[2,0,1,""],BranchPythonOperator:[2,0,1,""],DummyOperator:[2,0,1,""],EmailOperator:[2,0,1,""],ExternalTaskSensor:[2,0,1,""],GenericTransfer:[2,0,1,""],HdfsSensor:[2,0,1,""],HivePartitionSensor:[2,0,1,""],HttpSensor:[2,0,1,""],MetastorePartitionSensor:[2,0,1,""],MySqlOperator:[2,0,1,""],NamedHivePartitionSensor:[2,0,1,""],PrestoCheckOperator:[2,0,1,""],PrestoIntervalCheckOperator:[2,0,1,""],PrestoValueCheckOperator:[2,0,1,""],PythonOperator:[2,0,1,""],S3KeySensor:[2,0,1,""],ShortCircuitOperator:[2,0,1,""],SimpleHttpOperator:[2,0,1,""],SqlSensor:[2,0,1,""],TimeSensor:[2,0,1,""],TriggerDagRunOperator:[2,0,1,""],WebHdfsSensor:[2,0,1,""]},"airflow.operators.BashOperator":{execute:[2,2,1,""]},"airfl
ow.operators.HdfsSensor":{filter_for_filesize:[2,5,1,""],filter_for_ignored_ext:[2,5,1,""]},"airflow.operators.sensors":{BaseSensorOperator:[2,0,1,""]},airflow:{executors:[2,1,0,"-"],hooks:[2,1,0,"-"],macros:[2,1,0,"-"],models:[2,1,0,"-"],operators:[2,1,0,"-"]}},objnames:{"0":["py","class","Python class"],"1":["py","module","Python module"],"2":["py","method","Python method"],"3":["py","function","Python function"],"4":["py","attribute","Python attribute"],"5":["py","staticmethod","Python static method"]},objtypes:{"0":"py:class","1":"py:module","2":"py:method","3":"py:function","4":"py:attribute","5":"py:staticmethod"},terms:{"00am":2,"01t23":13,"10pm":3,"2am":2,"abstract":[2,3,6,10],"boolean":[2,3],"break":8,"byte":5,"case":[3,5,13,14,16],"class":[1,2,3,7,9,10,16],"default":[0,1,4,5,6,13,14,15,17],"export":[1,15],"function":[0,2,5,6,14,16],"import":[1,2,3,5,6,7,9,10,13,14],"int":2,"long":2,"new":[2,3,5,10,13,14],"return":[0,1,2,3,5,8,10],"short":2,"static":[2,6,10],"super":[2,14],
"switch":[4,14],"true":[2,3,4,5,13,14,16],"try":[2,3,4],"var":[2,4,8],"while":[2,3,5,6,9,13,14,15,16,17],AND:[2,9],AWS:[4,6],Added:3,And:14,Are:5,But:[4,8],FOR:[2,9],For:[0,2,3,4,5,7,9,12,14,16],IDs:2,LTS:4,Not:[2,3,9],One:[3,14,16],QDS:7,That:[13,16],The:[0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17],There:[2,3,5,10],These:[2,3,4,11],USE:9,Use:[1,2,4,6],Using:[2,3,6,10],Will:[2,10],With:14,__class__:3,__doc__:3,__init__:[2,3],__name__:[3,10],_cmd:4,_copying_:2,_host:14,abbrevi:3,abil:[3,14],abl:[2,4,5,14,15,16],about:[2,3,4,9,13,16],abov:[3,4,9,13,14,17],absenc:3,absolut:3,accept:[6,9],access:[1,2,3,4,10,14],access_logfil:1,access_token:17,accesskei:3,accord:14,accordingli:3,account:[2,14],achiev:14,acquir:2,across:[4,17],act:[2,3,9,16],action:[2,17],activ:[0,2,3,5,10,13,14,16],active_dag_id:2,actual:[2,3,5,16],acycl:[2,3,6],add:[1,2,3,4,5,9,14],add_task:2,added:2,addendum:9,adding:[1,13],addit:[2,4,6,9],addition:14,addprinc:14,adhoc:[1,2,6],adjust:[4,14],admin:[3,4,5,10,11,14,15],adm
in_view:10,advanc:16,advantag:14,advis:[2,9],affect:3,after:[2,3,6,13],afterward:2,against:[2,3,5,8,9,14,16],aggregate_db_message_job:3,aggregate_db_message_job_cmd:3,ago:5,agre:9,agreement:[3,9],airbnb:[3,12,16],airflow:[0,1,2,3,4,8,11,12,13,14,15,16,17],airflow__:4,airflow__core__sql_alchemy_conn:4,airflow__core__unit_test_mod:4,airflow_config:4,airflow_conn_:[3,4],airflow_conn_postgres_mast:3,airflow_hom:[4,5,10,15],airflow_set:3,airflowignor:2,airflowplugin:10,airflowtestplugin:10,alchemi:4,alert:[2,3,10],alex:8,algorithm:2,align:5,all:[1,2,3,4,6,7,9,13,14,16],all_db:7,all_don:[2,3],all_fail:[2,3],all_success:[2,3],alleg:9,alloc:2,allow:[1,2,3,4,5,6,10,11,13,14,15,16,17],allowed_st:2,allowed_team:14,alon:9,along:[5,9,16],alongsid:[2,9,13],alreadi:[1,2],alright:16,also:[2,3,4,5,7,9,10,13,14,16],alter:[2,3,4,5,14],altern:[2,13],altert:2,although:14,alwai:[2,3,17],amazon:[4,6],amount:[2,3],analys:17,analyz:3,anand:12,anchor:2,ani:[2,3,4,5,9,13,16,17],annot:9,announc:12,anomali:10,a
noth:[2,3,6,8],ansibl:4,anyon:14,anyth:[3,16],anytim:3,anywher:[2,3],aoen:12,apach:[2,9,12],api:[1,6,7,9,14],api_cli:0,api_kei:17,apikei:17,apiv2:2,appar:2,appear:[2,3,9],append:[2,4],appendix:9,appli:[2,3,7,9,13],applic:[9,10,11,14],appropri:[3,9,14],arbitrari:[3,6],architectur:6,archiv:9,are_dependencies_met:2,are_dependents_don:2,aren:[2,3],arg:2,argument:[1,2,6,13],aris:9,around:[2,6,16],arrai:[6,10],arthur:12,artwr:12,ascii:2,asf:6,assert:9,asset:2,assign:[2,5,6],associ:[2,3,9,13],assum:[2,3,4,9,14,16],asterisk:14,async:7,atom:[3,13],attach:[2,3,9],attach_to:2,attack:14,attempt:[1,2],attent:[2,5],attr:2,attribut:[1,2,3,9],audit:10,auth:[0,2,14],auth_backend:[0,14],authen:14,authent:[2,3,6,7],author:[2,4,6,9,14,16,17],authorship:9,auto:5,autocommit:2,automat:[2,3,4,14],autoscalingalgorithm:8,avail:[0,2,3,4,9,10,11],averag:2,avoid:3,awai:2,awar:[3,14],azkaban:6,back:[2,5,10],backend:[0,2,6,7,14,15],backfil:[1,2,3,5,6,15],background:[2,16],backoff:2,backup:4,bag:2,bag_dag:2,balanc
:3,bar:[2,3],bare:16,base64:5,base:[2,3,4,6,9,10,13,14],base_dag:2,base_executor:[2,10],base_hook:[2,10],base_url:[2,14],basedag:2,basedagbag:2,basedn:14,baseexecutor:[2,10],basehook:[2,3,10],baseoper:[3,6,10,16],basesensoroper:6,baseview:10,bash:[2,3,16],bash_1:3,bash_command:[2,3,16],bash_command_to_run:4,bash_oper:[2,16],bash_queu:16,bashoper:[2,3,16],basi:9,basic:[3,5,7,8,10,11,16],bcc:2,bcrypt:14,beauchemin:12,becaus:[2,3],becom:[2,3,6,10,16],been:[2,3,4,9,13,14],befor:[1,2,3,4,5,8,14],begin:[3,13],behalf:9,behav:[2,5],behavior:[2,3,13],behind:[2,4,7,13],being:[2,3,11],belong:14,below:[2,5,9,10,15,16],benefici:9,best:2,beta:8,better:[5,16],between:[2,3,8,13,16],beyond:[2,5],bigqueri:6,bigqueryhook:6,bin:3,bind:9,bind_password:14,bind_us:14,bit:4,bitshift:6,blob:16,block:[2,3,16,17],blog:[2,9],blueprint:10,bodi:2,boilerpl:9,bolk:12,bolkedebruin:12,bonu:2,bool:2,boot:4,both:[2,5],bother:16,bottleneck:17,box:[2,4,15],bracket:[9,16],branch:[2,6],branch_fals:3,branchpythonoper:[2,3]
,broker:[1,4],broker_api:1,broker_url:4,brought:12,brows:[3,5],browser:14,bruin:12,bucket:[2,4,8],bucket_kei:2,bucket_nam:2,buffer:2,bug:4,build:[3,4,6,11],built:[2,3,4,6,10,11,16],bulk:[5,17],bulk_dump:2,bulk_load:2,bump:3,bundl:2,burn:1,but_this_dag_will_not:3,cacert:14,cach:14,cal:8,call:[2,3,4,5,13,16],callabl:2,callback:14,can:[0,2,3,4,6,7,8,10,11,13,14,15,16,17],cannot:[2,3,9,16],cap_net_bind_servic:14,capabl:[10,13],capac:3,card:2,care:[1,4],carri:[3,9],cascad:3,cast:2,catchup:[2,6],catchup_by_default:13,categori:10,caus:[5,9],celeri:[1,3,5,6,7,14],celery_result_backend:4,celeryexecutor:[2,3,4,5,7,13],cell:2,central1:8,central:3,cert:14,certain:[2,3,8],certif:[1,14],cfg:[1,2,3,4,5,13,14,15,16],cfg_path:[1,2],chain:3,chanc:4,chang:[0,4,6,8,9,13,14,15,16],channel:12,charact:9,character:3,charg:[5,9],charset:2,chart:[6,10],chat:12,check:[1,2,3,5,8],check_oper:2,check_output:2,checkoper:2,chef:4,child:3,child_dag:3,child_dag_nam:3,chmod:14,choic:[1,2,16],choos:[2,3,9,14],chri:12,
circuit:2,claim:9,clariti:6,claus:2,clean:2,cleanup:[2,5],clear:[1,2,3,5,13,17],clear_xcom_data:2,cli:[2,3,4,5,6,13],click:[5,14,16,17],client:[0,2,7,14],client_id:14,client_secret:14,close:[2,5,14],close_conn:2,closest:2,closest_ds_partit:2,cloud:[4,6,7],cloudant:7,cluster:[4,6,8],cmd:2,code:[1,2,3,4,6,8,9,10,16],collabor:6,collect:[2,3,10],collect_dag:2,color:2,column:2,com:[2,3,14,16],combin:[3,9],come:[2,13],comma:[1,2,3],command:[2,3,4,6,7,13,15],command_as_list:2,comment:9,commerci:9,commit:[2,12,14],commit_everi:2,committ:6,common:[3,4,5,9,16],commonli:2,commun:[3,6,9,16],compani:10,compar:6,comparison:2,compil:[5,9],complet:[2,3,4,6,13],complex:[2,3,6,10],compli:9,complianc:9,complic:16,compon:10,compos:[3,14,16],composit:6,comprehens:17,comput:[2,9],concept:[4,5,6,13,16],conceptu:3,concern:3,concur:5,concurr:[1,2,5],concurrency_reach:2,condit:[2,3,7,9],conf:[1,2,4],config:[1,10,14],configur:[0,1,2,3,6,9,13,14,15,16,17],confirm:[1,5,13],confirm_prompt:2,confus:[2,3,5],conjun
ct:[1,3,15],conn_extra:1,conn_id:[1,2,3,4],conn_typ:2,conn_uri:1,connect:[1,2,6,7,10,11,14],connect_timeout:2,connecton:11,consequenti:9,consid:[2,3,4],consist:[3,6,9],consol:14,conspicu:9,constant:[2,3],constitu:3,constitut:9,constru:9,constructor:[2,16],contact:[4,9],contain:[2,3,4,5,9,13,14,16,17],content:[2,3,9,10,12,14],context:[2,4,5,6,16],continu:[2,6],contract:9,contrib:[2,3,8,14],contribut:[6,9],contributor:[9,12],contributori:9,control:[2,3,9,17],conveni:[2,3],convent:3,convers:9,convert:2,cool:2,copi:[2,4,8,9,14],copyright:[9,14],core:[2,4,6,10,14],correct:3,correpsond:3,correspond:[2,3,4,5],could:[2,3,14,16],count:[1,2],counterclaim:9,cours:14,cover:13,cpu:[2,3,10],craft:2,crawl:[2,13],crawl_for_task:2,creat:[0,2,3,4,6,10,11,13,14,15,16,17],create_dagrun:2,create_directori:2,creation:2,credenti:14,credit:14,criccomini:12,criteria:[2,3],criterion:2,critic:[1,2,3],cron:[2,3,5,13],cross:[3,9,16],crt:14,crud:1,crypto:[4,6,7],curli:16,current:[0,2,13,17],current_st:2,cursor:2
,custom:10,customari:9,cycl:[2,3,16],daemon:[1,4,13],dag:[0,1,2,4,6,8,14,15],dag_1:3,dag_2:3,dag_fold:[2,3,4],dag_id:[0,1,2,3,5,16],dag_md:3,dag_run:[0,2],dag_run_obj:2,dag_stat:1,dagbag:[1,2,5],dagbag_report:2,dagfil:3,dagrun:[1,2,5],dagrun_timeout:2,dags_fold:[4,5],dai:[2,5,8,13,15,16],daili:[2,5,13],damag:9,dan:[2,12],data:[2,3,6,7,10,14,16,17],data_profiler_filt:14,databas:[1,2,3,4,6,7,8,10,11,15,16],dataflow:6,dataflow_default_opt:8,dataflowhook:6,datafram:2,dataproc:6,dataprocpysparkoper:6,dataset:[2,13],datastor:6,date:[1,2,3,9,13,16],date_filter_column:2,datestamp:2,datetim:[2,3,5,13,16],dateutil:2,davydov:12,days_back:[2,8],dbapi_hook:2,dbapihook:2,deactiv:2,deactivate_stale_dag:2,deactivate_unknown_dag:2,deal:6,debug:[1,2,16],decid:4,decis:6,declar:2,deep:2,def:[2,3,10],default_arg:[2,3,5,8,16],default_imperson:14,default_login:14,default_param:5,default_queu:3,defend:9,defer:3,deferred_op:3,defin:[2,3,4,5,6,9,10,13,14,16],definit:[0,2,3,6,9,10],delai:[1,2,5],delet:[1,2,3,
13,17],delete_directori:2,delete_fil:2,deliber:[2,9],delimit:[1,2,3],demonstr:3,dep:2,dep_context:2,depcontext:2,depend:[1,2,3,4,5,6,7,13,17],depends_on_past:[1,2,3,5,8,16],deploi:10,deploy:10,depth:16,deriv:[2,4,9,10],descend:3,describ:[2,3,5,9],describe_directori:2,descript:[1,2,3,9],deseri:[1,2,3],deserialize_json:3,design:[3,9,13],desir:[2,16],destin:2,destination_conn_id:2,destination_t:2,detail:[2,3,5,14,17],detect:[2,10],detect_downstream_cycl:2,determin:[2,3,9],dev:[4,7,12],devel:7,devel_hadoop:7,develop:[1,2,14,16],deviat:2,dialog:13,dict:[1,2,3],dictionari:[2,3,5,16],did:16,die:3,differ:[2,3,5,8,9,10,16,17],direct:[2,3,4,6,9],directli:[2,3,5,11],directori:[1,2,3,4,14,16],dirti:15,disclaim:[6,9],discuss:9,displai:[4,9,13,16],disregard:[2,16],distinct:2,distribut:[2,3,7,9],do_pickl:1,doc:[2,3,5,16],doc_json:3,doc_md:3,doc_rst:3,doc_yaml:3,dockeroper:3,document:[2,4,5,9,12,14,16],doe:[1,2,3,4,5,6,7,9,14,16],doesn:[1,2,3,16],domain:14,domainnam:0,don:[2,3,4,7,9,13,14],done:[2,
3,5,13,14,16],donot_pickl:[1,2],door:3,dot:[2,3],doubl:[2,4,16],down:[1,3],download:[4,8],downstream:[1,2,3,13],downstream_list:2,downstream_task_id:2,drive:4,driven:10,driver:4,drop:[2,10],druid:7,dry:1,dry_run:[1,2],ds_add:[2,16],ds_format:2,ds_nodash:2,dttm:2,dubiou:2,due:4,dummi:[2,3,10],dummy_1:3,dummy_oper:3,dummy_task:3,dummyoper:[2,3],dump:2,durat:[1,6],dure:[3,4,17],dynam:[2,3,6],each:[2,3,5,9,12,13,16],eagerli:4,earlier:16,easi:[3,4,6,10,11,14,17],easier:[2,9],easiest:7,easili:[2,3,5,6,11,13,16],echo:[3,16],ecosystem:10,edit:[4,17],editori:9,effect:[2,3],effort:6,egg:4,either:[2,3,4,9,13,14,15],elabor:9,elaps:2,elast:2,electron:9,eleg:6,element:4,els:15,elsewher:3,email:[2,3,8,12,14,16],email_on_failur:[2,8,16],email_on_retri:[2,8,16],emailoper:[2,3],emit:[3,5,16],empti:[2,3],enabl:[0,2,3,4,6,7],enclos:9,encod:5,encrypt:[4,6,7,14],encrypt_s3_log:4,end:[2,9,13],end_dat:[1,2,13,16],endors:6,endpoint:[2,6],endpoint_url:0,enforc:[3,5],engin:[2,5,6],ensur:[3,5],enterpris:6,enti
r:[1,3,5],entiti:9,entri:[2,13,14],env:[2,3,4],environ:[2,3,4,6,7,11,13,16,17],ep_data_pipeline_db_msg_agg:3,equival:[3,7,16],era:5,error:[1,2],error_logfil:1,especi:[3,5,16],essenti:2,etc:[2,3,4,14],eval:2,evalu:[2,3,5,13,16],even:[2,3,9,11],event:[3,9,16],eventlet:1,ever:2,everi:[2,3,13,14,16],everyon:16,everyth:[16,17],exactli:[2,17],examin:13,exampl:[2,3,4,6,9,13,14,15],example1:15,example_bash_oper:15,example_dag:[3,16],except:[2,3,5,9,14,16],exchang:[3,6],exclud:[1,9],exclude_subdag:1,exclus:[9,13],exec_d:1,execut:[1,2,3,4,5,6,8,9,13,16],execution_d:[1,2,3,13,16],execution_date_fn:2,execution_delta:2,execution_tim:3,execution_timeout:[2,3],executor:[1,3,4,5,6,10,13],exercis:9,exhaust:4,exist:[0,1,2,3,4,5,14,16],exit:[1,14],expans:4,expect:[0,2,3,5,6],experiment:6,expir:2,expiration_d:2,explan:16,explicit:6,explicit_op:3,explicitli:[2,3,9,14,16],exponenti:2,expos:[0,2,3,4,10,14],express:[2,5,8,9,13],ext:2,extend:[2,6],extens:[2,6,8],extent:2,extern:[2,3,5,6,10],external_dag_id:
2,external_task_id:2,external_trigg:2,externaltasksensor:2,extra:[1,2,3,6,14],extra_dejson:2,extra_opt:2,fact:3,factori:3,fail:[1,2,3,4,5,13,16,17],failur:[2,4,6,9],fairli:15,fals:[1,2,4,8,13,16],familiar:[3,16],faq:[2,6],fast:15,fault:[2,3],featur:[2,3,4,7,10,16,17],feb:14,fee:9,fernet:5,fernet_kei:5,fetch:4,few:[2,4,11,15,16],field:[1,2,9],fifti:9,figur:[2,3,13],file:[0,1,2,3,4,6,8,9,10,13,14,15],file_path:2,file_s:2,filenam:[2,3],filepath:[1,2],files:2,filesystem:4,fill:[2,3,11,14],filter:[1,2,3,14],filter_by_own:14,filter_for_files:2,filter_for_ignored_ext:2,find:[2,4,5,16,17],fine:16,finish:2,fire:[2,3,4,5],first:[1,2,3,4,5,12,13,15,16],fit:[2,6,9],fix:[2,4,13],flag:[2,13],flask:[1,10,11],flask_admin:10,flask_blueprint:10,flask_login:14,flat:2,flavor:10,flexibl:[2,16],flower:[1,4],flower_conf:1,folder:[2,4,10,13,14,15,16],follow:[0,2,3,4,6,9,14,16],foo:[2,3],foo_:5,foo_dag_id:5,foobar:4,forc:[1,2],foreground:1,foreign:2,form:[2,3,6,9],format:[2,3,4,5,9],forward:[2,5],found:[2,4
,14],foundat:[6,9,12,15],four:3,framework:[4,10],free:[3,9],frm:2,from:[1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17],from_nam:2,ftp:2,ftp_conn_id:2,ftp_default:2,ftphook:2,full:[2,12],full_filepath:2,fulli:[0,2,5,6,14],functiontyp:2,fundament:16,further:[6,15,16],furthermor:14,futur:[2,13],galaxi:10,gantt:6,garag:3,gate:14,gcp:6,gcp_api:7,gcp_conn_id:8,gcp_dataflow_bas:8,gcp_default:8,gener:[1,2,3,6,9,10,14,16,17],generate_command:2,generictransf:2,get:[0,1,2,3,4,6,10,11,13,15,16,17],get_active_run:2,get_conn:2,get_connect:3,get_cursor:2,get_dag:2,get_dagrun:2,get_direct_rel:2,get_first:2,get_flat_rel:2,get_last_dagrun:2,get_pandas_df:2,get_partitions_by_filt:2,get_partitions_by_nam:2,get_record:2,get_task_inst:2,get_template_context:2,get_template_env:2,gevent:1,ghe:6,ghe_oauth:14,ghost:2,git:4,github:[6,9,12,16],github_enterpris:14,github_enterprise_auth:14,gitignor:2,gitter:12,give:[3,9],given:[0,2,3,8],glanc:17,global:[2,3,5],goal:16,goe:[2,5,16],going:15,good:3,goodby:3,goodwil:9,g
oogl:[4,6,7],google_auth:14,google_client_id:14,google_client_secret:14,googlecloudstoragehook:6,govern:9,grab:2,grade:4,grai:2,grant:9,graph:[2,3,6],great:[3,4,13],greater:2,green:2,grossli:9,group:[2,4,14],gunicorn:[1,7],had:[2,5,13],hadoop:[7,8,14],hand:[14,15],handl:[2,3,4,13],happen:[3,13,16],hard:[2,3],harmless:9,has:[1,2,3,4,5,6,8,9,10,13,14,16],has_dag:2,hash:14,hasn:[3,13],have:[2,3,4,5,6,7,8,9,10,11,13,14,16],haven:[2,3,16],hdf:[2,3,7],hdfs_conn_id:2,hdfs_default:2,hdfs_hook:2,hdfshook:2,hdfssensor:2,head:[15,16],header:2,heartbeat:[2,3,5],heavili:2,hello:[3,10],help:[1,9,10,14],here:[2,3,4,5,7,10,13,14,15,16,17],herebi:9,herein:9,hidden:17,hierarchi:16,high:2,highchart:11,higher:2,highli:4,highlight:16,hipchat:2,hipchat_oper:2,hipchatapioper:2,hipchatapisendroomnotificationoper:2,histori:6,hit:[2,3],hive2mysqloper:2,hive2sambaoper:7,hive:[2,3,4,7,8,10,14],hive_conn_id:2,hiveoper:[3,4,14],hivepartitionsensor:[2,3],hivesensoroper:2,hold:9,holder:0,home:15,homogen:4,hook:[6,
7,8,10,14,16],hope:13,horribl:16,host:[2,4,14],hostnam:[1,2,3,4,14],hour:[2,3,5,13],hourli:[2,5,13],hous:3,how:[2,3,4,6,9,10,13,16,17],howev:[3,4,9,14],hql:2,html:[2,10,16],html_content:2,http:[0,2,3,9,10,14,16],http_conn_id:2,http_default:2,httphook:2,httpoper:3,httpsensor:2,human:2,idea:[2,4,5,6],idempot:13,identif:9,identifi:[1,2,9,16,17],ids:2,ignor:[1,2],ignore_:2,ignore_all_dep:2,ignore_all_depend:1,ignore_copi:2,ignore_depend:1,ignore_depends_on_past:[1,2],ignore_first_depends_on_past:[1,2],ignore_task_dep:2,ignore_task_depsstore_tru:1,ignore_ti_st:[1,2],ignored_ext:2,iii:9,immedi:[2,3],imper:4,imperson:[2,6],implement:[2,3],impli:9,improv:9,inabl:9,inact:[2,5],inc:9,incident:9,includ:[1,2,3,4,9,13,16],include_adhoc:[1,2],include_downstream:2,include_exampl:2,include_prior_d:2,include_subdag:2,include_upstream:2,inclus:9,incorpor:9,incub:12,incur:9,indemn:9,indemnifi:9,independ:[2,3],index:2,indic:[2,3,6,9],indirect:9,individu:[2,9,13,16],ineffici:2,infer:[2,3],inferred_op:3,
infin:6,info:[0,1,2],inform:[2,3,4,9,13,14,16],infrastructur:6,infring:9,ingress:8,inherit:[2,16],init:4,init_on_load:2,initctl:4,initdb:[1,4,15],initi:[1,2,4,14,15],initialis:2,inject:10,input:2,input_format:2,insecur:14,insert:[2,3],insert_row:2,insid:3,inspect:[13,15],inspir:2,instal:[2,3,4,6,10,14,15],instanc:[1,2,4,5,6,7,13,14,15,16],instanti:[2,3,5,6,13],instead:[1,2,5,13],institut:9,instruct:[2,3,13],insur:2,integr:[2,6,7,10,14],intention:9,inter:3,interact:[2,3,4,10,11],interest:[2,16],interfac:[3,6,9,11,14,16],intergr:10,intern:[1,13],interpret:[2,3],interv:[2,5,13],intervalcheckoper:2,introduc:5,intuit:16,invari:[2,3],invok:2,involv:3,ipython:3,irrevoc:9,is_paus:2,is_prematur:2,ish:2,isn:[3,6],isoformat:2,issu:[2,3,6,9,12],item:[3,13],iter:2,its:[2,3,4,9,10],itself:[3,5,13,14,16],januari:[9,13],jar:8,jdbc:7,jeremiah:12,jinja2:2,jinja:[2,6,10],jinja_env:2,jira:12,jlowin:12,job:[1,2,3,4,5,8,13,15,17],job_id:2,join:[3,12],json:[1,2,3],json_client:0,june:12,jupo:2,jupyt:3,just
:[1,2,3,5,10,13,16],kadmin:14,kapil:4,keep:[2,3,6,16],kei:[1,2,3,4,5,13,14,17],kerber:[7,14],kerbero:[0,1,6,7],kernel:2,key1:2,key_fil:2,keytab:[0,1,14],keyword:2,kick:[4,13],kill:[2,3,5],kill_zombi:2,kind:9,know:[4,5,16],knowledg:10,known:[2,7],known_host:2,kruyk:12,kwarg:2,kwd:2,label:[2,11],lai:15,lambda:[2,3],land:[2,10],languag:[9,16],last:2,late:17,later:11,latest:[1,2,6,7,16],latest_d:2,latest_execution_d:2,latest_onli:3,latest_only_oper:3,latest_only_with_trigg:3,latestonlyoper:3,launch:14,law:9,lawsuit:9,layer:5,layout:16,ldap3:14,ldap:[6,7],ldap_auth:14,ldap_ca:14,lead:[2,3],lean:6,least:[2,3,9],leav:2,led:3,left:[3,14],legaci:5,legal:9,leo:12,less:2,let:[1,2,3,4,11,13,16,17],level:[2,3,6,13,14],leverag:[2,3,16],liabil:9,liabl:9,lib:[2,3,4],libffi:4,librari:[2,3,4,5,6,10,16],libz:3,licens:[6,14],licensor:9,life:3,lifetim:13,light:[2,3],lightweight:3,like:[2,3,4,5,7,10,14,16],limit:[2,3,4,6,9,13,15],line:[2,6,13,15],link:[3,6,9,10],linux:14,list:[1,2,3,4,6,7,9,10,12,14,16,1
7],list_dag:[1,5,16],list_directori:2,list_task:[1,5,16],listen:[3,5,14],lit:2,liter:2,litig:9,live:2,load:[1,2,3,4,8],load_test_config:4,local:[1,2,3,4,5,14,16],local_executor:2,local_full_path_or_buff:2,local_port:2,localexecutor:[1,2,3,4,5,13],localhost:3,locat:[1,2,3,4,14,16],lock:2,lock_for_upd:2,log:[1,2,6,10,14,16],log_fil:1,logfil:1,loggin:2,loggingmixin:2,logic:[2,3,16],login:[2,3,4,14],logist:10,longer:[2,5],look:[1,2,3,5,6,10,12,13,14,16],loss:9,lot:13,lowin:12,machin:[2,3,4,14],macro:[3,5,6,10,11,16],made:[2,3,9],mai:[2,3,4,5,9,13,16],mail:[9,12],main:[2,3,5,10],main_dag:3,maintain:6,make:[0,2,3,4,5,6,9,14,16,17],malfunct:9,malik:4,man:7,manag:[2,6,9,10,14],mani:[1,2,3,5,10,16,17],manner:6,manual:[2,3,4,13],map:2,march:12,mark:[1,2,3,9,13],mark_success:[1,2,13],markdown:3,markup:2,master:[3,16],match:[1,2,3],matter:2,max:[2,12],max_active_run:[2,5],max_partit:2,max_retry_delai:2,maxim:12,maxime_beauchemin:1,maximum:2,maxnumwork:8,mayb:3,mean:[2,3,4,5,8,9,14,16],meaning:2
,meant:[2,5],mechan:[2,9,14],media:9,medium:9,meet:[2,9],member:14,memberof:14,memori:2,menu:[3,5,6,10,15],menu_link:10,menulink:10,merchant:9,mere:9,merg:16,meso:[2,6],mesos_executor:2,mesosexecutor:[2,4,13],messag:[2,3,6],message_format:2,met:[2,4,5,13],metadata:[1,2,3,4,6,7,10,13,17],metastor:2,metastore_conn_id:2,metastore_default:2,metastore_mysql:2,metastorepartitionsensor:2,method:[2,3,5],metric:[2,8,10],metrics_threshold:2,microsoft:7,midnight:[5,13],might:[3,5,13,16],migrat:14,mime_subtyp:2,min:5,mind:[3,6],minimum:[7,13],minor:8,minut:[3,5,8,13,16],mismatch:3,miss:[2,3],mistercrunch:12,mix:[2,3,16],mkdir:3,mlsd:2,mode:[1,2,3,6,16],model:[3,6,10,14,16],modif:9,modifi:[2,3,4,9,11],modul:[2,3,5,6,10,14],modular:6,moment:5,monitor:[3,4,6,10,13,17],monospac:3,month:[5,13],monthli:[5,13],more:[2,3,4,6,9,10,12,13,14,16,17],morn:13,most:[2,3,8,13,14,16,17],mostli:[2,6,13],mount:4,move:[2,3,5,6],msg:2,mssql:7,much:[2,3,13],multi:6,multipl:[2,3],multiprocess:2,must:[2,3,4,5,9,14,16]
,mutat:3,my_conn_str:4,my_dag1:3,my_dag2:3,my_dag:3,my_databas:2,my_dict_var:2,my_funct:3,my_param:16,my_tabl:2,my_useful_packag:3,my_var:2,mypackag:14,mys3conn:4,mysql:[2,3,4,7],mysql_conn_id:2,mysql_default:2,mysqldb:2,mysqlhook:2,mysqloper:[2,4],name:[1,2,3,4,9,10,14],namedhivepartitionsensor:2,namespac:[2,5,10],nav:14,navig:14,ndfor:16,necessarili:[6,9],need:[2,3,4,5,6,7,10,13,14,15,16],neg:[2,13],neglig:9,neighbour:2,nest:[2,16],network:14,never:[1,2,5],new_user_email:14,new_user_nam:14,newli:6,newobject:2,next:[2,3,6,13,14],next_execution_d:2,next_retry_datetim:2,nice:3,night:3,nlst:2,no_confirm:1,no_host_key_check:2,node:[1,2,3],non:[1,2,4,9],none:[2,3,10,13],noram:14,norandkei:14,normal:[2,3,8,9],normalize_schedul:2,notat:2,note:[0,2,4,5,6,8,13,14,16],notebook:3,noth:[2,9],notic:[3,9,16],notif:2,notifi:2,notwithstand:9,now:[2,3,4,5,13,14,16],nuanc:3,num:2,num_run:1,number:[1,2,3,4,6],oauth2:14,oauth2callback:14,oauth:6,oauth_callback_rout:14,oauth_key_from_github_enterpris:1
4,oauth_secret_from_github_enterpris:14,obj:2,object:[1,2,3,4,5,9,10,13,16],objectclass:14,oblig:9,observ:2,obtain:9,occur:[2,3],octob:12,off:[2,4,13],offer:[9,10],offici:[2,12],offset:2,often:[2,3],on_failure_callback:2,on_kil:2,on_retry_callback:2,on_success_callback:2,onboard:5,onc:[2,3,4,5,13,14,16],one:[2,3,4,5,6,9,13,14,16],one_fail:[2,3],one_success:[2,3],ones:[2,5,8,17],onli:[0,1,2,4,5,6,7,9,13,14,15],only_fail:[1,2],only_if_upd:2,only_run:[1,2],oozi:6,op1:3,op2:3,op3:3,op4:3,op_arg:2,op_kwarg:2,open:[2,3,5,12,14],oper:[1,4,5,6,7,10,16],oppos:[2,3],optim:2,option:[1,2,3,6,8,11,13,14,15,16],orchestr:[2,6],order:[2,3,4,13,14],org:[2,9,10,12,16],organ:[3,10],organiz:14,orient:2,origin:9,orm:2,other:[1,2,3,4,6,9,11,13,16],otherwis:[2,9,16],our:16,out:[2,3,5,6,9,13,15],outag:10,outgrow:15,outlier:17,output:[2,16],output_encod:2,output_format:2,outsid:13,outstand:9,over:[1,2,3,4,15,17],overlap:17,overrid:[1,2,5,16],overridden:[2,5],overriden:2,overview:17,overwhelm:3,overwrit:4,ow
n:[2,3,4,6,9,13,16],owner:[2,3,8,9,14,16],ownership:9,packag:[2,4,6],package1:3,page:[3,5,9,12,17],pai:[2,5],pair:[2,17],panda:2,parallel:[2,3,4,15],param:[1,2,16],paramet:[1,2,3,4,5,11,14,16],parameter:[3,6,11],parent:[2,3,5],parent_dag:[2,3],parent_dag_nam:3,parent_modul:2,pars:[2,5,10,16],part:[2,3,9,11,14],parti:9,particular:[3,9],partit:[2,3],partition_nam:2,partitiontyp:8,partli:5,pass:[2,3,5,10,16],pass_valu:2,passwd:17,password:[2,3,4,6,7,17],password_auth:14,passwordus:14,past:[2,3,5,13,17],patent:9,path:[1,2,3,4,14],patrick:12,patrickleotardif:12,pattern:[2,3],paus:[1,2,3],payload:2,peopl:[2,10,16],percent:9,perfect:3,perform:[1,2,3,4,5,6,8,9,13,17],perhap:[3,17],period:[2,3,5,13,16],permiss:[9,14],perpetu:9,persist:[2,13],perspect:[1,3],pertain:9,phase:10,physic:2,pick:[3,4,11,13],pickl:[1,2,3,4],picklabl:2,pickle_id:2,pickleabl:2,pid:[1,15],pig:[2,3,8],piggi:10,pigoper:2,pip:[3,7,15],pipelin:[2,3,4,6,8,11,17],pk1:2,pk2:2,place:[0,2,3,5,9,16],placehold:2,plain:[2,4],plan:
5,platform:[3,6,7],pleas:[0,3,4,12,14,16],plug:10,plugin:6,plugin_macro:10,pluginexecutor:10,pluginhook:10,pluginoper:10,plugins_manag:10,point:[2,3,4,5,15,16],poke:2,poke_interv:2,polici:6,polymorph:2,pool:[1,2,5,6,16],pool_descript:1,pool_ful:2,popen:2,popul:16,port:[0,1,2,4,14,15],posit:2,possibl:[1,2,3,4,9,13,14,15,16,17],post:[0,2],post_execut:2,postgr:[3,4,7],postgres_mast:3,potenti:3,power:[3,6,9,16],practic:2,pre:5,pre_execut:2,preced:[2,4,16],prefer:[9,13,15],prefix:[3,4],preoper:2,prepar:[3,9],prepare_templ:2,prepped_request:2,present:2,preset:13,presto:2,presto_conn_id:2,presto_default:2,prestocheckoper:2,prestohook:2,prestointervalcheckoper:2,prestotomysqloper:3,prestovaluecheckoper:2,pretti:16,prev_execution_d:2,prevent:[2,3,14],previou:[1,2,3,5,16],previous:[3,5],previous_ti:2,primari:2,primit:2,princip:[0,1,3,14],print:[1,2,3,9,16],print_dat:16,prior:2,priorit:3,prioriti:[2,3],priority_weight:[2,3,16],privaci:9,privileg:14,probabl:[4,5,14],problem:3,problemat:3,proce:
[2,5],process:[1,2,3,4,6,13,16],process_fil:2,product:[2,3,4,6,9,11,13,16],profil:[2,6,14],program:12,programmat:6,progress:[2,6,16],project:[6,8],promin:9,prompt:1,propag:[2,3],propaget:2,proper:16,properli:[4,5],properti:[2,3,5,14],provid:[2,3,4,9,11,14,16,17],provide_context:[2,3],proxy_us:14,proxyus:14,psycopg2:4,publicli:9,publish:2,pull:[2,3],pull_funct:3,puppet:4,pure:3,purpl:2,purpos:[9,16],push:[2,3,10],push_funct:3,pushing_task:3,put:[2,3,5,10],pv1:2,pv2:2,pwd:3,pyhiv:2,pypi:[7,15],pyspark:8,python:[2,3,4,5,6,7,10,14,16],python_1:3,python_cal:[2,3],python_oper:2,pythonhost:10,pythonoper:[2,3],pythonpath:[4,14],qds:7,qualif:14,qualifi:[0,2,14],qualiti:[2,3],qubol:7,queri:[2,4,6,8],queu:[1,3,5],queue:[1,2,5,6,16],quick:[4,6,17],quickli:[11,15,16,17],quit:5,r39132:12,rabbitmq:[4,7],rais:[2,16],ran:[1,2],randkei:14,random:2,randomli:3,rang:[1,2,3,5,13,16],rather:3,ratio:2,raw:[2,5],reach:[2,3,4,5],reactiv:5,read:[2,4,5,14,16],readabl:[2,9,14],readi:[2,6,15],readm:4,readthedoc:
16,ready_for_retri:2,real:4,realli:[2,14,16],realm:[0,14],reason:[5,9,16],reboot:3,rebuild:1,recap:6,receiv:[2,3,9,13],recent:[2,3],recipi:[2,9],recogn:2,recommend:[2,4,5,9],record:[1,2,3,16],recreat:2,recurs:2,red:2,redhat:4,redi:4,redirect:[1,14],redistribut:9,redund:16,refer:[3,4,6,10,12,16],referenc:[3,16],reflect:[3,6,10,16],refrain:3,refresh:2,refresh_from_db:2,regard:[2,9],regardless:[1,2],regex:[1,2],regexp:1,regist:[10,11,14],regular:13,reinit_frequ:14,rel:[2,13,16],relat:[2,3,4,5,7,12],relationship:3,relativedelta:2,releas:8,relev:5,reli:[2,14],remain:[9,14],rememb:[2,16],remot:[2,4,5,13],remote_base_log_fold:4,remote_full_path:2,remote_host:2,remote_log_conn_id:4,remote_port:2,remov:2,renam:2,render:[1,2,3,10,16],render_templ:2,render_template_from_field:2,renew:[1,14],repeat:[3,13],repl:14,replac:[2,9,14],report:[1,2],repositori:[4,16],repres:[2,3,5,9,13],represent:[2,17],reproduc:9,reproduct:9,request:[1,2,3],requir:[1,2,3,4,5,6,7,9,14],rerun:1,resembl:3,reset:1,reset_d
ag_run:2,resetdb:1,resid:3,resolv:2,resourc:[1,2,3,6],respect:[1,2,3,16],respons:[2,9],response_check:2,rest:[2,6],restart:[3,4,5],restrict:14,restructuredtext:3,result:[2,3,9,11,16],resum:1,retain:9,retri:[1,2,3,8,16],retriev:3,retrieve_fil:2,retry_delai:[2,8,16],retry_exponential_backoff:2,return_valu:2,reus:10,revers:4,review:6,revis:9,riccomini:12,rich:[1,3,6,17],right:[2,3,9,11],rightmost:3,risk:9,roadmap:6,role:4,roll:6,room:2,room_id:2,root:[2,3,14],root_dag:2,round:[2,5],rout:14,routin:[2,3],row:[2,3,8],royalti:9,rule:[2,5,6,16],run:[1,2,4,6,14,15,17],run_and_check:2,run_as_own:14,run_as_us:[2,14],run_dur:1,run_id:[1,2,13],runme_0:15,runnabl:3,rwc:14,s3_conn_id:2,s3_default:2,s3filetransferoper:3,s3hook:4,s3keysensor:[2,7],s3prefixsensor:7,safe_mod:2,sai:[2,3,5,13],said:3,samaccountnam:14,samba:7,same:[2,3,4,9,11,14],sampl:4,sandbox:4,save:[2,4,5,11,16],scalabl:6,scale:6,scan:[2,3],scene:[7,13],schedul:[1,2,3,4,6,16],schedule_interv:[2,3,5,13,16],scheduler_interv:5,scheduler
_run:4,schema:[2,3,4],scope:6,scratch:16,screen:16,screenshot:6,script:[2,3,4,5,6],search:[2,3],search_path:4,search_scop:14,second:[1,2,3,5,16],secret:[14,17],secretkei:3,section:[2,3,4,5,8,9,14,15,16],secur:[2,3,6,9],see:[2,3,4,5,9,13,14,15,16,17],seen:[4,17],select:[2,3,11,13,14],self:[2,10],sell:9,semicolon:2,send:[1,2,3,12],send_room_notif:2,sender:2,sensor:[2,3,5],sensor_queu:3,sent:[2,3,4,9],separ:[3,9,16],sequenti:[2,13,15],sequentialexecutor:[2,3,15],seri:[3,13],serial:[1,2],serv:[1,4,16],serve_log:[1,4],server:[1,2,4,10,14,15,16],servic:[0,1,2,3,4,7,9,13,14],session:[2,14],set:[0,1,2,3,5,6,10,13,17],set_depend:[2,16],set_downstream:[2,3,16],set_the_password:14,set_upstream:[2,3,16],setgid:4,setuid:4,setup:[4,14],sever:3,shall:[2,9],shape:13,share:[2,3,4,9,10,11],shell:[2,14],ship:[1,2],ship_dag:1,shop:9,shortcircuitoper:2,shortcut:[4,17],should:[2,3,4,5,9,10,13,15,16],shouldn:[2,3],show:[1,2,3,5,10,17],shown:2,sid:12,siddharth:12,side:[2,4],signal:5,similar:[3,4,6],simpl:[
2,3,5,8,10,11,14,16],simplehttpoper:2,simpler:16,simplest:14,simpli:[2,3,10,13,14,16],simplifi:2,simul:16,simultan:3,sinc:[2,3,4,15,16],singl:[1,2,3,8,13,16],site:14,situat:3,size:[2,3],skew:10,skip:[1,2,3],sla:[2,6,10],sla_miss_callback:2,slack:7,slackapipostoper:7,slackoper:3,slave:4,sleep:16,slightli:6,slot:[1,2,3],slot_count:1,slowli:6,slug:14,small:[2,3],snakebit:2,snap:6,soft_fail:2,softwar:[6,9,12],sole:9,solut:6,some:[0,2,3,4,5,13,16,17],somehow:4,someth:[3,14,16],sometim:[2,3,16],somewhat:16,somewher:15,soon:[2,3,4,5,13],sort:[2,3],sound:[3,16],sourc:[2,3,9,11,12,17],source_conn_id:2,space:6,span:17,spark:[3,6,8],spawn:4,special:[2,3,5,9],specif:[1,2,3,4,5,8,9,13,16,17],specifi:[1,2,3,4,5,6,13,14,16],specific_valu:2,spent:17,split:13,sponsor:6,sql:[2,3,4,7,8,11],sql_alchemy_conn:4,sql_alchemy_conn_cmd:4,sqlachemi:2,sqlalchemi:[2,3,4],sqlite:[2,15],sqlitehook:2,sqloper:3,sqlsensor:2,sscursor:2,ssh:[2,14],ssh_default:2,ssh_hook:2,sshexecuteoper:2,sshhook:2,sshpass:2,ssl:[1,2,
6],ssl_cert:1,ssl_kei:1,stabil:6,stabl:7,stack:[7,10],stage:[3,8],stagingloc:8,stai:13,stamp:13,stand:3,standalon:3,standard:[2,3,5,14],stanza:4,start:[1,2,3,4,5,6,7,8,12,13,16],start_dat:[1,2,3,6,8,13,16],stat:2,state:[1,2,3,5,9,13,16],statement:[2,3,9],static_babynam:2,static_babynames_partit:2,static_fold:10,static_url_path:10,statu:[1,2,3,4,5,6,8,9,13,15,16,17],stderr:1,stdout:[1,2,16],step:[4,16,17],steven:12,still:[2,6,14],sting:2,stop:[2,4],stoppag:9,storag:[2,4,6],store:[1,2,3,4,10,14,15],store_fil:2,storm:6,str:[2,10,13],straightforward:15,stream:[2,6],string:[1,2,3,4,16],structur:[2,6,16],style:2,sub:[1,2],sub_dag:[2,3],sub_part:2,subclass:2,subcommand:[1,2,4,13,16],subdag:[1,2,6],subdag_oper:3,subdagoper:[2,3],subdir:1,subdirectori:3,subject:[2,9],sublicens:9,submiss:[9,13],submit:9,subpackag:[7,14],subpartit:2,subprocess:[2,5,13],subscrib:[3,12],subsect:[1,13],subsequ:[1,9,13],subset:2,subtask:14,subtl:3,subtract:2,subtre:14,succe:[2,3],succeed:[1,2,3,5,17],success:[2,3,
5,6,13,16],successfulli:[2,3,16],sudo:14,sudoer:14,suit:6,sum:3,sundai:13,super_us:14,supersed:9,superus:14,superuser_filt:14,suppli:[4,14],support:[0,1,2,4,7,8,9,14],sure:[0,3,4,5,14,16],surfac:16,surgeri:6,swagger:0,sync:[1,2,4,13],sync_tim:2,sync_to_db:2,synchron:[2,4],syntax:9,sys:3,sysconfig:4,system:[2,3,4,9],systemd:[6,15],syvineckruyk:12,tab:[2,14],tabl:[2,3,8,10],table_nam:3,tag:5,take:[2,3,4,5,12,14,15,16,17],talk:[2,16],tardif:12,target:[2,4,13],target_field:2,target_tim:2,task1:3,task2:3,task3:3,task4:3,task:[0,1,2,4,6,8,13,14,15],task_failed_dep:1,task_id:[0,1,2,3,8,13,16],task_inst:[2,3],task_instance_key_str:2,task_or_task_list:2,task_param:1,task_regex:[1,2],task_stat:1,taskinst:[2,3],tbl:2,team:[2,14],tell:[1,2,5],templat:[1,2,6,10,11],template_fold:10,template_searchpath:[2,16],templated_command:16,templates_dict:2,templates_ext:2,temporari:[2,3],temporarili:4,tenanc:6,tenant:14,term:9,termin:[3,9],test:[1,2,3,5,6,8,10,14],test_env:3,test_mod:[2,4],test_plugin:10,t
establ:6,testview:10,text:[2,4,9,17],than:[2,3,6,16],thei:[1,2,3,4,6,7,8,10,11,13,14,16],them:[1,2,3,10,13,14,16],themselv:3,theori:[5,9],thereof:9,thi:[0,1,2,3,4,5,6,9,10,13,14,15,16,17],thing:[2,3,5,16],think:[6,16],third:9,this_dag_will_be_discov:3,those:[2,3,4,9,14],though:[2,3,6],thousand:3,thread:2,three:[3,4],threshold:10,thrift:2,through:[0,2,3,7,9,13,15,16],throughout:2,ticket:[1,14],tighten:14,time:[2,3,4,5,10,13,14,16,17],timedelta:[2,3,5,8,13,16],timedeltasensor:[2,5],timefram:2,timeout:[1,2,3],timesensor:2,timestamp:[2,3,13],tip:3,titl:[3,9],tmp:[3,8],tmp_file:2,to_nam:2,todai:2,togeth:3,token:2,toler:[2,3,8],tomorrow:2,tomorrow_d:2,tomorrow_ds_nodash:2,too:[2,3,13,14],tool:[3,7,10],toolbox:10,top:[2,4,6,11],topic:4,topograph:2,topolog:2,topological_sort:2,tornado:1,tort:9,touch:2,tour:[15,16],track:[3,9,16],trade:9,trademark:9,tradition:3,train:9,transact:2,transfer:[2,3,8,9],transform:9,translat:[5,9],transpar:17,travers:2,treat:[2,3,14],tree:[1,2,5,6,16],tree_view:2,
tri:[2,14],trigger:[1,2,6,15,16],trigger_dag:[1,13],trigger_dag_id:2,trigger_rul:[2,3],triggerdagrunoper:2,triggerrul:[2,3],troubl:7,troubleshoot:[6,17],trust:14,truth:2,ts_nodash:2,tty:2,tunnel:[2,14],tupl:2,turn:[2,3,13,14],tutori:[6,15],two:[2,3,16],type:[1,2,3,5,9,11,14],typic:[2,3],ubuntu:4,uid:14,ultim:[2,16],unassign:3,unavail:4,unblock:13,undead:6,undefin:14,under:[2,3,4,9,10,12],undergo:6,underli:10,underscor:4,understand:[2,5,10,16,17],undesir:13,unexpect:3,unexpectedli:3,unicod:2,union:9,uniqu:[2,16],unit:[4,6,14],unit_test_mod:4,unittest:2,unix:[2,14],unless:[2,5,9],unmet:1,unpack:[2,3],unpaus:1,unsound:[2,3],until:[0,2,3,4,6],up_for_retri:2,updat:[2,3,5,14],upgrad:1,upgradedb:1,upon:[3,4,15],uppercas:3,upstart:6,upstream:[1,2,3,5,13],upstream_fail:3,upstream_list:2,upstream_task_id:2,uri:[1,2,3,4,14],url:[2,3,4,10,11,14],usag:[1,2,16],use:[0,1,2,3,4,5,9,10,11,13,14,16],use_beelin:14,used:[1,2,3,10,14,16,17],useful:[2,3,7,17],user:[1,2,3,4,6,7,14,16],user_defined_macro:2
,user_filt:14,user_name_attr:14,usernam:[2,4,14],uses:[2,3,4,6,14,15],using:[1,2,3,4,5,6,7,8,9,14,15],usr:4,usual:[3,4],utf8:2,utf:2,util:[2,3,6,15],uuid:2,val:1,valid:[3,5,6,14],valu:[1,2,3,4,5,8,13,14,16,17],valuecheckoper:2,vanboxel:8,variabl:[1,4,5,6,16],verbal:9,verbos:[2,16],veri:[1,2,3,5,12,15,16],verifi:[3,5],version:[1,2,3,6,7,9],vertica:7,via:[3,14],view:[1,2,3,4,5,6,10],virtual:2,virtualenv:3,visibl:[2,3],visual:[6,10,11,16,17],vocabulari:3,wai:[2,3,4,5,7,10,13,14,16,17],wait:[1,2,3],wait_for_downstream:[2,5],wait_for_empty_queu:3,walk:[2,16],want:[2,3,4,5,10,13,14,16],warranti:9,wasn:[2,5],wast:3,watch:[4,5],weapon:11,web:[1,3,4,6,10,15,16],web_server_port:14,web_server_ssl_cert:14,web_server_ssl_kei:14,webhdfs_conn_id:2,webhdfs_default:2,webhdfssensor:2,webserv:[0,1,2,4,5,14,15,16],webservic:6,week:13,weekli:13,weight:2,well:[2,3,4,5,13],were:2,what:[2,3,6,7,13,14],whatev:[3,4,7],when:[0,1,2,3,4,5,6,10,11,13,14,16],whenev:2,where:[2,3,4,5,9,13,14,16,17],wherev:[2,9],whe
ther:[2,9,13],which:[1,2,3,4,5,9,13,14,15,16],whitelist:14,who:10,whole:[2,3,9],whom:9,whose:13,why:[1,6],wiedmer:12,wiki:12,wildcard:2,wildcard_match:2,win:2,window:13,wire:3,within:[1,2,3,4,5,8,9],without:[1,2,3,9,13,14],won:[2,5,7],word:[1,3,4,13,17],work:[1,2,3,4,5,6,9,10,11,13,14,15],worker:[1,2,3,4,5,6,7,10,14,16],worker_timeout:1,workerclass:1,workflow:[2,6],workload:10,worldwid:9,worri:16,worth:2,would:[2,3,4,5,6,13,14,16],wouldn:[2,5],wrap:[2,16],wrapper:2,write:[4,6,9,10,11,14,16],written:[2,9,13,16],wrong:16,www:[2,3,9],xcom:[2,6,16],xcom_pul:[2,3],xcom_push:[2,3],xcom_return_kei:2,xml:14,xst:14,yaml:3,year:13,yearli:13,yellow:2,yes:1,yesterdai:2,yesterday_d:2,yesterday_ds_nodash:2,yet:[2,3,6,13,14,17],you:[2,3,4,5,6,7,9,10,11,13,14,15,16,17],your:[2,3,4,5,6,7,9,10,11,13,15,16,17],yourself:2,yum:7,yvinec:12,yyyi:[1,2,3],yyyymmdd:2,zero:2,zip:[2,3],zip_dag:3,zip_dag_cont:3,zombi:6,zone:8,zoom:3},titles:["Experimental Rest API","Command Line Interface","API Reference","Conc
epts","Configuration","FAQ","Apache Airflow (incubating) Documentation","Installation","Integration","License","Plugins","Data Profiling","Project","Scheduling & Triggers","Security","Quick Start","Tutorial","UI / Screenshots"],titleterms:{"default":[2,3,16],"function":3,"import":16,AWS:8,Using:14,addit:3,adhoc:11,after:5,airflow:[5,6,7,10],all:5,amazon:8,anoth:5,apach:6,api:[0,2],argument:[3,16],assign:3,authent:[0,14],backend:4,backfil:[13,16],base:5,baseoper:2,basesensoroper:2,beyond:6,bigqueri:8,bigquerycheckoper:8,bigqueryhook:8,bigqueryintervalcheckoper:8,bigqueryoper:8,bigquerytobigqueryoper:8,bigquerytocloudstorageoper:8,bigqueryvaluecheckoper:8,bitshift:3,branch:3,build:10,can:5,catchup:13,celeri:4,chart:[11,17],cli:0,cloud:8,cluster:3,code:17,command:[1,5,16],committ:12,commun:[2,4],composit:3,concept:3,configur:4,connect:[3,4,5],content:6,context:[3,17],contribut:[2,4],core:3,creat:5,crypto:5,dag:[3,5,13,16,17],data:11,dataflow:8,dataflowhook:8,dataflowjavaoper:8,data
proc:8,dataprochadoopoper:8,dataprochiveoper:8,dataprocpigoper:8,dataprocpysparkoper:8,dataprocsparkoper:8,dataprocsparksqloper:8,datastor:8,deal:5,definit:16,depend:16,document:[3,6],durat:17,dynam:5,enabl:14,encrypt:5,endpoint:0,enterpris:14,exampl:[10,16],executor:2,experiment:0,extern:13,extra:7,failur:5,faq:5,file:16,form:11,gantt:17,gcp:8,get:[5,7],ghe:14,github:14,googl:[8,14],googlecloudstoragedownloadoper:8,googlecloudstoragehook:8,googlecloudstoragetobigqueryoper:8,graph:17,histori:12,hook:[2,3],horizon:6,how:5,idea:3,imperson:14,incub:6,instal:[5,7],instanc:[3,17],instanti:16,integr:[4,8],interfac:[1,10],isn:5,jinja:[3,16],keep:13,kerbero:14,latest:3,ldap:14,licens:9,limit:14,line:[1,16],link:12,list:5,log:4,macro:2,manag:3,menu:17,meso:4,metadata:[5,16],mind:13,mode:4,model:2,modul:16,multi:14,next:[15,16],note:3,oauth:14,onli:3,oper:[2,3,8],option:4,out:4,own:14,packag:[3,7],password:[5,14],pipelin:16,platform:8,plugin:10,polici:3,pool:3,principl:6,process:5,profil:11,p
roject:12,queri:11,queue:3,quick:15,recap:16,refer:2,resourc:12,rest:0,roadmap:12,roll:14,rule:3,run:[3,5,13,16],scale:4,schedul:[5,13],scope:3,screenshot:[11,17],script:16,secur:14,set:[4,14,16],sla:3,ssl:14,start:15,start_dat:5,still:5,storag:8,subdag:3,systemd:4,task:[3,5,16,17],templat:[3,16],tenanc:14,test:[4,16],top:10,tree:17,trigger:[3,5,13],tutori:16,undead:3,upstart:4,valid:16,variabl:[2,3,17],view:17,web:14,webservic:8,what:[5,10,15,16],why:[5,10],workflow:3,xcom:3,your:14,zombi:3}})
\ No newline at end of file
+Search.setIndex({docnames:["api","cli","code","concepts","configuration","faq","index","installation","integration","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],envversion:52,filenames:["api.rst","cli.rst","code.rst","concepts.rst","configuration.rst","faq.rst","index.rst","installation.rst","integration.rst","license.rst","plugins.rst","profiling.rst","project.rst","scheduler.rst","security.rst","start.rst","tutorial.rst","ui.rst"],objects:{"airflow.contrib":{hooks:[2,1,0,"-"],operators:[2,1,0,"-"]},"airflow.contrib.executors.mesos_executor":{MesosExecutor:[2,0,1,""]},"airflow.contrib.hooks":{BigQueryHook:[2,0,1,""],FTPHook:[2,0,1,""],GoogleCloudStorageHook:[2,0,1,""]},"airflow.contrib.hooks.BigQueryHook":{get_conn:[2,2,1,""],get_pandas_df:[2,2,1,""],get_service:[2,2,1,""],insert_rows:[2,2,1,""],table_exists:[2,2,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[2,2,1,""],create_directory:[2,2,1,""],delete_directory:[2,2,1,""],delete_f
ile:[2,2,1,""],describe_directory:[2,2,1,""],get_conn:[2,2,1,""],list_directory:[2,2,1,""],rename:[2,2,1,""],retrieve_file:[2,2,1,""],store_file:[2,2,1,""]},"airflow.contrib.hooks.GoogleCloudStorageHook":{"delete":[2,2,1,""],copy:[2,2,1,""],download:[2,2,1,""],exists:[2,2,1,""],get_conn:[2,2,1,""],is_updated_after:[2,2,1,""],list:[2,2,1,""],upload:[2,2,1,""]},"airflow.contrib.hooks.bigquery_hook":{BigQueryHook:[8,0,1,""]},"airflow.contrib.hooks.bigquery_hook.BigQueryHook":{get_conn:[8,2,1,""],get_pandas_df:[8,2,1,""],get_service:[8,2,1,""],insert_rows:[8,2,1,""],table_exists:[8,2,1,""]},"airflow.contrib.hooks.datastore_hook":{DatastoreHook:[8,0,1,""]},"airflow.contrib.hooks.datastore_hook.DatastoreHook":{allocate_ids:[8,2,1,""],begin_transaction:[8,2,1,""],commit:[8,2,1,""],delete_operation:[8,2,1,""],export_to_storage_bucket:[8,2,1,""],get_conn:[8,2,1,""],get_operation:[8,2,1,""],import_from_storage_bucket:[8,2,1,""],lookup:[8,2,1,""],poll_operation_until_done:[8,2,1,""],rollback:[
8,2,1,""],run_query:[8,2,1,""]},"airflow.contrib.hooks.emr_hook":{EmrHook:[8,0,1,""]},"airflow.contrib.hooks.gcp_dataflow_hook":{DataFlowHook:[8,0,1,""]},"airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook":{get_conn:[8,2,1,""]},"airflow.contrib.hooks.gcp_mlengine_hook":{MLEngineHook:[8,0,1,""]},"airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook":{create_job:[8,2,1,""],create_model:[8,2,1,""],create_version:[8,2,1,""],delete_version:[8,2,1,""],get_conn:[8,2,1,""],get_model:[8,2,1,""],list_versions:[8,2,1,""],set_default_version:[8,2,1,""]},"airflow.contrib.hooks.gcs_hook":{GoogleCloudStorageHook:[8,0,1,""]},"airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook":{"delete":[8,2,1,""],copy:[8,2,1,""],download:[8,2,1,""],exists:[8,2,1,""],get_conn:[8,2,1,""],is_updated_after:[8,2,1,""],list:[8,2,1,""],upload:[8,2,1,""]},"airflow.contrib.hooks.wasb_hook":{WasbHook:[8,0,1,""]},"airflow.contrib.operators":{QuboleOperator:[2,0,1,""]},"airflow.contrib.operators.bigquery_check_operator"
:{BigQueryCheckOperator:[8,0,1,""],BigQueryIntervalCheckOperator:[8,0,1,""],BigQueryValueCheckOperator:[8,0,1,""]},"airflow.contrib.operators.bigquery_operator":{BigQueryOperator:[8,0,1,""]},"airflow.contrib.operators.bigquery_to_bigquery":{BigQueryToBigQueryOperator:[8,0,1,""]},"airflow.contrib.operators.bigquery_to_gcs":{BigQueryToCloudStorageOperator:[8,0,1,""]},"airflow.contrib.operators.databricks_operator":{DatabricksSubmitRunOperator:[8,0,1,""]},"airflow.contrib.operators.dataflow_operator":{DataFlowJavaOperator:[8,0,1,""],DataFlowPythonOperator:[8,0,1,""]},"airflow.contrib.operators.dataproc_operator":{DataProcHadoopOperator:[8,0,1,""],DataProcHiveOperator:[8,0,1,""],DataProcPigOperator:[8,0,1,""],DataProcPySparkOperator:[8,0,1,""],DataProcSparkOperator:[8,0,1,""],DataProcSparkSqlOperator:[8,0,1,""]},"airflow.contrib.operators.ecs_operator":{ECSOperator:[8,0,1,""]},"airflow.contrib.operators.emr_add_steps_operator":{EmrAddStepsOperator:[8,0,1,""]},"airflow.contrib.operators.
emr_create_job_flow_operator":{EmrCreateJobFlowOperator:[8,0,1,""]},"airflow.contrib.operators.emr_terminate_job_flow_operator":{EmrTerminateJobFlowOperator:[8,0,1,""]},"airflow.contrib.operators.file_to_wasb":{FileToWasbOperator:[8,0,1,""]},"airflow.contrib.operators.gcs_download_operator":{GoogleCloudStorageDownloadOperator:[8,0,1,""]},"airflow.contrib.operators.gcs_to_bq":{GoogleCloudStorageToBigQueryOperator:[8,0,1,""]},"airflow.contrib.operators.hipchat_operator":{HipChatAPIOperator:[2,0,1,""],HipChatAPISendRoomNotificationOperator:[2,0,1,""]},"airflow.contrib.operators.mlengine_operator":{MLEngineBatchPredictionOperator:[8,0,1,""],MLEngineModelOperator:[8,0,1,""],MLEngineTrainingOperator:[8,0,1,""],MLEngineVersionOperator:[8,0,1,""]},"airflow.contrib.sensors.wasb_sensor":{WasbBlobSensor:[8,0,1,""],WasbPrefixSensor:[8,0,1,""]},"airflow.executors":{LocalExecutor:[2,0,1,""],SequentialExecutor:[2,0,1,""]},"airflow.hooks":{DbApiHook:[2,0,1,""],DruidHook:[2,0,1,""],HiveCliHook:[2,0,
1,""],HiveMetastoreHook:[2,0,1,""],HiveServer2Hook:[2,0,1,""],HttpHook:[2,0,1,""],MySqlHook:[2,0,1,""],PostgresHook:[2,0,1,""],PrestoHook:[2,0,1,""],S3Hook:[2,0,1,""],SqliteHook:[2,0,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[2,2,1,""],bulk_load:[2,2,1,""],get_conn:[2,2,1,""],get_cursor:[2,2,1,""],get_first:[2,2,1,""],get_pandas_df:[2,2,1,""],get_records:[2,2,1,""],insert_rows:[2,2,1,""],run:[2,2,1,""]},"airflow.hooks.HiveCliHook":{load_df:[2,2,1,""],load_file:[2,2,1,""],run_cli:[2,2,1,""],test_hql:[2,2,1,""]},"airflow.hooks.HiveMetastoreHook":{check_for_named_partition:[2,2,1,""],check_for_partition:[2,2,1,""],get_databases:[2,2,1,""],get_metastore_client:[2,2,1,""],get_partitions:[2,2,1,""],get_table:[2,2,1,""],get_tables:[2,2,1,""],max_partition:[2,2,1,""],table_exists:[2,2,1,""]},"airflow.hooks.HiveServer2Hook":{get_pandas_df:[2,2,1,""],get_records:[2,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[2,2,1,""],run:[2,2,1,""],run_and_check:[2,2,1,""]},"airflow.hooks.MySqlHook":{bulk
_load:[2,2,1,""],get_conn:[2,2,1,""]},"airflow.hooks.PostgresHook":{copy_expert:[2,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[2,2,1,""],get_first:[2,2,1,""],get_pandas_df:[2,2,1,""],get_records:[2,2,1,""],run:[2,2,1,""]},"airflow.hooks.S3Hook":{check_for_bucket:[2,2,1,""],check_for_key:[2,2,1,""],check_for_prefix:[2,2,1,""],check_for_wildcard_key:[2,2,1,""],get_bucket:[2,2,1,""],get_key:[2,2,1,""],get_wildcard_key:[2,2,1,""],list_keys:[2,2,1,""],list_prefixes:[2,2,1,""],load_file:[2,2,1,""],load_string:[2,2,1,""],read_key:[2,2,1,""]},"airflow.hooks.S3_hook":{S3Hook:[8,0,1,""]},"airflow.hooks.SqliteHook":{get_conn:[2,2,1,""]},"airflow.macros":{ds_add:[2,3,1,""],ds_format:[2,3,1,""],hive:[2,1,0,"-"]},"airflow.macros.hive":{closest_ds_partition:[2,3,1,""],max_partition:[2,3,1,""]},"airflow.models":{BaseOperator:[2,0,1,""],Connection:[2,0,1,""],DAG:[2,0,1,""],DagBag:[2,0,1,""],TaskInstance:[2,0,1,""]},"airflow.models.BaseOperator":{clear:[2,2,1,""],dag:[2,4,1,""],deps:[2,4,1,""],det
ect_downstream_cycle:[2,2,1,""],downstream_list:[2,4,1,""],execute:[2,2,1,""],get_direct_relatives:[2,2,1,""],get_flat_relatives:[2,2,1,""],get_task_instances:[2,2,1,""],has_dag:[2,2,1,""],on_kill:[2,2,1,""],post_execute:[2,2,1,""],pre_execute:[2,2,1,""],prepare_template:[2,2,1,""],render_template:[2,2,1,""],render_template_from_field:[2,2,1,""],run:[2,2,1,""],schedule_interval:[2,4,1,""],set_downstream:[2,2,1,""],set_upstream:[2,2,1,""],upstream_list:[2,4,1,""],xcom_pull:[2,2,1,""],xcom_push:[2,2,1,""]},"airflow.models.Connection":{extra_dejson:[2,4,1,""]},"airflow.models.DAG":{add_task:[2,2,1,""],add_tasks:[2,2,1,""],clear:[2,2,1,""],cli:[2,2,1,""],concurrency_reached:[2,4,1,""],create_dagrun:[2,2,1,""],deactivate_stale_dags:[2,5,1,""],deactivate_unknown_dags:[2,5,1,""],filepath:[2,4,1,""],folder:[2,4,1,""],get_active_runs:[2,2,1,""],get_dagrun:[2,2,1,""],get_last_dagrun:[2,2,1,""],get_num_active_runs:[2,2,1,""],get_num_task_instances:[2,5,1,""],get_run_dates:[2,2,1,""],get_templa
te_env:[2,2,1,""],is_paused:[2,4,1,""],latest_execution_date:[2,4,1,""],normalize_schedule:[2,2,1,""],run:[2,2,1,""],set_dependency:[2,2,1,""],sub_dag:[2,2,1,""],subdags:[2,4,1,""],sync_to_db:[2,2,1,""],topological_sort:[2,2,1,""],tree_view:[2,2,1,""]},"airflow.models.DagBag":{bag_dag:[2,2,1,""],collect_dags:[2,2,1,""],dagbag_report:[2,2,1,""],get_dag:[2,2,1,""],kill_zombies:[2,2,1,""],process_file:[2,2,1,""],size:[2,2,1,""]},"airflow.models.TaskInstance":{are_dependencies_met:[2,2,1,""],are_dependents_done:[2,2,1,""],clear_xcom_data:[2,2,1,""],command:[2,2,1,""],command_as_list:[2,2,1,""],current_state:[2,2,1,""],error:[2,2,1,""],generate_command:[2,5,1,""],get_dagrun:[2,2,1,""],init_on_load:[2,2,1,""],is_premature:[2,4,1,""],key:[2,4,1,""],next_retry_datetime:[2,2,1,""],pool_full:[2,2,1,""],previous_ti:[2,4,1,""],ready_for_retry:[2,2,1,""],refresh_from_db:[2,2,1,""],try_number:[2,4,1,""],xcom_pull:[2,2,1,""],xcom_push:[2,2,1,""]},"airflow.operators":{BashOperator:[2,0,1,""],Branch
PythonOperator:[2,0,1,""],DummyOperator:[2,0,1,""],EmailOperator:[2,0,1,""],ExternalTaskSensor:[2,0,1,""],GenericTransfer:[2,0,1,""],HdfsSensor:[2,0,1,""],HiveOperator:[2,0,1,""],HivePartitionSensor:[2,0,1,""],HiveToDruidTransfer:[2,0,1,""],HiveToMySqlTransfer:[2,0,1,""],HttpSensor:[2,0,1,""],MetastorePartitionSensor:[2,0,1,""],MySqlOperator:[2,0,1,""],MySqlToHiveTransfer:[2,0,1,""],NamedHivePartitionSensor:[2,0,1,""],PostgresOperator:[2,0,1,""],PrestoCheckOperator:[2,0,1,""],PrestoIntervalCheckOperator:[2,0,1,""],PrestoValueCheckOperator:[2,0,1,""],PythonOperator:[2,0,1,""],S3KeySensor:[2,0,1,""],S3ToHiveTransfer:[2,0,1,""],ShortCircuitOperator:[2,0,1,""],SimpleHttpOperator:[2,0,1,""],SlackAPIOperator:[2,0,1,""],SlackAPIPostOperator:[2,0,1,""],SqlSensor:[2,0,1,""],TimeSensor:[2,0,1,""],TriggerDagRunOperator:[2,0,1,""],WebHdfsSensor:[2,0,1,""]},"airflow.operators.BashOperator":{execute:[2,2,1,""]},"airflow.operators.HdfsSensor":{filter_for_filesize:[2,5,1,""],filter_for_ignored_ext:
[2,5,1,""]},"airflow.operators.HiveToDruidTransfer":{construct_ingest_query:[2,2,1,""]},"airflow.operators.SlackAPIOperator":{construct_api_call_params:[2,2,1,""],execute:[2,2,1,""]},"airflow.operators.docker_operator":{DockerOperator:[2,0,1,""]},"airflow.operators.redshift_to_s3_operator":{RedshiftToS3Transfer:[8,0,1,""]},"airflow.operators.s3_file_transform_operator":{S3FileTransformOperator:[8,0,1,""]},"airflow.operators.s3_to_hive_operator":{S3ToHiveTransfer:[8,0,1,""]},"airflow.operators.sensors":{BaseSensorOperator:[2,0,1,""]},airflow:{executors:[2,1,0,"-"],hooks:[2,1,0,"-"],macros:[2,1,0,"-"],models:[2,1,0,"-"],operators:[2,1,0,"-"]}},objnames:{"0":["py","class","Python class"],"1":["py","module","Python module"],"2":["py","method","Python method"],"3":["py","function","Python function"],"4":["py","attribute","Python attribute"],"5":["py","staticmethod","Python static method"]},objtypes:{"0":"py:class","1":"py:module","2":"py:method","3":"py:function","4":"py:attribute","5":"
py:staticmethod"},terms:{"00am":2,"01t23":13,"03t00":8,"10gb":2,"10pm":3,"128m":2,"2am":2,"abstract":[2,3,6,10],"boolean":[2,3,8],"break":8,"byte":[2,4],"case":[2,3,4,5,8,13,14,16],"char":2,"class":[1,2,3,7,8,9,10,16],"default":[0,1,4,5,6,8,13,15,17],"export":[1,2,4,8,15],"final":[2,8],"float":2,"function":[0,2,5,6,8,14,16],"import":[1,2,3,4,5,6,7,8,9,10,13,14],"int":[2,8],"long":[2,8],"new":[2,3,5,8,10,13,14],"return":[0,1,2,3,5,8,10],"short":2,"static":[2,6,10],"super":14,"switch":[2,4,14],"true":[2,3,4,5,8,13,14,16],"try":[1,2,3,4],"var":[2,4,8],"while":[2,3,5,6,9,13,14,15,16,17],AND:[2,8,9],AWS:[2,4,6],Added:3,And:14,Are:5,But:[4,8],ECS:[2,8],FOR:[2,9],For:[0,2,3,4,7,8,9,12,14,16],GCS:8,IDs:[2,8],Its:[2,8],LTS:4,Not:[2,3,9],One:[3,8,14,16],QDS:[2,7],SAS:8,That:[13,16],The:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17],There:[2,3,5,8,10],These:[2,3,4,11],USE:[2,9],Use:[1,4,6,8],Used:2,Using:[3,6,8,10,16],Will:[2,10],With:14,__class__:3,__doc__:3,__init__:[2,3,8],__name__:[3,10],_c
md:4,_copying_:2,_filter:14,_host:14,a_pig_script:8,abbrevi:3,abil:[3,14],abl:[2,4,5,14,15,16],about:[2,3,4,8,9,13,16],abov:[3,4,8,9,13,14,17],absenc:3,absolut:3,accept:[2,6,8,9],access:[0,1,2,3,4,8,10,14],access_logfil:1,access_token:17,accesskei:3,accomplish:[2,8],accord:14,accordingli:[3,14],account:[2,8,14],achiev:14,acquir:2,across:[2,4,17],act:[2,3,9,16],action:[2,8,17],activ:[0,1,2,3,5,10,13,14,16],active_dag_id:2,actual:[2,3,5,8,16],acut:2,acycl:[2,3,6],add:[1,2,3,4,5,8,9,14],add_task:2,added:[2,8],addendum:9,adding:[1,8,13],addit:[2,4,6,8,9],addition:14,addprinc:14,address:4,adhoc:[1,2,6],adjust:[4,14],admin:[3,4,10,11,14,15],admin_view:10,advanc:[2,16],advantag:14,advis:[2,9],affect:3,after:[2,3,6,8,13],afterward:2,again:1,against:[2,3,5,8,9,14,16],aggregate_db_message_job:3,aggregate_db_message_job_cmd:3,ago:5,agre:9,agreement:[3,9],airbnb:[2,3,12,13,16],airflow:[0,1,2,3,4,8,11,12,13,15,16,17],airflow__:4,airflow__core__fernet_kei:4,airflow__core__sql_alchemy_conn:4,airfl
ow__core__unit_test_mod:4,airflow_config:4,airflow_conn_:[3,4],airflow_conn_postgres_mast:3,airflow_hom:[4,8,10,15],airflow_local_set:8,airflow_set:3,airflow_tmp_dir:2,airflowignor:2,airflowplugin:10,airflowtestplugin:10,aiyzngq:2,alchemi:4,alert:[2,3,10],alex:8,algorithm:2,align:5,all:[0,1,2,3,4,6,7,8,9,13,14,16],all_db:7,all_don:[2,3],all_fail:[2,3],all_success:[2,3],alleg:9,alloc:8,allocate_id:8,allocateid:8,allow:[1,2,3,4,5,6,8,10,11,13,14,15,16,17],allow_jagged_row:8,allow_large_result:[2,8],allow_quoted_newlin:8,allowed_st:2,allowed_team:14,allowinsert:2,alon:9,along:[2,5,9,13,16],alongsid:[2,9,13],alreadi:[1,2,8],alright:16,also:[2,3,4,5,7,9,10,13,14,16],alter:[2,3,4,5,14],altern:[2,4,13],altert:[2,8],alwai:[2,3,17],amazon:[4,6],amount:[1,2,3,8],analys:17,analyz:3,anand:12,anchor:2,ani:[0,2,3,4,5,8,9,13,16,17],annot:9,announc:12,anomali:10,anoth:[2,3,6,8],ansibl:4,anyon:14,anyth:[3,16],anytim:3,anywher:[2,3],aoen:12,apach:[2,4,7,9,12,15],api:[1,6,7,8,9,14],api_call_param:2,ap
i_cli:0,api_kei:17,api_param:2,api_vers:2,apikei:17,apiv2:2,app:2,app_id:2,appar:2,appear:[2,3,9],append:[2,4],appendix:9,appli:[2,3,7,9,13],applic:[2,8,9,10,11,14],appropri:[3,9,14],arbitrari:[3,6],architectur:6,archiv:[2,8,9],archive1:2,archive2:2,are_dependencies_met:2,are_dependents_don:2,aren:[2,3],arg:[2,8],argument:[2,6,8,13],aris:9,around:[2,6,16],arrai:[2,6,10],arriv:8,arthur:12,artwr:12,ascii:2,asf:6,assert:9,asset:2,assign:[2,5,6],associ:[2,3,9,13],assum:[2,3,4,9,14,16],asterisk:14,async:7,atom:[3,13],attach:[2,3,9],attach_to:2,attack:14,attempt:[1,2],attent:[2,5],attr:2,attribut:[1,2,3,9],audit:10,auth:[0,2,14],auth_backend:[0,14],auth_token:2,authen:14,authent:[2,3,6,7,8],authmechan:2,author:[2,4,6,8,9,14,16,17],authorship:9,auto:[2,5],autocommit:[2,8],automat:[2,3,4,14],autoscalingalgorithm:8,avail:[0,2,3,4,8,9,10,11],averag:[2,8],avoid:[2,3],awai:2,awar:[3,8,14],aws:8,aws_conn_id:[2,8],aws_default:[2,8],aws_hook:2,awshook:2,azkaban:6,azur:[2,6],back:[2,5,8,10],backend
:[0,2,6,7,8,14,15],backfil:[2,3,5,6,15],background:[2,16],backoff:2,backup:[4,8],backward:8,bag:2,bag_dag:2,balanc:3,bar:[2,3,16],bare:16,base64:4,base:[2,3,4,6,8,9,10,13,14],base_dag:2,base_executor:[2,10],base_hook:[2,10],base_log_fold:8,base_task_runn:8,base_url:[2,14],basedag:2,basedagbag:2,basedn:14,baseexecutor:[2,10],basehook:[2,3,10],baseoper:[3,6,8,10,16],basesensoroper:6,baseview:10,bash:[2,3,8,16],bash_1:3,bash_command:[2,3,16],bash_command_to_run:4,bash_oper:[2,13,16],bash_queu:16,bashoper:[2,3,13,16],basi:9,basic:[2,3,5,7,8,10,11,16],batch:8,bcc:2,bcrypt:14,beauchemin:12,becaus:[2,3,8],becom:[2,3,6,10,16],beelin:2,been:[1,2,3,4,8,9,13,14],befor:[1,2,3,4,8,14],begin:[2,3,8,13],begin_transact:8,begintransact:8,behalf:9,behav:[2,5],behavior:[2,3,13],behind:[2,4,7,13],being:[2,3,8,11],belong:[2,8,14],below:[2,4,5,8,9,10,15,16],benefici:9,best:2,beta:8,better:[5,16],between:[2,3,8,13,16],beyond:[2,5],bigqueri:[2,6],bigquery_check_oper:8,bigquery_conn_id:[2,8],bigquery_defaul
t:[2,8],bigquery_hook:8,bigquery_oper:[2,8],bigquery_to_bigqueri:8,bigquery_to_gc:[2,8],bigqueryhook:[2,6],bigqueryoper:2,bigquerytocloudstorageoper:2,bin:3,bind:9,bind_password:14,bind_us:14,bit:4,bitshift:6,blob:[2,6,13,16],blob_nam:[2,8],block:[2,3,8,16,17],blockblockservic:8,blog:[2,9],blueprint:10,bodi:[2,8],boilerpl:9,bolk:12,bolkedebruin:12,bool:[2,8],boot:4,both:[2,5,8],bother:16,boto3:[2,8],boto:2,bottleneck:17,boundary_queri:2,box:[2,4,15],bql:[2,8],bracket:[9,16],branch:[2,6],branch_fals:3,branchpythonoper:[2,3],broker:[1,4],broker_api:1,broker_url:4,brought:12,brows:[3,5],browser:14,bruin:12,bucket:[2,4,8],bucket_kei:2,bucket_nam:2,buffer:2,bug:4,build:[2,3,4,6,8,11],built:[2,3,4,6,10,11,16],bulk:[5,17],bulk_dump:2,bulk_load:2,bump:3,bundl:2,burn:1,but_this_dag_will_not:3,c12318391:2,cacert:14,cach:14,cal:8,call:[2,3,4,5,8,13,16],callabl:2,callback:14,can:[0,2,3,4,6,7,8,10,11,13,14,15,16,17],cancel:8,cannot:[2,3,8,9,16],cap_net_bind_servic:14,capabl:[10,13],capac:[2,3],c
apacityschedul:2,card:2,care:[1,4],carri:[3,9],cascad:3,cast:[2,8],cat:2,catchup:[2,6],catchup_by_default:13,categori:10,caus:[5,9],celeri:[1,3,5,6,7,14],celery_hostnam:1,celery_result_backend:4,celery_ssl_act:14,celery_ssl_cacert:14,celery_ssl_cert:14,celery_ssl_kei:14,celeryexecutor:[2,3,4,5,7,13,14],cell:2,central1:8,central:3,cert:[2,14],certain:[2,3,8],certif:[1,2,14],cfg:[1,2,3,4,8,13,14,15,16],cfg_path:[1,2],chain:3,chanc:4,chang:[0,4,6,8,9,13,14,15,16],channel:[2,12],charact:9,character:3,charg:[5,9],charset:2,chart:[6,10],chat:12,check:[1,2,3,5,8],check_for_blob:8,check_for_bucket:2,check_for_kei:2,check_for_named_partit:2,check_for_partit:2,check_for_prefix:[2,8],check_for_wildcard_kei:2,check_head:[2,8],check_oper:2,check_opt:8,checkoper:2,checkout:14,chef:4,child:[2,3],child_dag:3,child_dag_nam:3,chmod:14,choic:[1,2,8,16],choos:[2,3,9,14],chri:12,chrisr:[1,8],circuit:2,claim:9,clariti:6,claus:2,clean:2,cleanup:[2,5],clear:[2,3,5,6,13,17],clear_xcom_data:2,cli:[2,3,4,5,6,
8,13],click:[5,14,16,17],client:[0,2,7,8,14],client_id:14,client_secret:14,close:[2,5,14],close_conn:2,closest:2,closest_ds_partit:2,cloud:[2,4,6,7],cloudant:7,cluster:[2,4,6,8],cluster_label:2,cluster_nam:8,cmdline:2,code:[1,2,3,4,6,8,9,10,13,16],col:2,collabor:6,collect:[2,3,10],collect_dag:2,color:2,column:[2,8],com:[2,3,8,13,14,16],combin:[3,9],come:[2,13],comma:[1,2,3],command:[2,3,4,6,7,8,13,15],command_as_list:2,command_typ:2,comment:9,commerci:9,commit:[2,8,12,14],commit_everi:[2,8],committ:6,common:[2,3,4,5,8,9,16],commonli:2,commun:[3,6,8,9,16],compani:10,compar:6,comparison:2,compil:[5,9],complet:[2,3,4,6,8,13],complex:[2,3,6,8,10],compli:9,complianc:9,complic:16,compon:10,compos:[3,14,16],composit:6,comprehens:17,compress:[2,8],comput:[2,8,9],concept:[4,5,6,13,16],conceptu:3,concern:3,concur:5,concurr:[1,2,5],concurrency_reach:2,condit:[2,3,7,9],conf:[1,2,4],config:[1,8,10,14],config_templ:8,configur:[0,1,2,3,5,6,8,9,13,14,15,16,17],confirm:[1,5,13],confirm_prompt:2,conf
lict:[2,8],confus:[2,3,5],conjunct:[1,3,15],conn_extra:1,conn_host:1,conn_id:[1,2,3,4],conn_login:1,conn_password:1,conn_port:1,conn_schema:1,conn_typ:[1,2],conn_uri:1,connect:[2,6,7,8,10,11,14],connecton:11,consequenti:9,consid:[2,3,4,8],consider:[2,8],consist:[2,3,6,8,9],consol:[8,14],conspicu:9,constant:[2,3],constitu:3,constitut:9,constraint:2,constru:9,construct:[2,8],construct_api_call_param:2,construct_ingest_queri:2,constructor:[2,8,16],consult:4,contact:[4,9],contain:[2,3,4,5,6,9,13,14,16,17],container_nam:[2,8],content:[2,3,8,9,10,12,14],context:[2,4,5,6,16],continu:[2,6,8],contract:9,contrib:[2,3,8,14],contribut:[6,8,9],contributor:[9,12],contributori:9,control:[2,3,8,9,17],conveni:[2,3],convent:[2,3,8],convers:9,convert:2,coordin:2,copi:[2,4,8,9,14],copy_expert:2,copyright:[9,14],core:[2,4,6,10,14],correct:3,correpsond:3,correspond:[2,3,4,5],could:[2,3,8,14,16],count:[1,2,8],counterclaim:9,cours:14,cover:13,cpu:[2,3,10],craft:[2,8],crawl:[2,13],creat:[0,2,3,4,6,8,10,11,1
3,14,15,16,17],create_dagrun:2,create_directori:2,create_disposit:[2,8],create_if_need:[2,8],create_job:8,create_job_flow:8,create_model:8,create_vers:8,creation:[2,8],credenti:[2,8,14],credit:14,criccomini:12,criteria:[2,3,8],criterion:2,critic:[1,2,3,8],cron:[2,3,5,13],cross:[3,9,16],crt:14,crud:1,crypto:[4,6,7],cryptographi:4,csv:[2,8],curli:16,current:[0,2,8,13,17],current_st:2,cursor:[2,8],custom:[8,10,16],customari:9,cycl:[2,3,16],daemon:[1,2,4,13],dag:[0,1,2,4,6,8,14,15],dag_1:3,dag_2:3,dag_fold:[2,3,4],dag_id:[0,1,2,3,4,5,16],dag_md:3,dag_regex:1,dag_run:[0,2],dag_run_obj:2,dag_stat:6,dagbag:[1,2,5,8],dagbag_report:2,dagfil:3,dagrun:[1,2,5],dagrun_timeout:2,dags_fold:[4,5,8],dai:[2,5,8,13,15,16],daili:[2,5,13],damag:9,dan:[2,12],dask:6,dask_host:4,dask_port:4,daskexecutor:4,data:[2,3,6,7,8,10,14,16,17],data_format:8,data_format_unspecifi:8,data_profiler_filt:14,databas:[1,2,3,4,6,7,8,10,11,15,16],databrick:[2,6],databricks_conn_id:[2,8],databricks_default:[2,8],databricks_op
er:[2,8],databricks_retry_limit:[2,8],databrickssubmitrunoper:[2,6],databs:2,dataflow:6,dataflow_:8,dataflow_default_opt:8,dataflow_oper:8,dataflowhook:6,dataflowoper:8,datafram:[2,8],datafu:8,datapflow_exampl:8,dataproc:6,dataproc_:8,dataproc_hadoop_jar:8,dataproc_hadoop_properti:8,dataproc_hive_jar:8,dataproc_hive_properti:8,dataproc_oper:8,dataproc_pig:8,dataproc_pig_jar:8,dataproc_pig_properti:8,dataproc_pyspark_jar:8,dataproc_pyspark_properti:8,dataproc_spark_jar:8,dataproc_spark_properti:8,dataset:[2,8,13],dataset_id:[2,8],datasourc:2,datastor:6,datastore_conn_id:8,datastore_hook:8,datastorehook:6,date:[1,2,3,8,9,13,16],date_filter_column:[2,8],datestamp:2,datetim:[2,3,5,8,13,16],dateutil:2,davydov:12,days_back:[2,8],db3:[2,8],db_tabl:2,db_tap_id:2,db_update_kei:2,db_update_mod:2,dbapi_hook:2,dbapihook:[2,8],dbexportcmd:2,dbimportcmd:2,dbtap_id:2,dbtapquerycmd:2,deactiv:2,deactivate_stale_dag:2,deactivate_unknown_dag:2,deal:6,debug:[1,2,16],decid:[2,4,8],decis:6,declar:2,decom
press:[2,8],deep:2,def:[2,3,10],default_arg:[2,3,5,8,13,16],default_imperson:14,default_logging_config:8,default_login:14,default_param:5,default_queu:3,default_view:2,defend:9,defer:3,deferred_op:3,defin:[2,3,4,5,6,8,9,10,13,14,16],definit:[0,2,3,6,8,9,10],del:2,delai:[1,2,5],delay_on_limit:1,delay_on_limit_sec:2,deleg:[2,8],delegate_to:[2,8],delet:[1,2,3,8,13,17],delete_directori:2,delete_fil:2,delete_oper:8,delete_vers:8,deliber:[2,9],delimit:[1,2,3,8],demonstr:3,deni:0,deny_al:0,dep:2,dep_context:2,depcontext:2,depend:[1,2,3,4,5,6,7,8,13,17],depends_on_past:[1,2,3,5,8,13,16],deploi:10,deploy:10,deploymenturl:8,depth:16,deriv:[2,4,9,10],descend:3,describ:[2,3,5,9],describe_directori:2,descript:[1,2,3,9],deseri:[1,2,3],deserialize_json:3,design:[3,9,13],desir:[2,8,16],dest:8,dest_aws_conn_id:8,dest_s3_kei:8,destin:[2,8],destination_bucket:[2,8],destination_cloud_storage_uri:[2,8],destination_conn_id:2,destination_dataset_t:[2,8],destination_object:[2,8],destination_project_dataset
_t:8,destination_t:2,detail:[2,3,4,5,8,14,17],detect:[2,10],detect_downstream_cycl:2,determin:[2,3,8,9],dev:[4,7,12],devel:7,devel_hadoop:7,develop:[1,2,14,16],deviat:[2,8],dialect:[2,8],dialog:13,dict:[1,2,3,8,16],dictionari:[2,3,5,8,16],did:16,die:3,differ:[2,3,5,8,9,10,16,17],dimens:2,direct:[2,3,4,6,9],directli:[2,3,5,8,11],directori:[1,2,3,4,8,14,16],dirti:15,disabl:2,disclaim:[6,9],discuss:9,disk:2,displai:[4,9,13,16],disposit:8,disregard:[2,16],distinct:2,distribut:[2,3,4,7,9],do_pickl:1,doamin:8,doc:[2,3,5,8,16],doc_json:3,doc_md:3,doc_rst:3,doc_yaml:3,docker:2,docker_conn_id:2,docker_oper:2,docker_url:2,dockeroper:[2,3],document:[2,4,5,9,12,14,16],doe:[1,2,3,4,5,6,7,8,9,14,16],does_not_exist:2,doesn:[1,2,3,8,16],domain:[2,8,14],domainnam:0,don:[2,3,4,7,9,13,14],done:[2,3,5,8,13,14,16],donot_pickl:[1,2],door:3,dot:[2,3,8],doubl:[2,4,16],down:[1,3],download:[2,4,8],downstream:[1,2,3,13],downstream_list:2,downstream_task_id:2,drive:4,driven:[2,10],driver:4,drop:[2,8,10],druid:
[2,7],druid_datasourc:2,druid_ingest_conn_id:2,druid_ingest_default:2,druidhook:2,dry:1,dry_run:[1,2,8],ds_add:[2,16],ds_format:2,ds_nodash:[2,8],dttm:2,dubiou:[2,8],due:4,dummi:[2,3,10],dummy_1:3,dummy_oper:3,dummy_task:3,dummyoper:[2,3],dump:2,durat:[1,2,6],dure:[2,3,4,8,17],dynam:[2,3,6],each:[2,3,4,8,9,12,13,16],eagerli:4,earlier:16,easi:[3,4,6,10,11,14,17],easier:[2,9],easiest:7,easili:[2,3,5,6,11,13,16],ec2:[2,6],echo:[3,16],ecosystem:10,ecs:[2,8],ecs_oper:[2,8],ecsoper:[2,6],edit:[4,17],editori:9,effect:[2,3],effici:[2,8],effort:6,egg:4,either:[2,3,4,8,9,13,14,15],elabor:9,elaps:2,elast:2,electron:9,eleg:6,element:4,els:15,elsewher:3,email:[2,3,8,12,13,14,16],email_on_failur:[2,8,13,16],email_on_retri:[2,8,13,16],emailoper:[2,3],emit:[3,5,16],empti:[2,3,8],emr:6,emr_add_steps_oper:8,emr_conn_id:8,emr_connect:8,emr_create_job_flow_oper:8,emr_default:8,emr_hook:8,emr_terminate_job_flow_oper:8,emraddstepsoper:6,emrcreatejobflowoper:6,emrhook:6,emrterminatejobflowoper:6,enabl:[0,
2,3,4,6,7,8],enclos:9,encod:[2,4],encrypt:[2,4,6,7,14],encrypt_s3_log:4,end:[2,8,9,13],end_dat:[1,2,13,16],endors:6,endpoint:[2,6,8],endpoint_url:0,enforc:[3,5],engin:[2,5,6],ensur:[3,5,14],enterpris:6,entir:[1,3,5],entiti:[8,9],entity_filt:8,entri:[2,13,14],env:[2,3,4],environ:[2,3,4,6,7,11,13,16,17],ep_data_pipeline_db_msg_agg:3,equal:[2,8],equival:[2,3,7,16],era:5,error:[1,2,8],error_logfil:1,especi:[3,5,16],essenti:2,etc:[2,3,4,14],europ:8,eval:[2,8],evalu:[2,3,5,8,13,16],even:[2,3,9,11],event:[3,9,16],eventlet:1,eventu:8,ever:2,everi:[2,3,8,13,14,16],everyon:16,everyth:[16,17],exact:1,exactli:[2,8,17],examin:13,exampl:[2,3,4,6,8,9,13,14,15],example1:15,example_bash_oper:[8,15],example_dag:[3,8,13,16],exce:2,except:[2,3,5,8,9,14,16],exchang:[3,6],exclud:[1,9],exclude_subdag:1,exclus:[9,13],exec_d:1,execut:[1,2,3,4,5,6,8,9,13,16],execution_d:[1,2,3,4,13,16],execution_date_fn:2,execution_delta:2,execution_tim:3,execution_timeout:[2,3],executor:[1,3,4,5,6,8,10,13],exercis:9,exhaust
:4,exist:[0,1,2,3,4,8,14,16],existing_cluster_id:[2,8],exit:[1,14],expandus:8,expans:4,expect:[0,2,3,5,6,8],experiment:[6,14],expir:2,expiration_d:2,explain:2,explan:16,explicit:6,explicit_op:3,explicitli:[3,9,14,16],exponenti:2,export_dir:2,export_format:[2,8],export_to_storage_bucket:8,exportingmultipl:[2,8],expos:[0,2,3,4,10,14],express:[2,5,8,9,13],ext:2,extend:[2,6],extens:[2,6,8],extern:[2,3,5,6,10],external_dag_id:2,external_task_id:2,external_trigg:2,externaltasksensor:2,extra:[1,2,3,6,8,14],extra_dejson:2,extra_opt:2,extract:[2,8],extract_queri:2,fact:3,factori:3,fail:[1,2,3,4,5,8,13,16,17],failur:[2,4,6,9],fair:2,fairli:15,fals:[1,2,4,8,13,16],familiar:[3,16],faq:[2,6],fast:15,fault:[2,3],featur:[2,3,4,7,10,16,17],feb:14,fee:9,fernet:4,fernet_kei:4,fetch:[2,4,8],few:[2,4,11,15,16],field:[1,2,8,9],field_delimit:[2,8],field_dict:[2,8],fields_terminated_bi:2,fifti:9,figur:[2,3,13],file1:2,file2:2,file:[0,1,2,3,4,6,8,9,10,13,14,15],file_path:[2,8],file_s:2,file_to_wasb:[2,8],f
ilenam:[2,3,8],filename_templ:8,filepath:[1,2],files:2,filesystem:[4,8],filetowasboper:[2,6],fill:[2,3,8,11,14],filter:[1,2,3,8,14,16],filter_by_own:14,filter_for_files:2,filter_for_ignored_ext:2,find:[2,4,5,16,17],fine:16,finish:[2,8],fire:[2,3,4,5],first:[1,2,3,4,5,8,12,13,15,16],fit:[2,6,9],fix:[2,4,13],flag:[2,13],flask:[1,10,11],flask_admin:10,flask_blueprint:10,flask_login:14,flat:2,flavor:10,flexibl:[2,16],flower:[4,6],flower_conf:1,folder:[2,4,10,13,14,15,16],follow:[0,2,3,4,6,8,9,14,16],foo:[2,3,8,16],foo_:5,foo_dag_id:5,foobar:4,forc:[1,2],force_pul:2,foreground:1,foreign:2,form:[2,3,6,9],format:[2,3,4,5,8,9],formatt:8,forward:[2,5],found:[2,4,14],foundat:[6,9,12,15],four:3,framework:[4,10],free:[3,9],frm:2,from:[1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17],from_nam:2,ftp:2,ftp_conn_id:2,ftp_default:2,ftphook:2,full:[2,8,12],full_filepath:2,fulli:[0,2,5,6,14],functiontyp:2,fundament:16,further:[6,15,16],furthermor:14,futur:[2,13],galaxi:10,gantt:[2,6],garag:3,gate:14,gcp:6,gcp
_api:7,gcp_api_base_hook:2,gcp_conn_id:8,gcp_dataflow_bas:8,gcp_dataflow_hook:8,gcp_default:8,gcp_mlengine_hook:8,gcs:8,gcs_download_oper:[2,8],gcs_hook:[2,8],gcs_log_fold:8,gcs_task_handl:8,gcs_to_bq:8,gcstaskhandl:8,gender:2,gener:[1,2,3,4,6,8,9,10,14,16,17],generate_command:2,generate_kei:4,generictransf:2,get:[0,1,2,3,4,6,8,10,11,13,15,16,17],get_active_run:2,get_bucket:2,get_conn:[2,8],get_connect:3,get_cursor:2,get_dag:2,get_dagrun:2,get_databas:2,get_direct_rel:2,get_first:2,get_flat_rel:2,get_kei:2,get_last_dagrun:2,get_metastore_cli:2,get_model:8,get_num_active_run:2,get_num_task_inst:2,get_oper:8,get_pandas_df:[2,8],get_partit:2,get_partitions_by_filt:2,get_partitions_by_nam:2,get_record:2,get_run_d:2,get_servic:[2,8],get_tabl:2,get_task_inst:2,get_template_context:2,get_template_env:2,get_wildcard_kei:2,gevent:1,ghe:6,ghe_oauth:14,ghost:2,git:4,github:[2,6,8,9,12,13,16],github_enterpris:14,github_enterprise_auth:14,githubusercont:2,gitignor:2,gitter:12,give:[2,3,8,9],give
n:[0,2,3,8],glanc:17,global:[2,3,5],goal:16,goe:[2,5,13,16],going:15,good:[3,8],goodby:3,goodwil:9,googl:[2,4,6,7],google_auth:14,google_client_id:14,google_client_secret:14,google_cloud_datastore_default:8,google_cloud_default:8,google_cloud_storage_conn_id:[2,8],google_cloud_storage_default:[2,8],googlecloudbasehook:2,googlecloudstoragedownloadoper:2,googlecloudstoragehook:[2,6],govern:9,gpig:8,grab:2,grade:4,grai:2,grant:9,graph:[2,3,6],great:[3,4,13],greater:[2,8],green:2,grossli:9,group:[2,4,14],group_member_attr:14,groupmembership:14,guid:[2,8],gunicorn:[1,7],had:[2,4,13],hadoop:[2,6,7,8],hadoop_dependency_coordin:2,hadoopcmd:2,hand:[14,15],handl:[0,2,3,4,8,13],handler:[2,8],happen:[3,13,16],hard:[2,3],harmless:9,has:[1,2,3,4,5,6,8,9,10,13,14,16],has_dag:2,hash:[2,14],hasn:[2,3,8,13],have:[1,2,3,4,5,6,7,8,9,10,11,13,14,16],haven:[2,3,16],hdf:[2,3,7],hdfs_conn_id:2,hdfs_default:2,hdfs_hook:2,hdfshook:2,hdfssensor:2,head:[15,16],header:[2,8],heartbeat:[2,3,5],heavier:2,heavili:2
,hello:[2,3,10,16],help:[1,9,10,14],here:[2,3,4,5,7,8,10,13,14,15,16,17],herebi:9,herein:9,hex:2,hidden:17,hierarchi:[2,16],high:2,highchart:11,higher:2,highli:4,highlight:16,hipchat:2,hipchat_oper:2,hipchatapioper:2,hipchatapisendroomnotificationoper:2,histori:6,hit:[2,3],hive2mysqloper:2,hive2sambaoper:7,hive:[2,3,4,7,8,10,14],hive_cli_conn_id:[2,8],hive_cli_default:[2,8],hive_cli_param:2,hive_conf:2,hive_conn_id:2,hive_t:[2,8],hiveclihook:2,hivecmd:2,hiveconf:2,hiveconf_jinja_transl:2,hivemetastorehook:2,hiveoper:[2,3,4,8,14],hivepartitionsensor:[2,3],hivesensoroper:2,hiveserver2_conn_id:2,hiveserver2_default:2,hiveserver2hook:2,hivetodruidtransf:2,hivetomysqltransf:2,hold:9,holder:0,home:15,homogen:4,hook:[6,7,10,14,16],hope:13,horribl:16,host:[1,2,4,8,14],hostnam:[1,2,3,4,14],hour:[2,3,5,13],hourli:[2,5,13],hous:3,how:[2,3,4,6,9,10,13,16,17],howev:[3,4,9,14],hql:2,html:[2,8,10,16],html_content:2,http:[0,2,3,8,9,10,13,14,16],http_conn_id:2,http_default:2,httphook:2,httpoper:3,ht
tpsensor:2,human:2,icon:2,icon_url:2,idea:[2,4,5,6],idempot:[2,13],identif:9,identifi:[1,2,8,9,16,17],ids:2,ignor:[1,2],ignore_:2,ignore_all_dep:2,ignore_all_depend:1,ignore_copi:2,ignore_depend:1,ignore_depends_on_past:[1,2],ignore_first_depends_on_past:[1,2],ignore_task_dep:[1,2],ignore_ti_st:[1,2],ignored_ext:2,iii:9,imag:2,immedi:[2,3],imper:4,imperson:[2,6,8],implement:[2,3,8],impli:9,import_from_storage_bucket:8,improv:9,impyla:2,inabl:9,inact:[2,5],inc:9,incident:9,includ:[1,2,3,4,8,9,13,16],include_adhoc:[1,2],include_downstream:2,include_exampl:2,include_prior_d:2,include_subdag:2,include_upstream:2,inclus:9,incompatbil:8,incomplet:8,incorpor:9,incremenetd:2,incub:12,incur:9,indemn:9,indemnifi:9,independ:[2,3],index:[2,8],indic:[2,3,6,8,9],indirect:9,individu:[2,9,13,16],ineffici:2,infer:[2,3,8],inferred_op:3,infil:2,infin:6,info:[0,1,2,8],inform:[2,3,4,8,9,13,14,16],infrastructur:[2,6],infring:9,ingest:2,ingress:8,inherit:[2,16],init:4,init_on_load:2,initctl:4,initdb:[4,6,
15],initi:[1,2,4,14,15],initialis:2,inject:10,inlin:2,input:[2,8],input_compress:[2,8],input_format:2,input_path:8,insecur:14,insert:[2,3,8],insert_row:[2,8],insid:[2,3],inspect:[13,15],inspir:2,instal:[2,3,4,6,8,10,14,15],instanc:[1,2,4,5,6,7,8,13,14,15,16],instanti:[2,3,5,6,8,13],instead:[1,2,5,8,13],insteadnhttp:2,institut:9,instruct:[2,3,13],insur:2,int32:[2,8],integ:[2,8],integr:[6,7,10,14],intention:9,inter:3,interact:[2,3,4,8,10,11],interest:[2,16],interfac:[3,6,8,9,11,14,16],intergr:10,intern:[1,8,13],interpret:[2,3,8],interv:[2,5,13],intervalcheckoper:2,intoth:2,introduc:5,intuit:16,invari:[2,3],invok:2,involv:3,ipython:3,irrevoc:9,is_paus:2,is_prematur:2,is_updated_aft:[2,8],ish:2,isn:[2,3,6,8],isoformat:2,issu:[2,3,6,8,9,12],item:[2,3,8,13],iter:[2,8],its:[2,3,4,8,9,10],itself:[2,3,5,13,14,16],januari:[9,13],jar:[2,8],java:[2,8],jdbc:[2,7],jdbcoper:3,jeremiah:12,jinja2:2,jinja:[2,6,8,10],jinja_env:2,jira:12,jlowin:12,job:[1,2,3,4,5,8,13,15,17],job_flow:8,job_flow_id:8,job
_flow_overrid:8,job_id:[2,8],job_nam:8,jobflow:8,jobid:8,jobsclusterspecnewclust:[2,8],jobsnotebooktask:[2,8],jobssparkjartask:[2,8],jobtrack:2,join:[3,12],json:[1,2,3,8],json_client:0,june:12,jupo:2,jupyt:3,just:[1,2,3,5,8,10,13,16],kadmin:14,kapil:4,keep:[2,3,4,6,16],keepalives_idl:2,kei:[1,2,3,4,8,13,14,17],kerber:[7,14],kerbero:[0,6,7],kerberos_auth:0,kernel:2,key1:2,keytab:[0,1,14],keyword:[2,8],kick:[4,13],kill:[2,3,5],kill_zombi:2,kind:9,know:[4,5,16],knowledg:10,known:[2,7],kruyk:12,kwarg:[2,8],l447:[2,8],label:[2,8,11],lai:15,lambda:[2,3,16],land:[2,10],landing_tim:2,languag:[2,9,16],larg:[2,8],last:2,late:17,later:11,latest:[1,2,6,7,8,16],latest_d:2,latest_execution_d:2,latest_onli:3,latest_only_oper:3,latest_only_with_trigg:3,latestonlyoper:3,latin_stat:2,launch:[8,14],law:9,lawsuit:9,layer:5,layout:16,ldap3:14,ldap:[6,7],ldap_auth:14,ldap_ca:14,lead:[2,3],lean:6,least:[2,3,4,9],leav:[2,8],led:3,left:[3,14],legaci:[2,5,8],legal:9,len:2,leo:12,less:[2,8],let:[1,3,4,11,13,1
6,17],level:[2,3,6,8,13,14],leverag:[2,3,16],liabil:9,liabl:9,lib:[2,3,4,8],libffi:4,librari:[2,3,4,5,6,8,10,16],libz:3,licens:[6,14],licensor:9,life:3,lifetim:13,light:3,lighter:2,lightweight:3,like:[2,3,4,5,7,8,10,14,16],limit:[1,2,3,4,6,8,9,13,15],line:[2,6,8,13,15],link:[3,6,9,10],linux:14,list:[1,2,3,4,6,7,8,9,10,12,14,16,17],list_dag:[5,6,16],list_directori:2,list_kei:2,list_prefix:2,list_task:[5,6,16],list_vers:8,listen:[3,5,14],lit:2,liter:2,litig:9,live:2,load:[1,2,3,4,8],load_df:2,load_fil:[2,8],load_opt:[2,8],load_str:2,load_test_config:4,local:[1,2,3,4,5,8,14,16],local_full_path_or_buff:2,local_infil:2,localexecutor:[1,2,3,4,5,13],localhost:3,locat:[1,2,3,4,8,13,14,16],lock:2,lock_for_upd:2,log:[1,2,6,10,14,16],log_config:8,log_fil:1,logfil:1,logger:8,logging_config:8,logging_config_class:8,logging_mixin:2,loggingmixin:2,logic:[2,3,16],login:[1,2,3,4,8,14],logist:10,longer:[2,5,8,13],look:[1,2,3,5,6,8,10,12,13,14,16],lookup:8,loss:9,lot:13,low:2,lowin:12,machin:[1,2,3,4,
14],macro:[3,5,6,10,11,16],made:[2,3,9],mai:[2,3,4,5,8,9,13,16],mail:[9,12],main:[2,3,5,8,10],main_class:8,main_dag:3,main_jar:8,maintain:6,make:[0,2,3,4,5,6,8,9,14,16,17],malfunct:9,malik:4,man:7,manag:[2,6,8,9,10,14],managedlibrarieslibrari:[2,8],mani:[1,2,3,5,10,16,17],manner:6,manual:[2,3,4,13],map:[2,8],mapr:2,mapred_job_nam:2,mapred_queu:2,mapred_queue_prior:2,march:12,mark:[1,2,3,9,13],mark_success:[1,2,13],markdown:3,markup:2,master:[2,3,8,13,16],match:[1,2,3,8],matter:2,max:[2,8,12],max_active_run:[1,2,5],max_bad_record:8,max_id_kei:8,max_ingestion_tim:2,max_partit:2,max_retry_delai:2,max_tri:13,max_worker_count:8,maxim:12,maximum:[1,2,8],maximum_billing_ti:[2,8],maxnumwork:8,maxresult:[2,8],mayb:3,mean:[2,3,4,5,8,9,14,16],meaning:2,meant:[2,5],mechan:[0,2,9,14],media:9,medium:9,meet:[2,9],mem_limit:2,member:14,memberof:14,memori:2,menu:[3,5,6,10,15],menu_link:10,menulink:10,merchant:9,mere:9,merg:[2,8,16],meso:[2,6],mesos_executor:2,mesosexecutor:[2,4,13],messag:[2,3,6],me
ssage_format:2,met:[2,4,5,13],metadata:[1,2,3,4,6,7,8,10,13,17],metastor:2,metastore_conn_id:2,metastore_default:2,metastore_mysql:2,metastorepartitionsensor:2,method:[2,3,5,8],metric:[2,8,10],metric_spec:2,metrics_threshold:[2,8],metrics_threshold_dict_kei:8,microsoft:[6,7],midnight:[5,13],might:[2,3,5,8,13,16],migrat:14,mime:[2,8],mime_subtyp:2,mime_typ:[2,8],min:5,mind:[3,6],minim:8,minimum:[7,13],minor:8,minut:[3,5,8,13,16],mismatch:3,miss:[2,3],mistercrunch:12,mix:[2,3,16],mkdir:3,mlengin:8,mlengine_oper:8,mlsd:2,mode:[1,2,3,6,8,16],model:[3,6,8,10,14,16],model_nam:8,modif:9,modifi:[2,3,4,8,9,11],modul:[2,3,5,6,8,10,14],modular:6,moment:5,monitor:[2,3,4,6,10,13,17],monospac:3,month:[5,13],monthli:[5,13],more:[2,3,4,6,8,9,10,12,13,14,16,17],moreov:16,morn:13,most:[2,3,8,13,14,16,17],mostli:[6,13],mount:[2,4],move:[2,3,5,6,8],msg:2,mssql:7,mssqloper:3,much:[2,3,8,13],multi:6,multipl:[1,2,3,8],multipli:[2,8],multiprocess:2,must:[2,3,4,5,8,9,14,16],mutat:3,my_conn_str:4,my_dag1:3,m
y_dag2:3,my_dag:3,my_databas:2,my_dict_var:2,my_funct:3,my_job_id:8,my_model:8,my_param:16,my_project:8,my_tabl:2,my_useful_packag:3,my_var:2,my_vers:8,myarg:2,mypackag:14,mys3conn:4,mysql:[2,3,4,7],mysql_conn_id:2,mysql_default:2,mysql_postoper:2,mysql_preoper:2,mysql_tabl:2,mysqldb:2,mysqlhook:2,mysqloper:[2,3,4],mysqltohivetransf:2,name:[2,3,4,6,8,9,10,14,16],namedhivepartitionsensor:2,namespac:[2,5,8,10],nav:14,navig:14,ndfor:16,neccessari:8,necessari:2,necessarili:[6,9],need:[2,3,4,5,6,7,8,10,13,14,15,16],neg:[2,13],neglig:9,neighbour:2,nest:[2,16],network:[2,4,14],network_mod:2,never:[1,2,5],new_clust:[2,8],new_user_email:14,new_user_nam:14,newli:[6,8],next:[2,3,4,6,13,14],next_execution_d:2,next_retry_datetim:2,nhere:2,nice:3,night:3,nlst:2,no_confirm:1,node:[1,2,3],non:[1,2,4,9],none:[2,3,8,10,13],nopasswd:14,noram:14,norandkei:14,normal:[2,3,8,9],normalize_schedul:2,notat:[2,8],note:[0,2,4,5,6,8,13,14,16],note_id:2,notebook:[2,3,8],notebook_path:[2,8],notebook_run:[2,8],not
ebook_task:[2,8],noth:[2,9],notic:[3,9,16],notif:2,notifi:2,notwithstand:9,now:[2,3,4,5,13,14,16],nuanc:3,num:2,num_run:1,num_shard:2,num_work:[2,8],number:[1,2,3,4,6,8],oauth2:14,oauth2callback:14,oauth:6,oauth_callback_rout:14,oauth_key_from_github_enterpris:14,oauth_secret_from_github_enterpris:14,obj:2,object:[1,2,3,4,5,8,9,10,13,16],objectclass:14,oblig:9,observ:2,obtain:9,occur:[2,3,8],occurr:2,octet:[2,8],octob:12,off:[2,4,13],offer:[9,10],offici:[2,12],offset:2,often:[2,3],old:8,omit:[2,8],on_failure_callback:2,on_kil:2,on_retry_callback:2,on_success_callback:2,onboard:5,onc:[2,3,4,5,13,14,16],one:[2,3,4,5,6,8,9,13,14,16],one_fail:[2,3],one_success:[2,3],ones:[2,4,5,8,17],onli:[0,1,2,4,5,6,7,8,9,13,14,15],only_fail:[1,2],only_if_upd:2,only_run:[1,2],oozi:6,op1:3,op2:3,op3:3,op4:3,op_arg:2,op_kwarg:2,open:[0,2,3,4,12,14],oper:[1,4,5,6,7,10,13,16],oppos:[2,3],optim:2,option:[1,2,3,6,8,11,13,14,15,16],oracleoper:3,orchestr:[2,6],order:[2,3,4,13,14],org:[2,8,9,10,12,16],organ:[3
,10],organiz:14,orient:2,origin:[8,9],orm:2,other:[1,2,3,4,6,8,9,11,13,16],othercas:2,otherwis:[2,8,9,16],our:[2,8,16],out:[2,3,5,6,8,9,13,15],outag:10,outgrow:15,outlier:17,output:[2,8,16],output_encod:2,output_format:2,output_path:8,outsid:13,outstand:9,over:[1,2,3,4,8,15,17],overlap:17,overlord:2,overrid:[1,2,5,8,16],overridden:[2,5,8],overview:17,overwhelm:3,overwrit:[2,4],own:[2,3,4,6,9,13,16],owner:[2,3,8,9,13,14,16],ownership:9,packag:[2,4,6,8],package1:3,package_uri:8,page:[2,3,5,8,9,12,17],pai:[2,5],pair:[2,17],panda:[2,8],pandas_kwarg:2,parallel:[2,3,4,8,15],param:[1,2,8,16],paramet:[1,2,3,4,5,8,11,14,16],parameter:[3,6,11],parent:[2,3,5],parent_dag:[2,3],parent_dag_nam:3,parent_modul:2,pars:[2,5,10,16],part:[2,3,9,11,14],parti:9,partial:8,partialkei:8,particular:[3,8,9],partit:[2,3,8],partition_nam:2,partition_spec:2,partitiontyp:8,partli:5,pass:[2,3,5,8,10,16],pass_valu:[2,8],passwd:17,password:[1,2,3,4,6,7,8,17],password_auth:14,passwordus:14,past:[2,3,5,13,17],patent:9
,path2:2,path:[1,2,3,4,8,14],patrick:12,patrickleotardif:12,pattern:[2,3,8],paus:[2,3,6],payload:[2,8],pem:2,peopl:[10,16],pep:[2,8],per:8,percent:[8,9],perfect:3,perform:[1,2,3,4,5,6,8,9,13,17],perhap:[3,17],period:[2,3,5,13,16],perman:[2,8],permiss:[9,14],perpetu:9,persist:[2,8,13],perspect:[1,3],pertain:9,phase:10,physic:2,pick:[3,4,11,13],pickl:[1,2,3,4],picklabl:2,pickle_id:2,pickleabl:2,pid:[1,15],pig:[2,3,8],pigcmd:2,piggi:10,pigoper:2,pin_100:2,pip:[3,4,7,15],pipelin:[2,3,4,6,8,11,17],pk1:2,pk2:2,place:[0,2,3,4,5,9,16],placehold:2,plain:[2,4],plan:5,platform:[2,3,6,7],pleas:[0,3,4,8,12,14,16],plug:10,plugin:6,plugin_macro:10,pluginexecutor:10,pluginhook:10,pluginoper:10,plugins_manag:10,png:2,point:[2,3,4,5,8,15,16],poke:2,poke_interv:2,polici:6,poll:[2,8],poll_operation_until_don:8,polling_interval_in_second:8,polling_period_second:[2,8],pool:[2,5,6,16],pool_descript:1,pool_ful:2,popul:[8,16],port:[0,1,2,4,14,15],portion:8,posit:[2,6,8],possibl:[1,2,3,4,8,9,13,14,15,16,17],
post:[0,2],post_execut:2,postgr:[2,3,4,7],postgres_conn_id:2,postgres_default:2,postgres_mast:3,postgreshook:2,postgresoper:[2,3],potenti:3,power:[3,6,9,16],practic:[2,8],pre:5,pre_execut:[2,3],preced:[2,4,8,16],predict:8,prefer:[9,13,15],prefix:[2,3,4,8],preoper:2,prepar:[3,9],prepare_templ:2,preparedata:[2,8],prepped_request:2,present:[2,8],preset:13,presto:2,presto_conn_id:2,presto_default:2,prestocheckoper:2,prestocmd:2,prestohook:2,prestointervalcheckoper:2,prestotomysqloper:3,prestovaluecheckoper:2,pretti:16,prev_execution_d:2,prevent:[2,3,8,14],previou:[1,2,3,5,16],previous:[3,5],previous_ti:2,price:[2,8],primari:2,primit:2,princip:[0,1,3,14],print:[1,2,3,4,8,9,16],print_dat:16,print_head:[2,8],prior:[2,8],priorit:3,prioriti:[2,3],priority_weight:[2,3,16],privaci:9,privat:2,privileg:14,probabl:[0,4,5,14],problem:3,problemat:3,proce:[2,5],process:[1,2,3,4,6,8,13,16],process_fil:2,produc:[2,8],product:[2,3,4,6,8,9,11,13,16],profil:[2,6,14],program:[2,8,12],programmat:6,progress
:[2,6,8,16],project:[2,6,8],project_id:[2,8],promin:9,prompt:1,propag:[2,3],propaget:2,proper:16,properli:[4,5,14],properti:[2,3,5,14],protocol:8,provid:[2,3,4,8,9,11,14,16,17],provide_context:[2,3],proxy_us:14,proxyus:14,psycopg2:[2,4],publicli:[0,9],publish:[2,8],pull:[2,3,8],pull_funct:3,puppet:4,pure:3,purpl:2,purpos:[9,16],push:[2,3,8,10],push_funct:3,pushing_task:3,put:[2,3,5,8,10],pv1:2,pv2:2,pwd:3,py_fil:8,py_opt:8,pydata:[2,8],pyfil:8,pyhiv:2,pypi:[7,15],pyspark:8,python:[2,3,4,5,6,7,8,10,14,16],python_1:3,python_cal:[2,3],python_oper:2,pythonhost:10,pythonoper:[2,3],pythonpath:[4,8,14],qds:[2,7],qualif:14,qualifi:[0,2,14],qualiti:[2,3,8],qubol:[2,7],qubole_conn_id:2,qubole_default:2,quboleoper:2,queri:[2,4,6,8],query_granular:2,query_param:[2,8],query_uri:8,queu:[1,3,5],queue:[1,2,4,5,6,16],quick:[4,6,17],quickli:[11,15,16,17],quit:5,quote_charact:8,r39132:12,rabbitmq:[4,7],rais:[2,4,8,16],ran:[1,2],randkei:14,random:2,randomli:3,rang:[1,2,3,5,13,16],rate:[2,8],rather:3,ra
tio:[2,8],raw:[2,5,8],reach:[1,2,3,4,5,8],reactiv:5,read:[2,4,5,8,14,16],read_consist:8,read_kei:2,readabl:[2,9,14],readi:[2,6,15],readm:4,readthedoc:[2,8,16],ready_for_retri:2,real:[0,4,8],realli:[14,16],realm:[0,14],reason:[5,9,16],reboot:3,rebuild:1,recap:6,receiv:[2,3,8,9,13],recent:[2,3],recipi:[2,9],recogn:[2,8],recommend:[0,2,4,5,8,9],record:[1,2,3,13,16],recreat:[2,8],recurs:2,red:2,redhat:4,redi:[4,7],redirect:[1,14],redistribut:9,redshift:[2,6],redshift_conn_id:8,redshift_default:8,redshift_to_s3_oper:8,redshifttos3transf:6,redund:16,refer:[3,4,6,8,10,12,16],referenc:[3,16],reflect:[3,6,8,10,16],refrain:3,refresh:2,refresh_from_db:2,regard:[9,16],regardless:[1,2],regex:[1,2],regex_kei:2,regexp:1,region:[2,8],region_nam:[2,8],regist:[10,11,14,16],registri:2,regular:[2,13],reinit_frequ:14,rel:[2,13,16],relat:[2,3,4,5,7,12],relationship:3,relativedelta:2,releas:8,relev:5,reli:[2,14],remain:[9,14],rememb:16,remot:[2,4,5,8,13],remote_base_log_fold:4,remote_full_path:2,remote_lo
g_conn_id:[4,8],remov:2,renam:[2,8],render:[2,3,6,8,10,16],render_templ:2,render_template_from_field:2,renew:[1,14],repeat:[3,13],repl:14,replac:[2,4,8,9,14],report:[1,2],repositori:[4,16],repres:[2,3,5,8,9,13],represent:[2,17],reproduc:9,reproduct:9,request:[1,2,3,8],request_param:2,requir:[0,1,2,3,4,5,6,7,8,9,14],rerun:1,resembl:3,reset:1,reset_dag_run:2,resetdb:6,resid:[2,3,8],resolv:[2,8],resourc:[1,2,3,6,8],respect:[1,2,3,16],respons:[2,8,9],response_check:2,rest:[2,6,8,14],restart:[3,4,8],restrict:14,restructuredtext:3,result:[2,3,8,9,11,16],resum:1,retain:9,retri:[1,2,3,8,13,16],retriev:[2,3,8],retrieve_fil:2,retry_delai:[2,8,13,16],retry_exponential_backoff:2,return_valu:2,reus:[8,10],revers:4,review:6,revis:9,riccomini:12,rich:[1,3,6,17],rid:2,right:[2,3,9,11],rightmost:3,risk:9,roadmap:6,role:4,roll:[6,8],rollback:8,room:2,room_id:2,root:[2,3,14],root_dag:2,round:[2,5],rout:14,routin:[2,3],row:[2,3,8],royalti:9,rule:[2,5,6,16],run:[2,4,6,8,14,15,17],run_a:2,run_and_check:2
,run_as_own:[2,14],run_as_us:[2,14],run_cli:2,run_dur:1,run_id:[1,2,13],run_nam:[2,8],run_queri:8,run_task:[2,8],run_this_last:8,runme_0:15,runnabl:3,runqueri:8,runtim:8,runtime_vers:8,rwc:14,s3_bucket:8,s3_default:8,s3_file_transform_oper:8,s3_hook:8,s3_kei:[2,8],s3_to_hive_oper:8,s3distcp:2,s3filetransferoper:3,s3filetransformoper:6,s3hook:[2,4,6],s3keysensor:[2,7],s3prefixsensor:7,s3tohivetransf:[2,6],safe:8,safe_mod:2,sai:[2,3,5,8,13],said:3,samaccountnam:14,samba:7,same:[2,3,4,8,9,11,14,16],sampl:[2,4],sample_s:2,sandbox:4,sanit:2,sas_token:8,save:[2,4,8,11,16],savedmodel:8,scala2:[2,8],scala:2,scalabl:6,scale:6,scale_ti:8,scaleti:8,scan:[2,3],scene:[7,13],schedul:[2,3,4,6,8,16],schedule_interv:[2,3,5,13,16],scheduler_interv:5,scheduler_run:4,schema:[1,2,3,4,8],schema_field:8,schema_object:8,schema_update_opt:8,scope:6,scratch:16,screen:16,screenshot:6,script:[2,3,4,5,6,8],script_begin_tag:2,script_loc:2,search:[1,2,3],search_path:4,search_scop:14,second:[1,2,3,5,8,16],secret:[
14,17],secretkei:3,section:[2,3,4,5,8,9,14,15,16],secur:[2,3,4,6,9],see:[2,3,4,5,8,9,13,14,15,16,17],seen:[4,17],segment:2,segment_granular:2,select:[2,3,8,11,13,14],self:[2,8,10],sell:9,semicolon:2,send:[1,2,3,12],send_room_notif:2,sender:2,sensor:[2,3,5,7,8],sensor_queu:3,sent:[2,3,4,9],separ:[0,2,3,9,16],sequenti:[2,13,15],sequentialexecutor:[2,3,8,15],seri:[3,13],serial:[1,2,8],serv:[1,2,4,8,16],serve_log:[4,6],server:[1,2,4,10,14,15,16],servic:[0,1,2,3,4,6,7,9,13,14],session:[2,14],set:[0,1,2,3,5,6,8,10,13,17],set_default_vers:8,set_depend:[2,16],set_downstream:[2,3,16],set_the_password:14,set_upstream:[2,3,16],setgid:4,setuid:4,setup:[4,14],sever:3,shall:[2,9],shape:13,share:[2,3,4,9,10,11],shell:[2,14],shellcmd:2,ship:[1,2],ship_dag:1,shop:9,shortcircuitoper:2,shortcut:[4,17],should:[0,2,3,4,5,8,9,10,13,14,15,16],shouldn:[2,3],show:[1,2,3,5,8,10,17],shown:2,sid:12,siddharth:12,side:[2,4,8],signal:[2,5],similar:
<TRUNCATED>
[08/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/cli.html
----------------------------------------------------------------------
diff --git a/cli.html b/cli.html
index 60008af..5e422ad 100644
--- a/cli.html
+++ b/cli.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -91,7 +96,118 @@
<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
-<li class="toctree-l1 current"><a class="current reference internal" href="#">Command Line Interface</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Command Line Interface</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#Positional Arguments">Positional Arguments</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#Sub-commands:">Sub-commands:</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#resetdb">resetdb</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#render">render</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat1">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat1">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#variables">variables</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat2">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#connections">connections</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat3">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#pause">pause</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat2">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat4">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#task_failed_deps">task_failed_deps</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat3">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat5">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#version">version</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#trigger_dag">trigger_dag</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat4">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat6">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#initdb">initdb</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#test">test</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat5">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat7">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#unpause">unpause</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat6">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat8">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#dag_state">dag_state</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat7">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat9">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#run">run</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat8">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat10">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#list_tasks">list_tasks</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat9">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat11">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#backfill">backfill</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat10">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat12">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#list_dags">list_dags</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat13">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#kerberos">kerberos</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat11">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat14">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#worker">worker</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat15">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#webserver">webserver</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat16">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#flower">flower</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat17">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#scheduler">scheduler</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat18">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#task_state">task_state</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat12">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat19">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#pool">pool</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat20">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#serve_logs">serve_logs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#clear">clear</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#Positional Arguments_repeat13">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#Named Arguments_repeat21">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#upgradedb">upgradedb</a></li>
+</ul>
+</li>
+</ul>
+</li>
<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
@@ -172,1032 +288,1159 @@ development and testing.</p>
<span class="o">...</span>
</pre></div>
</div>
-<dl class="docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments">
+<h2>Positional Arguments<a class="headerlink" href="#Positional Arguments" title="Permalink to this headline">¶</a></h2>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">subcommand</span></kbd></td>
-<td><p class="first">sub-command help</p>
-<p class="last">Possible choices: resetdb, render, variables, connections, pause, task_failed_deps, version, trigger_dag, initdb, test, unpause, dag_state, run, list_tasks, backfill, list_dags, kerberos, worker, webserver, flower, scheduler, task_state, pool, serve_logs, clear, upgradedb</p>
+<kbd>subcommand</kbd></td>
+<td><p class="first">Possible choices: resetdb, render, variables, connections, pause, task_failed_deps, version, trigger_dag, initdb, test, unpause, dag_state, run, list_tasks, backfill, list_dags, kerberos, worker, webserver, flower, scheduler, task_state, pool, serve_logs, clear, upgradedb</p>
+<p class="last">sub-command help</p>
</td></tr>
</tbody>
</table>
-</dd>
-<dt>Sub-commands:</dt>
-<dd><dl class="first last docutils">
-<dt><strong>resetdb</strong></dt>
-<dd><p class="first">Burn down and rebuild the metadata database</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">resetdb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">y</span><span class="p">]</span>
+</div>
+<div class="section" id="Sub-commands:">
+<h2>Sub-commands:<a class="headerlink" href="#Sub-commands:" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="resetdb">
+<h3>resetdb<a class="headerlink" href="#resetdb" title="Permalink to this headline">¶</a></h3>
+<p>Burn down and rebuild the metadata database</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">resetdb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">y</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Named Arguments">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-y=False</span>, <span class="option">--yes=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Do not prompt to confirm reset. Use with care!</td></tr>
+<tr><td class="option-group">
+<kbd>-y, --yes</kbd></td>
+<td><p class="first">Do not prompt to confirm reset. Use with care!</p>
+<p class="last">Default: False</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>render</strong></dt>
-<dd><p class="first">Render a task instance’s template(s)</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">render</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</div>
+</div>
+<div class="section" id="render">
+<h3>render<a class="headerlink" href="#render" title="Permalink to this headline">¶</a></h3>
+<p>Render a task instance’s template(s)</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">render</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat1">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat1" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">task_id</span></kbd></td>
+<kbd>task_id</kbd></td>
<td>The id of the task</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">execution_date</span></kbd></td>
+<kbd>execution_date</kbd></td>
<td>The execution date of the DAG</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat1">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat1" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>variables</strong></dt>
-<dd><p class="first">CRUD operations on variables</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">variables</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">KEY</span> <span class="n">VAL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">g</span> <span class="n">KEY</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">j</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span> <span class="n">VAL</span><span class="p">]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">i</span> <span class="n">FILEPATH</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">FILEPATH</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">x</span> <span class="n">KEY</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="variables">
+<h3>variables<a class="headerlink" href="#variables" title="Permalink to this headline">¶</a></h3>
+<p>CRUD operations on variables</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">variables</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">KEY</span> <span class="n">VAL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">g</span> <span class="n">KEY</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">j</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span> <span class="n">VAL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">i</span> <span class="n">FILEPATH</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">FILEPATH</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">x</span> <span class="n">KEY</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Named Arguments_repeat2">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat2" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">-s</span>, <span class="option">--set</span></kbd></td>
+<kbd>-s, --set</kbd></td>
<td>Set a variable</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-g</span>, <span class="option">--get</span></kbd></td>
+<kbd>-g, --get</kbd></td>
<td>Get value of a variable</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-j=False</span>, <span class="option">--json=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Deserialize JSON variable</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-d</span>, <span class="option">--default</span></kbd></td>
+<kbd>-j, --json</kbd></td>
+<td><p class="first">Deserialize JSON variable</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-d, --default</kbd></td>
<td>Default value returned if variable does not exist</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-i</span>, <span class="option">--import</span></kbd></td>
+<kbd>-i, --import</kbd></td>
<td>Import variables from JSON file</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-e</span>, <span class="option">--export</span></kbd></td>
+<kbd>-e, --export</kbd></td>
<td>Export variables to JSON file</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-x</span>, <span class="option">--delete</span></kbd></td>
+<kbd>-x, --delete</kbd></td>
<td>Delete a variable</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>connections</strong></dt>
-<dd><p class="first">List/Add/Delete connections</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">connections</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">a</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_id</span> <span class="n">CONN_ID</span><span class="p">]</span>
- <span class="p">[</span><span class="o">--</span><span class="n">conn_uri</span> <span class="n">CONN_URI</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_extra</span> <span class="n">CONN_EXTRA</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="connections">
+<h3>connections<a class="headerlink" href="#connections" title="Permalink to this headline">¶</a></h3>
+<p>List/Add/Delete connections</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">connections</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">a</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_id</span> <span class="n">CONN_ID</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">conn_uri</span> <span class="n">CONN_URI</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_extra</span> <span class="n">CONN_EXTRA</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">conn_type</span> <span class="n">CONN_TYPE</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_host</span> <span class="n">CONN_HOST</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">conn_login</span> <span class="n">CONN_LOGIN</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_password</span> <span class="n">CONN_PASSWORD</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">conn_schema</span> <span class="n">CONN_SCHEMA</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">conn_port</span> <span class="n">CONN_PORT</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Named Arguments_repeat3">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat3" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-l=False</span>, <span class="option">--list=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>List all connections</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-a=False</span>, <span class="option">--add=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Add a connection</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-d=False</span>, <span class="option">--delete=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Delete a connection</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--conn_id</span></kbd></td>
+<kbd>-l, --list</kbd></td>
+<td><p class="first">List all connections</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-a, --add</kbd></td>
+<td><p class="first">Add a connection</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-d, --delete</kbd></td>
+<td><p class="first">Delete a connection</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>--conn_id</kbd></td>
<td>Connection id, required to add/delete a connection</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--conn_uri</span></kbd></td>
-<td>Connection URI, required to add a connection</td></tr>
+<kbd>--conn_uri</kbd></td>
+<td>Connection URI, required to add a connection without conn_type</td></tr>
+<tr><td class="option-group">
+<kbd>--conn_extra</kbd></td>
+<td>Connection <cite>Extra</cite> field, optional when adding a connection</td></tr>
+<tr><td class="option-group">
+<kbd>--conn_type</kbd></td>
+<td>Connection type, required to add a connection without conn_uri</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--conn_extra</span></kbd></td>
-<td>Connection `Extra` field, optional when adding a connection</td></tr>
+<kbd>--conn_host</kbd></td>
+<td>Connection host, optional when adding a connection</td></tr>
+<tr><td class="option-group">
+<kbd>--conn_login</kbd></td>
+<td>Connection login, optional when adding a connection</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd>--conn_password</kbd></td>
+</tr>
+<tr><td> </td><td>Connection password, optional when adding a connection</td></tr>
+<tr><td class="option-group">
+<kbd>--conn_schema</kbd></td>
+<td>Connection schema, optional when adding a connection</td></tr>
+<tr><td class="option-group">
+<kbd>--conn_port</kbd></td>
+<td>Connection port, optional when adding a connection</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>pause</strong></dt>
-<dd><p class="first">Pause a DAG</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">pause</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
+</div>
+</div>
+<div class="section" id="pause">
+<h3>pause<a class="headerlink" href="#pause" title="Permalink to this headline">¶</a></h3>
+<p>Pause a DAG</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">pause</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat2">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat2" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat4">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat4" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>task_failed_deps</strong></dt>
-<dd><p class="first">Returns the unmet dependencies for a task instance from the perspective of the scheduler. In other words, why a task instance doesn’t get scheduled and then queued by the scheduler, and then run by an executor).</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">task_failed_deps</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span>
- <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</div>
+</div>
+<div class="section" id="task_failed_deps">
+<h3>task_failed_deps<a class="headerlink" href="#task_failed_deps" title="Permalink to this headline">¶</a></h3>
+<p>Returns the unmet dependencies for a task instance from the perspective of the scheduler. In other words, why a task instance doesn’t get scheduled and then queued by the scheduler, and then run by an executor).</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">task_failed_deps</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat3">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat3" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">task_id</span></kbd></td>
+<kbd>task_id</kbd></td>
<td>The id of the task</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">execution_date</span></kbd></td>
+<kbd>execution_date</kbd></td>
<td>The execution date of the DAG</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat5">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat5" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>version</strong></dt>
-<dd><p class="first">Show the version</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">version</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="version">
+<h3>version<a class="headerlink" href="#version" title="Permalink to this headline">¶</a></h3>
+<p>Show the version</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">version</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-</dl>
-</dd>
-<dt><strong>trigger_dag</strong></dt>
-<dd><p class="first">Trigger a DAG run</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">trigger_dag</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">r</span> <span class="n">RUN_ID</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span> <span class="n">CONF</span><span class="p">]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">EXEC_DATE</span><span class="p">]</span>
- <span class="n">dag_id</span>
+</div>
+<div class="section" id="trigger_dag">
+<h3>trigger_dag<a class="headerlink" href="#trigger_dag" title="Permalink to this headline">¶</a></h3>
+<p>Trigger a DAG run</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">trigger_dag</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">r</span> <span class="n">RUN_ID</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span> <span class="n">CONF</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">EXEC_DATE</span><span class="p">]</span>
+ <span class="n">dag_id</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat4">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat4" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat6">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat6" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-r</span>, <span class="option">--run_id</span></kbd></td>
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-r, --run_id</kbd></td>
<td>Helps to identify this run</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-c</span>, <span class="option">--conf</span></kbd></td>
-<td>JSON string that gets pickled into the DagRun’s conf attribute</td></tr>
+<kbd>-c, --conf</kbd></td>
+<td>JSON string that gets pickled into the DagRun’s conf attribute</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-e</span>, <span class="option">--exec_date</span></kbd></td>
+<kbd>-e, --exec_date</kbd></td>
</tr>
<tr><td> </td><td>The execution date of the DAG</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>initdb</strong></dt>
-<dd><p class="first">Initialize the metadata database</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">initdb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="initdb">
+<h3>initdb<a class="headerlink" href="#initdb" title="Permalink to this headline">¶</a></h3>
+<p>Initialize the metadata database</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">initdb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-</dl>
-</dd>
-<dt><strong>test</strong></dt>
-<dd><p class="first">Test a task instance. This will run a task without checking for dependencies or recording it’s state in the database.</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">test</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">dr</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">tp</span> <span class="n">TASK_PARAMS</span><span class="p">]</span>
- <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</div>
+<div class="section" id="test">
+<h3>test<a class="headerlink" href="#test" title="Permalink to this headline">¶</a></h3>
+<p>Test a task instance. This will run a task without checking for dependencies or recording it’s state in the database.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">test</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">dr</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">tp</span> <span class="n">TASK_PARAMS</span><span class="p">]</span>
+ <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat5">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat5" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">task_id</span></kbd></td>
+<kbd>task_id</kbd></td>
<td>The id of the task</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">execution_date</span></kbd></td>
+<kbd>execution_date</kbd></td>
<td>The execution date of the DAG</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat7">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat7" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-dr, --dry_run</kbd></td>
+<td><p class="first">Perform a dry run</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-dr=False</span>, <span class="option">--dry_run=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Perform a dry run</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-tp</span>, <span class="option">--task_params</span></kbd></td>
+<kbd>-tp, --task_params</kbd></td>
</tr>
<tr><td> </td><td>Sends a JSON params dict to the task</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>unpause</strong></dt>
-<dd><p class="first">Resume a paused DAG</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">unpause</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
+</div>
+</div>
+<div class="section" id="unpause">
+<h3>unpause<a class="headerlink" href="#unpause" title="Permalink to this headline">¶</a></h3>
+<p>Resume a paused DAG</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">unpause</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat6">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat6" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat8">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat8" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>dag_state</strong></dt>
-<dd><p class="first">Get the status of a dag run</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">dag_state</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">execution_date</span>
+</div>
+</div>
+<div class="section" id="dag_state">
+<h3>dag_state<a class="headerlink" href="#dag_state" title="Permalink to this headline">¶</a></h3>
+<p>Get the status of a dag run</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">dag_state</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">execution_date</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat7">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat7" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">execution_date</span></kbd></td>
+<kbd>execution_date</kbd></td>
<td>The execution date of the DAG</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat9">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat9" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>run</strong></dt>
-<dd><p class="first">Run a single task instance</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">run</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">m</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">f</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pool</span> <span class="n">POOL</span><span class="p">]</span>
- <span class="p">[</span><span class="o">--</span><span class="n">cfg_path</span> <span class="n">CFG_PATH</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">A</span> <span class="n">IGNORE_ALL_DEPENDENCIES</span><span class="p">]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">i</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">I</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">ship_dag</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PICKLE</span><span class="p">]</span>
- <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</div>
+</div>
+<div class="section" id="run">
+<h3>run<a class="headerlink" href="#run" title="Permalink to this headline">¶</a></h3>
+<p>Run a single task instance</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">run</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">m</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">f</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pool</span> <span class="n">POOL</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">cfg_path</span> <span class="n">CFG_PATH</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">A</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">i</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">I</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">ship_dag</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PICKLE</span><span class="p">]</span>
+ <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat8">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat8" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">task_id</span></kbd></td>
+<kbd>task_id</kbd></td>
<td>The id of the task</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">execution_date</span></kbd></td>
+<kbd>execution_date</kbd></td>
<td>The execution date of the DAG</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat10">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat10" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-m=False</span>, <span class="option">--mark_success=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Mark jobs as succeeded without running them</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-f=False</span>, <span class="option">--force=False</span></kbd></td>
+<kbd>-m, --mark_success</kbd></td>
</tr>
-<tr><td> </td><td>Ignore previous task instance state, rerun regardless if task already succeeded/failed</td></tr>
+<tr><td> </td><td><p class="first">Mark jobs as succeeded without running them</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--pool</span></kbd></td>
+<kbd>-f, --force</kbd></td>
+<td><p class="first">Ignore previous task instance state, rerun regardless if task already succeeded/failed</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>--pool</kbd></td>
<td>Resource pool to use</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--cfg_path</span></kbd></td>
+<kbd>--cfg_path</kbd></td>
<td>Path to config file to use instead of airflow.cfg</td></tr>
+<tr><td class="option-group">
+<kbd>-l, --local</kbd></td>
+<td><p class="first">Run the task using the LocalExecutor</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-l=False</span>, <span class="option">--local=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Run the task using the LocalExecutor</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-A</span>, <span class="option">--ignore_all_dependencies</span></kbd></td>
-</tr>
-<tr><td> </td><td>Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_depsstore_true</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-i=False</span>, <span class="option">--ignore_dependencies=False</span></kbd></td>
+<kbd>-A, --ignore_all_dependencies</kbd></td>
</tr>
-<tr><td> </td><td>Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies</td></tr>
+<tr><td> </td><td><p class="first">Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_deps</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-I=False</span>, <span class="option">--ignore_depends_on_past=False</span></kbd></td>
+<kbd>-i, --ignore_dependencies</kbd></td>
</tr>
-<tr><td> </td><td>Ignore depends_on_past dependencies (but respect upstream dependencies)</td></tr>
+<tr><td> </td><td><p class="first">Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">--ship_dag=False</span></kbd></td>
+<kbd>-I, --ignore_depends_on_past</kbd></td>
</tr>
-<tr><td> </td><td>Pickles (serializes) the DAG and ships it to the worker</td></tr>
+<tr><td> </td><td><p class="first">Ignore depends_on_past dependencies (but respect upstream dependencies)</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-p</span>, <span class="option">--pickle</span></kbd></td>
+<kbd>--ship_dag</kbd></td>
+<td><p class="first">Pickles (serializes) the DAG and ships it to the worker</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-p, --pickle</kbd></td>
<td>Serialized pickle object of the entire dag (used internally)</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>list_tasks</strong></dt>
-<dd><p class="first">List the tasks within a DAG</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">list_tasks</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
+</div>
+</div>
+<div class="section" id="list_tasks">
+<h3>list_tasks<a class="headerlink" href="#list_tasks" title="Permalink to this headline">¶</a></h3>
+<p>List the tasks within a DAG</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">list_tasks</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat9">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat9" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat11">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat11" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-t=False</span>, <span class="option">--tree=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Tree view</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd>-t, --tree</kbd></td>
+<td><p class="first">Tree view</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>backfill</strong></dt>
-<dd><p class="first">Run subsections of a DAG for a specified date range</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">backfill</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">TASK_REGEX</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">START_DATE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">END_DATE</span><span class="p">]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">m</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">x</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">a</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">i</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">I</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span>
- <span class="p">[</span><span class="o">--</span><span class="n">pool</span> <span class="n">POOL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">dr</span><span class="p">]</span>
- <span class="n">dag_id</span>
+</div>
+</div>
+<div class="section" id="backfill">
+<h3>backfill<a class="headerlink" href="#backfill" title="Permalink to this headline">¶</a></h3>
+<p>Run subsections of a DAG for a specified date range</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">backfill</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">TASK_REGEX</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">START_DATE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">END_DATE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">m</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">x</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">a</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">i</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">I</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pool</span> <span class="n">POOL</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">delay_on_limit</span> <span class="n">DELAY_ON_LIMIT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">dr</span><span class="p">]</span>
+ <span class="n">dag_id</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat10">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat10" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
-<kbd><span class="option">dag_id</span></kbd></td>
+<kbd>dag_id</kbd></td>
<td>The id of the dag</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat12">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat12" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-t</span>, <span class="option">--task_regex</span></kbd></td>
+<kbd>-t, --task_regex</kbd></td>
</tr>
<tr><td> </td><td>The regex to filter specific task_ids to backfill (optional)</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-s</span>, <span class="option">--start_date</span></kbd></td>
+<kbd>-s, --start_date</kbd></td>
</tr>
<tr><td> </td><td>Override start_date YYYY-MM-DD</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-e</span>, <span class="option">--end_date</span></kbd></td>
+<kbd>-e, --end_date</kbd></td>
<td>Override end_date YYYY-MM-DD</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-m=False</span>, <span class="option">--mark_success=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Mark jobs as succeeded without running them</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-l=False</span>, <span class="option">--local=False</span></kbd></td>
+<kbd>-m, --mark_success</kbd></td>
</tr>
-<tr><td> </td><td>Run the task using the LocalExecutor</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-x=False</span>, <span class="option">--donot_pickle=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code.</td></tr>
+<tr><td> </td><td><p class="first">Mark jobs as succeeded without running them</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-l, --local</kbd></td>
+<td><p class="first">Run the task using the LocalExecutor</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-a=False</span>, <span class="option">--include_adhoc=False</span></kbd></td>
+<kbd>-x, --donot_pickle</kbd></td>
</tr>
-<tr><td> </td><td>Include dags with the adhoc parameter.</td></tr>
+<tr><td> </td><td><p class="first">Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code.</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-i=False</span>, <span class="option">--ignore_dependencies=False</span></kbd></td>
+<kbd>-a, --include_adhoc</kbd></td>
</tr>
-<tr><td> </td><td>Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex</td></tr>
+<tr><td> </td><td><p class="first">Include dags with the adhoc parameter.</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-I=False</span>, <span class="option">--ignore_first_depends_on_past=False</span></kbd></td>
+<kbd>-i, --ignore_dependencies</kbd></td>
</tr>
-<tr><td> </td><td>Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past).</td></tr>
+<tr><td> </td><td><p class="first">Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+<kbd>-I, --ignore_first_depends_on_past</kbd></td>
</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td> </td><td><p class="first">Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past).</p>
+<p class="last">Default: False</p>
+</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--pool</span></kbd></td>
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>--pool</kbd></td>
<td>Resource pool to use</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-dr=False</span>, <span class="option">--dry_run=False</span></kbd></td>
+<kbd>--delay_on_limit</kbd></td>
</tr>
-<tr><td> </td><td>Perform a dry run</td></tr>
+<tr><td> </td><td><p class="first">Amount of time in seconds to wait when the limit on maximum active dag runs (max_active_runs) has been reached before trying to execute a dag run again.</p>
+<p class="last">Default: 1.0</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-dr, --dry_run</kbd></td>
+<td><p class="first">Perform a dry run</p>
+<p class="last">Default: False</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>list_dags</strong></dt>
-<dd><p class="first">List all the DAGs</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">list_dags</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">r</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="list_dags">
+<h3>list_dags<a class="headerlink" href="#list_dags" title="Permalink to this headline">¶</a></h3>
+<p>List all the DAGs</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">list_dags</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">r</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Named Arguments_repeat13">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat13" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
-</tr>
-<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-r=False</span>, <span class="option">--report=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Show DagBag loading report</td></tr>
+<tr><td class="option-group">
+<kbd>-sd, --subdir</kbd></td>
+<td><p class="first">File location or directory from which to look for the dag</p>
+<p class="last">Default: /Users/chrisr/airflow/dags</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-r, --report</kbd></td>
+<td><p class="first">Show DagBag loading report</p>
+<p class="last">Default: False</p>
+</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>kerberos</strong></dt>
-<dd><p class="first">Start a kerberos ticket renewer</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">kerberos</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">kt</span> <span class="p">[</span><span class="n">KEYTAB</span><span class="p">]]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span>
- <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
- <span class="p">[</span><span class="n">principal</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="kerberos">
+<h3>kerberos<a class="headerlink" href="#kerberos" title="Permalink to this headline">¶</a></h3>
+<p>Start a kerberos ticket renewer</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">kerberos</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">kt</span> <span class="p">[</span><span class="n">KEYTAB</span><span class="p">]]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+ <span class="p">[</span><span class="n">principal</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Required Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Positional Arguments_repeat11">
+<h4>Positional Arguments<a class="headerlink" href="#Positional Arguments_repeat11" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">principal=airflow</span></kbd></td>
-</tr>
-<tr><td> </td><td>kerberos principal</td></tr>
+<tr><td class="option-group">
+<kbd>principal</kbd></td>
+<td><p class="first">kerberos principal</p>
+<p class="last">Default: airflow</p>
+</td></tr>
</tbody>
</table>
-</dd>
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+</div>
+<div class="section" id="Named Arguments_repeat14">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat14" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-kt=airflow.keytab</span>, <span class="option">--keytab=airflow.keytab</span></kbd></td>
-</tr>
-<tr><td> </td><td>keytab</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--pid</span></kbd></td>
+<kbd>-kt, --keytab</kbd></td>
+<td><p class="first">keytab</p>
+<p class="last">Default: airflow.keytab</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>--pid</kbd></td>
<td>PID file location</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Daemonize instead of running in the foreground</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--stdout</span></kbd></td>
+<kbd>-D, --daemon</kbd></td>
+<td><p class="first">Daemonize instead of running in the foreground</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>--stdout</kbd></td>
<td>Redirect stdout to this file</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--stderr</span></kbd></td>
+<kbd>--stderr</kbd></td>
<td>Redirect stderr to this file</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<kbd>-l, --log-file</kbd></td>
<td>Location of the log file</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>worker</strong></dt>
-<dd><p class="first">Start a Celery worker node</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">worker</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">q</span> <span class="n">QUEUES</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span> <span class="n">CONCURRENCY</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="worker">
+<h3>worker<a class="headerlink" href="#worker" title="Permalink to this headline">¶</a></h3>
+<p>Start a Celery worker node</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">worker</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">q</span> <span class="n">QUEUES</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span> <span class="n">CONCURRENCY</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">cn</span> <span class="n">CELERY_HOSTNAME</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
</pre></div>
</div>
-<dl class="last docutils">
-<dt>Optional Arguments</dt>
-<dd><table class="first last docutils option-list" frame="void" rules="none">
+<div class="section" id="Named Arguments_repeat15">
+<h4>Named Arguments<a class="headerlink" href="#Named Arguments_repeat15" title="Permalink to this headline">¶</a></h4>
+<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-p=False</span>, <span class="option">--do_pickle=False</span></kbd></td>
+<kbd>-p, --do_pickle</kbd></td>
</tr>
-<tr><td> </td><td>Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.</td></tr>
+<tr><td> </td><td><p class="first">Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>-q, --queues</kbd></td>
+<td><p class="first">Comma delimited list of queues to serve</p>
+<p class="last">Default: default</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-q=default</span>, <span class="option">--queues=default</span></kbd></td>
+<kbd>-c, --concurrency</kbd></td>
</tr>
-<tr><td> </td><td>Comma delimited list of queues to serve</td></tr>
+<tr><td> </td><td><p class="first">The number of worker processes</p>
+<p class="last">Default: 16</p>
+</td></tr>
<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-c=16</span>, <span class="option">--concurrency=16</span></kbd></td>
+<kbd>-cn, --celery_hostname</kbd></td>
</tr>
-<tr><td> </td><td>The number of worker processes</td></tr>
+<tr><td> </td><td>Set the hostname of celery worker if you have multiple workers on a single machine.</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--pid</span></kbd></td>
+<kbd>--pid</kbd></td>
<td>PID file location</td></tr>
-<tr><td class="option-group" colspan="2">
-<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
-</tr>
-<tr><td> </td><td>Daemonize instead of running in the foreground</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--stdout</span></kbd></td>
+<kbd>-D, --daemon</kbd></td>
+<td><p class="first">Daemonize instead of running in the foreground</p>
+<p class="last">Default: False</p>
+</td></tr>
+<tr><td class="option-group">
+<kbd>--stdout</kbd></td>
<td>Redirect stdout to this file</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">--stderr</span></kbd></td>
+<kbd>--stderr</kbd></td>
<td>Redirect stderr to this file</td></tr>
<tr><td class="option-group">
-<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<kbd>-l, --log-file</kbd></td>
<td>Location of the log file</td></tr>
</tbody>
</table>
-</dd>
-</dl>
-</dd>
-<dt><strong>webserver</strong></dt>
-<dd><p class="first">Start a Airflow webserver instance</p>
-<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">webserver</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PORT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">w</span> <span class="n">WORKERS</span><span class="p">]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">k</span> <span class="p">{</span><span class="n">sync</span><span class="p">,</span><span class="n">eventlet</span><span class="p">,</span><span class="n">gevent</span><span class="p">,</span><span class="n">tornado</span><span class="p">}]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">WORKER_TIMEOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">hn</span> <span class="n">HOSTNAME</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span>
- <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span>
- <span class="p">[</span><span class="o">-</span><span class="n">A</span> <span class="n">ACCESS_LOGFILE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">E</span> <span class="n">ERROR_LOGFILE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
- <span class="p">[</span><span class="o">--</span><span class="n">ssl_cert</span> <span class="n">SSL_CERT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">ssl_key</span> <span class="n">SSL_KEY</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span><span class="p">]</span>
+</div>
+</div>
+<div class="section" id="webserver">
+<h3>webserver<a class="headerlink" href="#webserver" title="Permalink to this headline">¶</a></h3>
+<p>Start a Airflow webserver instance</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">webserver</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PORT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">w</span> <span class="n">WORKERS</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">k</span> <span class="p">{</span><span class="n">sync</span><span class="p">,</span><span class="n">eventlet</span><span class="p">,</span><span class="n">gevent</span><span class="p">,</span><span class="n">tornado</span><span class="p">}]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">WORKER_TIMEOUT</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">hn</span> <span class="n">HOSTNAME</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">A</span> <span class="n">ACCESS_LOGFILE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">E</span> <span class="n">ERROR_LOGFILE</span><sp
<TRUNCATED>
[30/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/wasb_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/wasb_hook.html b/_modules/airflow/contrib/hooks/wasb_hook.html
new file mode 100644
index 0000000..2ff6ba4
--- /dev/null
+++ b/_modules/airflow/contrib/hooks/wasb_hook.html
@@ -0,0 +1,345 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.hooks.wasb_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.hooks.wasb_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.hooks.wasb_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
+
+<span class="kn">from</span> <span class="nn">azure.storage.blob</span> <span class="k">import</span> <span class="n">BlockBlobService</span>
+
+
+<div class="viewcode-block" id="WasbHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.wasb_hook.WasbHook">[docs]</a><span class="k">class</span> <span class="nc">WasbHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interacts with Azure Blob Storage through the wasb:// protocol.</span>
+<span class="sd"> </span>
+<span class="sd"> Additional options passed in the 'extra' field of the connection will be</span>
+<span class="sd"> passed to the `BlockBlockService()` constructor. For example, authenticate</span>
+<span class="sd"> using a SAS token by adding {"sas_token": "YOUR_TOKEN"}.</span>
+<span class="sd"> </span>
+<span class="sd"> :param wasb_conn_id: Reference to the wasb connection.</span>
+<span class="sd"> :type wasb_conn_id: str</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">wasb_conn_id</span><span class="o">=</span><span class="s1">'wasb_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">wasb_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""Return the BlockBlobService object."""</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
+ <span class="n">service_options</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span>
+ <span class="k">return</span> <span class="n">BlockBlobService</span><span class="p">(</span><span class="n">account_name</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="n">account_key</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">,</span> <span class="o">**</span><span class="n">service_options</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">check_for_blob</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Check if a blob exists on Azure Blob Storage.</span>
+<span class="sd"> </span>
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param blob_name: Name of the blob.</span>
+<span class="sd"> :type blob_name: str</span>
+<span class="sd"> :param kwargs: Optional keyword arguments that</span>
+<span class="sd"> `BlockBlobService.exists()` takes.</span>
+<span class="sd"> :type kwargs: object</span>
+<span class="sd"> :return: True if the blob exists, False otherwise.</span>
+<span class="sd"> :rtype bool</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">check_for_prefix</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Check if a prefix exists on Azure Blob storage.</span>
+<span class="sd"> </span>
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param prefix: Prefix of the blob.</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param kwargs: Optional keyword arguments that</span>
+<span class="sd"> `BlockBlobService.list_blobs()` takes.</span>
+<span class="sd"> :type kwargs: object</span>
+<span class="sd"> :return: True if blobs matching the prefix exist, False otherwise.</span>
+<span class="sd"> :rtype bool</span>
+<span class="sd"> """</span>
+ <span class="n">matches</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">list_blobs</span><span class="p">(</span><span class="n">container_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span>
+ <span class="n">num_results</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">matches</span><span class="p">))</span> <span class="o">></span> <span class="mi">0</span>
+
+ <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_path</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Upload a file to Azure Blob Storage.</span>
+<span class="sd"> </span>
+<span class="sd"> :param file_path: Path to the file to load.</span>
+<span class="sd"> :type file_path: str</span>
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param blob_name: Name of the blob.</span>
+<span class="sd"> :type blob_name: str</span>
+<span class="sd"> :param kwargs: Optional keyword arguments that</span>
+<span class="sd"> `BlockBlobService.create_blob_from_path()` takes.</span>
+<span class="sd"> :type kwargs: object</span>
+<span class="sd"> """</span>
+ <span class="c1"># Reorder the argument order from airflow.hooks.S3_hook.load_file.</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">create_blob_from_path</span><span class="p">(</span><span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span>
+ <span class="n">file_path</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">load_string</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string_data</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Upload a string to Azure Blob Storage.</span>
+<span class="sd"> </span>
+<span class="sd"> :param string_data: String to load.</span>
+<span class="sd"> :type string_data: str</span>
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param blob_name: Name of the blob.</span>
+<span class="sd"> :type blob_name: str</span>
+<span class="sd"> :param kwargs: Optional keyword arguments that</span>
+<span class="sd"> `BlockBlobService.create_blob_from_text()` takes.</span>
+<span class="sd"> :type kwargs: object</span>
+<span class="sd"> """</span>
+ <span class="c1"># Reorder the argument order from airflow.hooks.S3_hook.load_string.</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">create_blob_from_text</span><span class="p">(</span><span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span>
+ <span class="n">string_data</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/bigquery_check_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/bigquery_check_operator.html b/_modules/airflow/contrib/operators/bigquery_check_operator.html
new file mode 100644
index 0000000..cbad19f
--- /dev/null
+++ b/_modules/airflow/contrib/operators/bigquery_check_operator.html
@@ -0,0 +1,358 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.bigquery_check_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.bigquery_check_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.bigquery_check_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="k">import</span> <span class="n">BigQueryHook</span>
+<span class="kn">from</span> <span class="nn">airflow.operators.check_operator</span> <span class="k">import</span> <span class="n">CheckOperator</span><span class="p">,</span> <span class="n">ValueCheckOperator</span><span class="p">,</span> <span class="n">IntervalCheckOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="BigQueryCheckOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryCheckOperator</span><span class="p">(</span><span class="n">CheckOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Performs checks against BigQuery. The ``BigQueryCheckOperator`` expects</span>
+<span class="sd"> a sql query that will return a single row. Each value on that</span>
+<span class="sd"> first row is evaluated using python ``bool`` casting. If any of the</span>
+<span class="sd"> values return ``False`` the check is failed and errors out.</span>
+
+<span class="sd"> Note that Python bool casting evals the following as ``False``:</span>
+
+<span class="sd"> * ``False``</span>
+<span class="sd"> * ``0``</span>
+<span class="sd"> * Empty string (``""``)</span>
+<span class="sd"> * Empty list (``[]``)</span>
+<span class="sd"> * Empty dictionary or set (``{}``)</span>
+
+<span class="sd"> Given a query like ``SELECT COUNT(*) FROM foo``, it will fail only if</span>
+<span class="sd"> the count ``== 0``. You can craft much more complex query that could,</span>
+<span class="sd"> for instance, check that the table has the same number of rows as</span>
+<span class="sd"> the source table upstream, or that the count of today's partition is</span>
+<span class="sd"> greater than yesterday's partition, or that a set of metrics are less</span>
+<span class="sd"> than 3 standard deviation for the 7 day average.</span>
+
+<span class="sd"> This operator can be used as a data quality check in your pipeline, and</span>
+<span class="sd"> depending on where you put it in your DAG, you have the choice to</span>
+<span class="sd"> stop the critical path, preventing from</span>
+<span class="sd"> publishing dubious data, or on the side and receive email alterts</span>
+<span class="sd"> without stopping the progress of the DAG.</span>
+
+<span class="sd"> :param sql: the sql to be executed</span>
+<span class="sd"> :type sql: string</span>
+<span class="sd"> :param bigquery_conn_id: reference to the BigQuery database</span>
+<span class="sd"> :type bigquery_conn_id: string</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span> <span class="o">=</span> <span class="n">bigquery_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+
+ <span class="k">def</span> <span class="nf">get_db_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="BigQueryValueCheckOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryValueCheckOperator</span><span class="p">(</span><span class="n">ValueCheckOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Performs a simple value check using sql code.</span>
+
+<span class="sd"> :param sql: the sql to be executed</span>
+<span class="sd"> :type sql: string</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">pass_value</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryValueCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
+ <span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">,</span> <span class="n">pass_value</span><span class="o">=</span><span class="n">pass_value</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="n">tolerance</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span> <span class="o">=</span> <span class="n">bigquery_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">get_db_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="BigQueryIntervalCheckOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryIntervalCheckOperator</span><span class="p">(</span><span class="n">IntervalCheckOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that the values of metrics given as SQL expressions are within</span>
+<span class="sd"> a certain tolerance of the ones from days_back before.</span>
+
+<span class="sd"> This method constructs a query like so:</span>
+
+<span class="sd"> SELECT {metrics_threshold_dict_key} FROM {table}</span>
+<span class="sd"> WHERE {date_filter_column}=<date></span>
+
+<span class="sd"> :param table: the table name</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param days_back: number of days between ds and the ds we want to check</span>
+<span class="sd"> against. Defaults to 7 days</span>
+<span class="sd"> :type days_back: int</span>
+<span class="sd"> :param metrics_threshold: a dictionary of ratios indexed by metrics, for</span>
+<span class="sd"> example 'COUNT(*)': 1.5 would require a 50 percent or less difference</span>
+<span class="sd"> between the current day, and the prior days_back.</span>
+<span class="sd"> :type metrics_threshold: dict</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">metrics_thresholds</span><span class="p">,</span>
+ <span class="n">date_filter_column</span><span class="o">=</span><span class="s1">'ds'</span><span class="p">,</span> <span class="n">days_back</span><span class="o">=-</span><span class="mi">7</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryIntervalCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
+ <span class="n">table</span><span class="o">=</span><span class="n">table</span><span class="p">,</span> <span class="n">metrics_thresholds</span><span class="o">=</span><span class="n">metrics_thresholds</span><span class="p">,</span>
+ <span class="n">date_filter_column</span><span class="o">=</span><span class="n">date_filter_column</span><span class="p">,</span> <span class="n">days_back</span><span class="o">=</span><span class="n">days_back</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span> <span class="o">=</span> <span class="n">bigquery_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">get_db_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/bigquery_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/bigquery_operator.html b/_modules/airflow/contrib/operators/bigquery_operator.html
index c7fcfd9..5a74ba1 100644
--- a/_modules/airflow/contrib/operators/bigquery_operator.html
+++ b/_modules/airflow/contrib/operators/bigquery_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
<link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
<link rel="up" title="Module code" href="../../../index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../../../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../../../../index.html">Docs</a> »</li>
-
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
<li><a href="../../../index.html">Module code</a> »</li>
-
- <li>airflow.contrib.operators.bigquery_operator</li>
+
+ <li>airflow.contrib.operators.bigquery_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,16 +181,47 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="kn">import</span> <span class="n">BigQueryHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="k">import</span> <span class="n">BigQueryHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
-<div class="viewcode-block" id="BigQueryOperator"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.operators.bigquery_operator.BigQueryOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+<div class="viewcode-block" id="BigQueryOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.bigquery_operator.BigQueryOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Executes BigQuery SQL queries in a specific BigQuery database</span>
+
+<span class="sd"> :param bql: the sql code to be executed</span>
+<span class="sd"> :type bql: Can receive a str representing a sql statement,</span>
+<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
+<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
+<span class="sd"> :param destination_dataset_table: A dotted</span>
+<span class="sd"> (<project>.|<project>:)<dataset>.<table> that, if set, will store the results</span>
+<span class="sd"> of the query.</span>
+<span class="sd"> :type destination_dataset_table: string</span>
+<span class="sd"> :param write_disposition: Specifies the action that occurs if the destination table</span>
+<span class="sd"> already exists. (default: 'WRITE_EMPTY')</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param create_disposition: Specifies whether the job is allowed to create new tables.</span>
+<span class="sd"> (default: 'CREATE_IF_NEEDED')</span>
+<span class="sd"> :type create_disposition: string</span>
+<span class="sd"> :param bigquery_conn_id: reference to a specific BigQuery hook.</span>
+<span class="sd"> :type bigquery_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> :param udf_config: The User Defined Function configuration for the query.</span>
+<span class="sd"> See https://cloud.google.com/bigquery/user-defined-functions for details.</span>
+<span class="sd"> :type udf_config: list</span>
+<span class="sd"> :param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).</span>
+<span class="sd"> :type use_legacy_sql: boolean</span>
+<span class="sd"> :param maximum_billing_tier: Positive integer that serves as a multiplier of the basic price.</span>
+<span class="sd"> Defaults to None, in which case it uses the value set in the project.</span>
+<span class="sd"> :type maximum_billing_tier: integer</span>
+<span class="sd"> :param query_params: a dictionary containing query parameter types and values, passed to</span>
+<span class="sd"> BigQuery.</span>
+<span class="sd"> :type query_params: dict</span>
+
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bql'</span><span class="p">,</span> <span class="s1">'destination_dataset_table'</span><span class="p">)</span>
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
@@ -169,51 +230,47 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">bql</span><span class="p">,</span>
- <span class="n">destination_dataset_table</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span>
- <span class="n">write_disposition</span> <span class="o">=</span> <span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
- <span class="n">allow_large_results</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">destination_dataset_table</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
+ <span class="n">allow_large_results</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
- <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">udf_config</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">udf_config</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">use_legacy_sql</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">maximum_billing_tier</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
+ <span class="n">query_params</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Create a new BigQueryOperator.</span>
-
-<span class="sd"> :param bql: the sql code to be executed</span>
-<span class="sd"> :type bql: Can receive a str representing a sql statement,</span>
-<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
-<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
-<span class="sd"> :param destination_dataset_table: A dotted dataset.table that, if set,</span>
-<span class="sd"> will store the results of the query.</span>
-<span class="sd"> :type destination_dataset_table: string</span>
-<span class="sd"> :param bigquery_conn_id: reference to a specific BigQuery hook.</span>
-<span class="sd"> :type bigquery_conn_id: string</span>
-<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
-<span class="sd"> For this to work, the service account making the request must have domain-wide delegation enabled.</span>
-<span class="sd"> :type delegate_to: string</span>
-<span class="sd"> :param udf_config: The User Defined Function configuration for the query.</span>
-<span class="sd"> See https://cloud.google.com/bigquery/user-defined-functions for details.</span>
-<span class="sd"> :type udf_config: list</span>
-<span class="sd"> """</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bql</span> <span class="o">=</span> <span class="n">bql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">destination_dataset_table</span> <span class="o">=</span> <span class="n">destination_dataset_table</span>
<span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span> <span class="o">=</span> <span class="n">write_disposition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create_disposition</span> <span class="o">=</span> <span class="n">create_disposition</span>
<span class="bp">self</span><span class="o">.</span><span class="n">allow_large_results</span> <span class="o">=</span> <span class="n">allow_large_results</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span> <span class="o">=</span> <span class="n">bigquery_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
<span class="bp">self</span><span class="o">.</span><span class="n">udf_config</span> <span class="o">=</span> <span class="n">udf_config</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">use_legacy_sql</span> <span class="o">=</span> <span class="n">use_legacy_sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">maximum_billing_tier</span> <span class="o">=</span> <span class="n">maximum_billing_tier</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query_params</span> <span class="o">=</span> <span class="n">query_params</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bql</span><span class="p">))</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bql</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
- <span class="n">cursor</span><span class="o">.</span><span class="n">run_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bql</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">destination_dataset_table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">allow_large_results</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">udf_config</span><span class="p">)</span></div>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">run_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bql</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">destination_dataset_table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">allow_large_results</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">udf_config</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">use_legacy_sql</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">maximum_billing_tier</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create_disposition</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">query_params</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -246,7 +303,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../../../../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/bigquery_to_bigquery.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/bigquery_to_bigquery.html b/_modules/airflow/contrib/operators/bigquery_to_bigquery.html
new file mode 100644
index 0000000..ab12d51
--- /dev/null
+++ b/_modules/airflow/contrib/operators/bigquery_to_bigquery.html
@@ -0,0 +1,316 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.bigquery_to_bigquery — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.bigquery_to_bigquery</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.bigquery_to_bigquery</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="k">import</span> <span class="n">BigQueryHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="BigQueryToBigQueryOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryToBigQueryOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Copies data from one BigQuery table to another. See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy</span>
+
+<span class="sd"> For more details about these parameters.</span>
+
+<span class="sd"> :param source_project_dataset_tables: One or more</span>
+<span class="sd"> dotted (project:|project.)<dataset>.<table> BigQuery tables to use as the</span>
+<span class="sd"> source data. If <project> is not included, project will be the project defined</span>
+<span class="sd"> in the connection json. Use a list if there are multiple source tables.</span>
+<span class="sd"> :type source_project_dataset_tables: list|string</span>
+<span class="sd"> :param destination_project_dataset_table: The destination BigQuery</span>
+<span class="sd"> table. Format is: (project:|project.)<dataset>.<table></span>
+<span class="sd"> :type destination_project_dataset_table: string</span>
+<span class="sd"> :param write_disposition: The write disposition if the table already exists.</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param create_disposition: The create disposition if the table doesn't exist.</span>
+<span class="sd"> :type create_disposition: string</span>
+<span class="sd"> :param bigquery_conn_id: reference to a specific BigQuery hook.</span>
+<span class="sd"> :type bigquery_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'source_project_dataset_tables'</span><span class="p">,</span>
+ <span class="s1">'destination_project_dataset_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e6f0e4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">source_project_dataset_tables</span><span class="p">,</span>
+ <span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryToBigQueryOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_project_dataset_tables</span> <span class="o">=</span> <span class="n">source_project_dataset_tables</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span> <span class="o">=</span> <span class="n">destination_project_dataset_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span> <span class="o">=</span> <span class="n">write_disposition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create_disposition</span> <span class="o">=</span> <span class="n">create_disposition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span> <span class="o">=</span> <span class="n">bigquery_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Executing copy of </span><span class="si">%s</span><span class="s1"> into: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_project_dataset_tables</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span>
+ <span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">run_copy</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_project_dataset_tables</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create_disposition</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[10/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/sensors.html
----------------------------------------------------------------------
diff --git a/_modules/sensors.html b/_modules/sensors.html
index 604c19e..ffa2704 100644
--- a/_modules/sensors.html
+++ b/_modules/sensors.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -178,23 +183,25 @@
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">print_function</span>
<span class="kn">from</span> <span class="nn">future</span> <span class="k">import</span> <span class="n">standard_library</span>
+
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
<span class="kn">from</span> <span class="nn">time</span> <span class="k">import</span> <span class="n">sleep</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
-<span class="kn">import</span> <span class="nn">airflow</span>
-<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">hooks</span><span class="p">,</span> <span class="n">settings</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span><span class="p">,</span> <span class="n">AirflowSensorTimeout</span><span class="p">,</span> <span class="n">AirflowSkipException</span>
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">TaskInstance</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.hdfs_hook</span> <span class="k">import</span> <span class="n">HDFSHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.http_hook</span> <span class="k">import</span> <span class="n">HttpHook</span>
<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="k">import</span> <span class="n">State</span>
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
@@ -223,7 +230,7 @@
<span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">7</span><span class="p">,</span>
<span class="n">soft_fail</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span> <span class="o">=</span> <span class="n">poke_interval</span>
<span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span> <span class="o">=</span> <span class="n">soft_fail</span>
<span class="bp">self</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timeout</span>
@@ -236,20 +243,20 @@
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Override me.'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">started_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">started_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="k">while</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">poke</span><span class="p">(</span><span class="n">context</span><span class="p">):</span>
- <span class="k">if</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">started_at</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">:</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">started_at</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowSkipException</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowSensorTimeout</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
<span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Success criteria met. Exiting."</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Success criteria met. Exiting."</span><span class="p">)</span>
<div class="viewcode-block" id="SqlSensor"><a class="viewcode-back" href="../code.html#airflow.operators.SqlSensor">[docs]</a><span class="k">class</span> <span class="nc">SqlSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Runs a sql statement until a criteria is met. It will keep trying until</span>
+<span class="sd"> Runs a sql statement until a criteria is met. It will keep trying while</span>
<span class="sd"> sql returns no row, or if the first cell in (0, '0', '').</span>
<span class="sd"> :param conn_id: The connection to run the sensor against</span>
@@ -265,12 +272,12 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_id</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">conn_id</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="n">records</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">records</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
@@ -278,8 +285,7 @@
<span class="k">if</span> <span class="nb">str</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'0'</span><span class="p">,</span> <span class="s1">''</span><span class="p">,):</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="kc">True</span>
- <span class="nb">print</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span></div>
+ <span class="k">return</span> <span class="kc">True</span></div>
<div class="viewcode-block" id="MetastorePartitionSensor"><a class="viewcode-back" href="../code.html#airflow.operators.MetastorePartitionSensor">[docs]</a><span class="k">class</span> <span class="nc">MetastorePartitionSensor</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">):</span>
@@ -321,7 +327,7 @@
<span class="c1"># The inheritance model needs to be reworked in order to support overriding args/</span>
<span class="c1"># kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the</span>
<span class="c1"># constructor below and apply_defaults will no longer throw an exception.</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span><span class="p">:</span>
@@ -360,7 +366,7 @@
<span class="sd"> ExternalTaskSensor, but not both.</span>
<span class="sd"> :type execution_delta: datetime.timedelta</span>
<span class="sd"> :param execution_date_fn: function that receives the current execution date</span>
-<span class="sd"> and returns the desired execution date to query. Either execution_delta</span>
+<span class="sd"> and returns the desired execution dates to query. Either execution_delta</span>
<span class="sd"> or execution_date_fn can be passed to ExternalTaskSensor, but not both.</span>
<span class="sd"> :type execution_date_fn: callable</span>
<span class="sd"> """</span>
@@ -375,7 +381,7 @@
<span class="n">execution_delta</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">execution_date_fn</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">ExternalTaskSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">ExternalTaskSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span> <span class="o">=</span> <span class="n">allowed_states</span> <span class="ow">or</span> <span class="p">[</span><span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">]</span>
<span class="k">if</span> <span class="n">execution_delta</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">execution_date_fn</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
@@ -395,11 +401,15 @@
<span class="k">else</span><span class="p">:</span>
<span class="n">dttm</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="n">dttm_filter</span> <span class="o">=</span> <span class="n">dttm</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">dttm</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">dttm</span><span class="p">]</span>
+ <span class="n">serialized_dttm_filter</span> <span class="o">=</span> <span class="s1">','</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">datetime</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span> <span class="k">for</span> <span class="n">datetime</span> <span class="ow">in</span> <span class="n">dttm_filter</span><span class="p">])</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s1">'Poking for '</span>
<span class="s1">'</span><span class="si">{self.external_dag_id}</span><span class="s1">.'</span>
<span class="s1">'</span><span class="si">{self.external_task_id}</span><span class="s1"> on '</span>
- <span class="s1">'</span><span class="si">{dttm}</span><span class="s1"> ... '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="s1">'</span><span class="si">{}</span><span class="s1"> ... '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">serialized_dttm_filter</span><span class="p">,</span> <span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="n">TI</span> <span class="o">=</span> <span class="n">TaskInstance</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
@@ -407,11 +417,11 @@
<span class="n">TI</span><span class="o">.</span><span class="n">dag_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_dag_id</span><span class="p">,</span>
<span class="n">TI</span><span class="o">.</span><span class="n">task_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_task_id</span><span class="p">,</span>
<span class="n">TI</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span><span class="p">),</span>
- <span class="n">TI</span><span class="o">.</span><span class="n">execution_date</span> <span class="o">==</span> <span class="n">dttm</span><span class="p">,</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">execution_date</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">dttm_filter</span><span class="p">),</span>
<span class="p">)</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
<span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">count</span></div>
+ <span class="k">return</span> <span class="n">count</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">dttm_filter</span><span class="p">)</span></div>
<div class="viewcode-block" id="NamedHivePartitionSensor"><a class="viewcode-back" href="../code.html#airflow.operators.NamedHivePartitionSensor">[docs]</a><span class="k">class</span> <span class="nc">NamedHivePartitionSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
@@ -442,7 +452,7 @@
<span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span> <span class="o">*</span> <span class="mi">3</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">NamedHivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">NamedHivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">poke_interval</span><span class="o">=</span><span class="n">poke_interval</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">partition_names</span><span class="p">,</span> <span class="n">basestring</span><span class="p">):</span>
@@ -462,16 +472,16 @@
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Could not parse '</span> <span class="o">+</span> <span class="n">partition</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
-
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'hook'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HiveMetastoreHook</span><span class="p">(</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveMetastoreHook</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span>
<span class="n">metastore_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke_partition</span><span class="p">(</span><span class="n">partition</span><span class="p">):</span>
<span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_partition_name</span><span class="p">(</span><span class="n">partition</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s1">'Poking for </span><span class="si">{schema}</span><span class="s1">.</span><span class="si">{table}</span><span class="s1">/</span><span class="si">{partition}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
<span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">check_for_named_partition</span><span class="p">(</span>
@@ -507,7 +517,7 @@
<span class="sd"> :type metastore_conn_id: str</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'schema'</span><span class="p">,</span> <span class="s1">'table'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,)</span>
- <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#2b2d42'</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#C5CAE9'</span>
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
@@ -517,7 +527,7 @@
<span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
<span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">3</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">poke_interval</span><span class="o">=</span><span class="n">poke_interval</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">partition</span><span class="p">:</span>
<span class="n">partition</span> <span class="o">=</span> <span class="s2">"ds='{{ ds }}'"</span>
@@ -529,11 +539,12 @@
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s1">'Poking for table </span><span class="si">{self.schema}</span><span class="s1">.</span><span class="si">{self.table}</span><span class="s1">, '</span>
<span class="s1">'partition </span><span class="si">{self.partition}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'hook'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HiveMetastoreHook</span><span class="p">(</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveMetastoreHook</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span>
<span class="n">metastore_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">check_for_partition</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">)</span></div>
@@ -556,7 +567,7 @@
<span class="n">file_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">hook</span><span class="o">=</span><span class="n">HDFSHook</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span> <span class="o">=</span> <span class="n">hdfs_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file_size</span> <span class="o">=</span> <span class="n">file_size</span>
@@ -564,56 +575,61 @@
<span class="bp">self</span><span class="o">.</span><span class="n">ignore_copying</span> <span class="o">=</span> <span class="n">ignore_copying</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hook</span>
- <span class="nd">@staticmethod</span>
-<div class="viewcode-block" id="HdfsSensor.filter_for_filesize"><a class="viewcode-back" href="../code.html#airflow.operators.HdfsSensor.filter_for_filesize">[docs]</a> <span class="k">def</span> <span class="nf">filter_for_filesize</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+<div class="viewcode-block" id="HdfsSensor.filter_for_filesize"><a class="viewcode-back" href="../code.html#airflow.operators.HdfsSensor.filter_for_filesize">[docs]</a> <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">filter_for_filesize</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Will test the filepath result and test if its size is at least self.filesize</span>
+
<span class="sd"> :param result: a list of dicts returned by Snakebite ls</span>
<span class="sd"> :param size: the file size in MB a file should be at least to trigger True</span>
<span class="sd"> :return: (bool) depending on the matching criteria</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">size</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Filtering for file size >= </span><span class="si">%s</span><span class="s1"> in files: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">))</span>
+ <span class="n">log</span> <span class="o">=</span> <span class="n">LoggingMixin</span><span class="p">()</span><span class="o">.</span><span class="n">log</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Filtering for file size >= </span><span class="si">%s</span><span class="s1"> in files: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">))</span>
<span class="n">size</span> <span class="o">*=</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEGABYTE</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">result</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">'length'</span><span class="p">]</span> <span class="o">>=</span> <span class="n">size</span><span class="p">]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after size filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after size filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">return</span> <span class="n">result</span></div>
- <span class="nd">@staticmethod</span>
-<div class="viewcode-block" id="HdfsSensor.filter_for_ignored_ext"><a class="viewcode-back" href="../code.html#airflow.operators.HdfsSensor.filter_for_ignored_ext">[docs]</a> <span class="k">def</span> <span class="nf">filter_for_ignored_ext</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">ignored_ext</span><span class="p">,</span> <span class="n">ignore_copying</span><span class="p">):</span>
+<div class="viewcode-block" id="HdfsSensor.filter_for_ignored_ext"><a class="viewcode-back" href="../code.html#airflow.operators.HdfsSensor.filter_for_ignored_ext">[docs]</a> <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">filter_for_ignored_ext</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">ignored_ext</span><span class="p">,</span> <span class="n">ignore_copying</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Will filter if instructed to do so the result to remove matching criteria</span>
+
<span class="sd"> :param result: (list) of dicts returned by Snakebite ls</span>
-<span class="sd"> :param ignored_ext: (list) of ignored extentions</span>
+<span class="sd"> :param ignored_ext: (list) of ignored extensions</span>
<span class="sd"> :param ignore_copying: (bool) shall we ignore ?</span>
-<span class="sd"> :return:</span>
+<span class="sd"> :return: (list) of dicts which were not removed</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">ignore_copying</span><span class="p">:</span>
+ <span class="n">log</span> <span class="o">=</span> <span class="n">LoggingMixin</span><span class="p">()</span><span class="o">.</span><span class="n">log</span>
<span class="n">regex_builder</span> <span class="o">=</span> <span class="s2">"^.*\.(</span><span class="si">%s</span><span class="s2">$)$"</span> <span class="o">%</span> <span class="s1">'$|'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">ignored_ext</span><span class="p">)</span>
<span class="n">ignored_extentions_regex</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">regex_builder</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Filtering result for ignored extentions: </span><span class="si">%s</span><span class="s1"> in files </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">ignored_extentions_regex</span><span class="o">.</span><span class="n">pattern</span><span class="p">,</span>
- <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">))</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span>
+ <span class="s1">'Filtering result for ignored extensions: </span><span class="si">%s</span><span class="s1"> in files </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">ignored_extentions_regex</span><span class="o">.</span><span class="n">pattern</span><span class="p">,</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">)</span>
+ <span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">result</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">ignored_extentions_regex</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">])]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after ext filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after ext filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">return</span> <span class="n">result</span></div>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">sb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"snakebite"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">WARNING</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1"> '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># IMOO it's not right here, as there no raise of any kind.</span>
<span class="c1"># if the filepath is let's say '/data/mydirectory', it's correct but if it is '/data/mydirectory/*',</span>
<span class="c1"># it's not correct as the directory exists and sb does not raise any error</span>
<span class="c1"># here is a quick fix</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">f</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">sb</span><span class="o">.</span><span class="n">ls</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">],</span> <span class="n">include_toplevel</span><span class="o">=</span><span class="kc">False</span><span class="p">)]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">filter_for_ignored_ext</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">ignored_ext</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">ignore_copying</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">filter_for_filesize</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">file_size</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Caught an exception !: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Caught an exception !: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="k">return</span> <span class="kc">False</span></div>
@@ -629,14 +645,14 @@
<span class="n">filepath</span><span class="p">,</span>
<span class="n">webhdfs_conn_id</span><span class="o">=</span><span class="s1">'webhdfs_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">WebHdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">WebHdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
<span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span> <span class="o">=</span> <span class="n">webhdfs_conn_id</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">c</span> <span class="o">=</span> <span class="n">airflow</span><span class="o">.</span><span class="n">hooks</span><span class="o">.</span><span class="n">webhdfs_hook</span><span class="o">.</span><span class="n">WebHDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1"> '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.webhdfs_hook</span> <span class="k">import</span> <span class="n">WebHDFSHook</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="n">WebHDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">return</span> <span class="n">c</span><span class="o">.</span><span class="n">check_for_path</span><span class="p">(</span><span class="n">hdfs_path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">)</span></div>
@@ -654,8 +670,8 @@
<span class="sd"> :param wildcard_match: whether the bucket_key should be interpreted as a</span>
<span class="sd"> Unix wildcard pattern</span>
<span class="sd"> :type wildcard_match: bool</span>
-<span class="sd"> :param s3_conn_id: a reference to the s3 connection</span>
-<span class="sd"> :type s3_conn_id: str</span>
+<span class="sd"> :param aws_conn_id: a reference to the s3 connection</span>
+<span class="sd"> :type aws_conn_id: str</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket_key'</span><span class="p">,</span> <span class="s1">'bucket_name'</span><span class="p">)</span>
@@ -664,9 +680,9 @@
<span class="bp">self</span><span class="p">,</span> <span class="n">bucket_key</span><span class="p">,</span>
<span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">wildcard_match</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
- <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">S3KeySensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3KeySensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="c1"># Parse</span>
<span class="k">if</span> <span class="n">bucket_name</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">parsed_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">bucket_key</span><span class="p">)</span>
@@ -681,13 +697,13 @@
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span> <span class="o">=</span> <span class="n">bucket_key</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="kn">import</span> <span class="nn">airflow.hooks.S3_hook</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">airflow</span><span class="o">.</span><span class="n">hooks</span><span class="o">.</span><span class="n">S3_hook</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
<span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for key : </span><span class="si">{full_url}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for key : </span><span class="si">{full_url}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
<span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
@@ -718,21 +734,21 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span>
<span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">'/'</span><span class="p">,</span>
- <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">S3PrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3PrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="c1"># Parse</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span>
<span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
<span class="bp">self</span><span class="o">.</span><span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="n">bucket_name</span> <span class="o">+</span> <span class="s1">'/'</span> <span class="o">+</span> <span class="n">prefix</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for prefix : </span><span class="si">{self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
- <span class="s1">'in bucket s3://</span><span class="si">{self.bucket_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
- <span class="kn">import</span> <span class="nn">airflow.hooks.S3_hook</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">airflow</span><span class="o">.</span><span class="n">hooks</span><span class="o">.</span><span class="n">S3_hook</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for prefix : </span><span class="si">{self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'in bucket s3://</span><span class="si">{self.bucket_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
<span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_prefix</span><span class="p">(</span>
<span class="n">prefix</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">prefix</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
@@ -750,13 +766,12 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">target_time</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">TimeSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">target_time</span> <span class="o">=</span> <span class="n">target_time</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s1">'Checking if the time (</span><span class="si">{0}</span><span class="s1">) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">target_time</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time (</span><span class="si">%s</span><span class="s1">) has come'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span></div>
<span class="k">class</span> <span class="nc">TimeDeltaSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
@@ -773,15 +788,15 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">delta</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">TimeDeltaSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeDeltaSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">delta</span> <span class="o">=</span> <span class="n">delta</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">dag</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'dag'</span><span class="p">]</span>
<span class="n">target_dttm</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">following_schedule</span><span class="p">(</span><span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">])</span>
<span class="n">target_dttm</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">delta</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time (</span><span class="si">{0}</span><span class="s1">) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">target_dttm</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">></span> <span class="n">target_dttm</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time (</span><span class="si">%s</span><span class="s1">) has come'</span><span class="p">,</span> <span class="n">target_dttm</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">></span> <span class="n">target_dttm</span>
<div class="viewcode-block" id="HttpSensor"><a class="viewcode-back" href="../code.html#airflow.operators.HttpSensor">[docs]</a><span class="k">class</span> <span class="nc">HttpSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
@@ -791,10 +806,12 @@
<span class="sd"> :param http_conn_id: The connection to run the sensor against</span>
<span class="sd"> :type http_conn_id: string</span>
+<span class="sd"> :param method: The HTTP request method to use</span>
+<span class="sd"> :type method: string</span>
<span class="sd"> :param endpoint: The relative part of the full url</span>
<span class="sd"> :type endpoint: string</span>
-<span class="sd"> :param params: The parameters to be added to the GET url</span>
-<span class="sd"> :type params: a dictionary of string key/value pairs</span>
+<span class="sd"> :param request_params: The parameters to be added to the GET url</span>
+<span class="sd"> :type request_params: a dictionary of string key/value pairs</span>
<span class="sd"> :param headers: The HTTP headers to be added to the GET request</span>
<span class="sd"> :type headers: a dictionary of string key/value pairs</span>
<span class="sd"> :param response_check: A check against the 'requests' response object.</span>
@@ -806,31 +823,34 @@
<span class="sd"> depends on the option that's being modified.</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">'request_params'</span><span class="p">)</span>
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">endpoint</span><span class="p">,</span>
<span class="n">http_conn_id</span><span class="o">=</span><span class="s1">'http_default'</span><span class="p">,</span>
- <span class="n">params</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">method</span><span class="o">=</span><span class="s1">'GET'</span><span class="p">,</span>
+ <span class="n">request_params</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">response_check</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">extra_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HttpSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HttpSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="n">params</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">request_params</span> <span class="o">=</span> <span class="n">request_params</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_check</span> <span class="o">=</span> <span class="n">response_check</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">http_hook</span><span class="o">.</span><span class="n">HttpHook</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s1">'GET'</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="n">http_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">HttpHook</span><span class="p">(</span>
+ <span class="n">method</span><span class="o">=</span><span class="n">method</span><span class="p">,</span>
+ <span class="n">http_conn_id</span><span class="o">=</span><span class="n">http_conn_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
- <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span>
+ <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">request_params</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">,</span>
<span class="n">extra_options</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">:</span>
[25/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/mlengine_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/mlengine_operator.html b/_modules/airflow/contrib/operators/mlengine_operator.html
new file mode 100644
index 0000000..b322476
--- /dev/null
+++ b/_modules/airflow/contrib/operators/mlengine_operator.html
@@ -0,0 +1,795 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.mlengine_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.mlengine_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.mlengine_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1">#</span>
+<span class="c1"># Licensed to the Apache Software Foundation (ASF) under one or more</span>
+<span class="c1"># contributor license agreements. See the NOTICE file distributed with</span>
+<span class="c1"># this work for additional information regarding copyright ownership.</span>
+<span class="c1"># The ASF licenses this file to You under the Apache License, Version 2.0</span>
+<span class="c1"># (the 'License'); you may not use this file except in compliance with</span>
+<span class="c1"># the License. You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an 'AS IS' BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">import</span> <span class="nn">re</span>
+
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">settings</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_mlengine_hook</span> <span class="k">import</span> <span class="n">MLEngineHook</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">apiclient</span> <span class="k">import</span> <span class="n">errors</span>
+
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
+<span class="n">log</span> <span class="o">=</span> <span class="n">LoggingMixin</span><span class="p">()</span><span class="o">.</span><span class="n">log</span>
+
+
+<span class="k">def</span> <span class="nf">_create_prediction_input</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">region</span><span class="p">,</span>
+ <span class="n">data_format</span><span class="p">,</span>
+ <span class="n">input_paths</span><span class="p">,</span>
+ <span class="n">output_path</span><span class="p">,</span>
+ <span class="n">model_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">version_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">uri</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">max_worker_count</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">runtime_version</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create the batch prediction input from the given parameters.</span>
+
+<span class="sd"> Args:</span>
+<span class="sd"> A subset of arguments documented in __init__ method of class</span>
+<span class="sd"> MLEngineBatchPredictionOperator</span>
+
+<span class="sd"> Returns:</span>
+<span class="sd"> A dictionary representing the predictionInput object as documented</span>
+<span class="sd"> in https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs.</span>
+
+<span class="sd"> Raises:</span>
+<span class="sd"> ValueError: if a unique model/version origin cannot be determined.</span>
+<span class="sd"> """</span>
+ <span class="n">prediction_input</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'dataFormat'</span><span class="p">:</span> <span class="n">data_format</span><span class="p">,</span>
+ <span class="s1">'inputPaths'</span><span class="p">:</span> <span class="n">input_paths</span><span class="p">,</span>
+ <span class="s1">'outputPath'</span><span class="p">:</span> <span class="n">output_path</span><span class="p">,</span>
+ <span class="s1">'region'</span><span class="p">:</span> <span class="n">region</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="n">uri</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">model_name</span> <span class="ow">or</span> <span class="n">version_name</span><span class="p">:</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
+ <span class="s1">'Ambiguous model origin: Both uri and model/version name are provided.'</span>
+ <span class="p">)</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Ambiguous model origin.'</span><span class="p">)</span>
+ <span class="n">prediction_input</span><span class="p">[</span><span class="s1">'uri'</span><span class="p">]</span> <span class="o">=</span> <span class="n">uri</span>
+ <span class="k">elif</span> <span class="n">model_name</span><span class="p">:</span>
+ <span class="n">origin_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/models/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">version_name</span><span class="p">:</span>
+ <span class="n">prediction_input</span><span class="p">[</span><span class="s1">'modelName'</span><span class="p">]</span> <span class="o">=</span> <span class="n">origin_name</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">prediction_input</span><span class="p">[</span><span class="s1">'versionName'</span><span class="p">]</span> <span class="o">=</span> \
+ <span class="n">origin_name</span> <span class="o">+</span> <span class="s1">'/versions/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">version_name</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
+ <span class="s1">'Missing model origin: Batch prediction expects a model, '</span>
+ <span class="s1">'a model & version combination, or a URI to savedModel.'</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Missing model origin.'</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">max_worker_count</span><span class="p">:</span>
+ <span class="n">prediction_input</span><span class="p">[</span><span class="s1">'maxWorkerCount'</span><span class="p">]</span> <span class="o">=</span> <span class="n">max_worker_count</span>
+ <span class="k">if</span> <span class="n">runtime_version</span><span class="p">:</span>
+ <span class="n">prediction_input</span><span class="p">[</span><span class="s1">'runtimeVersion'</span><span class="p">]</span> <span class="o">=</span> <span class="n">runtime_version</span>
+
+ <span class="k">return</span> <span class="n">prediction_input</span>
+
+
+<span class="k">def</span> <span class="nf">_normalize_mlengine_job_id</span><span class="p">(</span><span class="n">job_id</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Replaces invalid MLEngine job_id characters with '_'.</span>
+
+<span class="sd"> This also adds a leading 'z' in case job_id starts with an invalid</span>
+<span class="sd"> character.</span>
+
+<span class="sd"> Args:</span>
+<span class="sd"> job_id: A job_id str that may have invalid characters.</span>
+
+<span class="sd"> Returns:</span>
+<span class="sd"> A valid job_id representation.</span>
+<span class="sd"> """</span>
+ <span class="n">match</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="sa">r</span><span class="s1">'\d'</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">match</span> <span class="ow">and</span> <span class="n">match</span><span class="o">.</span><span class="n">start</span><span class="p">()</span> <span class="ow">is</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">job_id</span> <span class="o">=</span> <span class="s1">'z_</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">job_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="s1">'[^0-9a-zA-Z]+'</span><span class="p">,</span> <span class="s1">'_'</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="MLEngineBatchPredictionOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator">[docs]</a><span class="k">class</span> <span class="nc">MLEngineBatchPredictionOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Start a Google Cloud ML Engine prediction job.</span>
+
+<span class="sd"> NOTE: For model origin, users should consider exactly one from the</span>
+<span class="sd"> three options below:</span>
+<span class="sd"> 1. Populate 'uri' field only, which should be a GCS location that</span>
+<span class="sd"> points to a tensorflow savedModel directory.</span>
+<span class="sd"> 2. Populate 'model_name' field only, which refers to an existing</span>
+<span class="sd"> model, and the default version of the model will be used.</span>
+<span class="sd"> 3. Populate both 'model_name' and 'version_name' fields, which</span>
+<span class="sd"> refers to a specific version of a specific model.</span>
+
+<span class="sd"> In options 2 and 3, both model and version name should contain the</span>
+<span class="sd"> minimal identifier. For instance, call</span>
+<span class="sd"> MLEngineBatchPredictionOperator(</span>
+<span class="sd"> ...,</span>
+<span class="sd"> model_name='my_model',</span>
+<span class="sd"> version_name='my_version',</span>
+<span class="sd"> ...)</span>
+<span class="sd"> if the desired model version is</span>
+<span class="sd"> "projects/my_project/models/my_model/versions/my_version".</span>
+
+
+<span class="sd"> :param project_id: The Google Cloud project name where the</span>
+<span class="sd"> prediction job is submitted.</span>
+<span class="sd"> :type project_id: string</span>
+
+<span class="sd"> :param job_id: A unique id for the prediction job on Google Cloud</span>
+<span class="sd"> ML Engine.</span>
+<span class="sd"> :type job_id: string</span>
+
+<span class="sd"> :param data_format: The format of the input data.</span>
+<span class="sd"> It will default to 'DATA_FORMAT_UNSPECIFIED' if is not provided</span>
+<span class="sd"> or is not one of ["TEXT", "TF_RECORD", "TF_RECORD_GZIP"].</span>
+<span class="sd"> :type data_format: string</span>
+
+<span class="sd"> :param input_paths: A list of GCS paths of input data for batch</span>
+<span class="sd"> prediction. Accepting wildcard operator *, but only at the end.</span>
+<span class="sd"> :type input_paths: list of string</span>
+
+<span class="sd"> :param output_path: The GCS path where the prediction results are</span>
+<span class="sd"> written to.</span>
+<span class="sd"> :type output_path: string</span>
+
+<span class="sd"> :param region: The Google Compute Engine region to run the</span>
+<span class="sd"> prediction job in.:</span>
+<span class="sd"> :type region: string</span>
+
+<span class="sd"> :param model_name: The Google Cloud ML Engine model to use for prediction.</span>
+<span class="sd"> If version_name is not provided, the default version of this</span>
+<span class="sd"> model will be used.</span>
+<span class="sd"> Should not be None if version_name is provided.</span>
+<span class="sd"> Should be None if uri is provided.</span>
+<span class="sd"> :type model_name: string</span>
+
+<span class="sd"> :param version_name: The Google Cloud ML Engine model version to use for</span>
+<span class="sd"> prediction.</span>
+<span class="sd"> Should be None if uri is provided.</span>
+<span class="sd"> :type version_name: string</span>
+
+<span class="sd"> :param uri: The GCS path of the saved model to use for prediction.</span>
+<span class="sd"> Should be None if model_name is provided.</span>
+<span class="sd"> It should be a GCS path pointing to a tensorflow SavedModel.</span>
+<span class="sd"> :type uri: string</span>
+
+<span class="sd"> :param max_worker_count: The maximum number of workers to be used</span>
+<span class="sd"> for parallel processing. Defaults to 10 if not specified.</span>
+<span class="sd"> :type max_worker_count: int</span>
+
+<span class="sd"> :param runtime_version: The Google Cloud ML Engine runtime version to use</span>
+<span class="sd"> for batch prediction.</span>
+<span class="sd"> :type runtime_version: string</span>
+
+<span class="sd"> :param gcp_conn_id: The connection ID used for connection to Google</span>
+<span class="sd"> Cloud Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must</span>
+<span class="sd"> have doamin-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+
+<span class="sd"> Raises:</span>
+<span class="sd"> ValueError: if a unique model/version origin cannot be determined.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="s2">"prediction_job_request"</span><span class="p">,</span>
+ <span class="p">]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span>
+ <span class="n">job_id</span><span class="p">,</span>
+ <span class="n">region</span><span class="p">,</span>
+ <span class="n">data_format</span><span class="p">,</span>
+ <span class="n">input_paths</span><span class="p">,</span>
+ <span class="n">output_path</span><span class="p">,</span>
+ <span class="n">model_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">version_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">uri</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">max_worker_count</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">runtime_version</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MLEngineBatchPredictionOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">prediction_input</span> <span class="o">=</span> <span class="n">_create_prediction_input</span><span class="p">(</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">region</span><span class="p">,</span> <span class="n">data_format</span><span class="p">,</span> <span class="n">input_paths</span><span class="p">,</span> <span class="n">output_path</span><span class="p">,</span>
+ <span class="n">model_name</span><span class="p">,</span> <span class="n">version_name</span><span class="p">,</span> <span class="n">uri</span><span class="p">,</span> <span class="n">max_worker_count</span><span class="p">,</span>
+ <span class="n">runtime_version</span><span class="p">)</span>
+ <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
+ <span class="s1">'Cannot create batch prediction job request due to: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">e</span>
+ <span class="p">)</span>
+ <span class="k">raise</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">prediction_job_request</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'jobId'</span><span class="p">:</span> <span class="n">_normalize_mlengine_job_id</span><span class="p">(</span><span class="n">job_id</span><span class="p">),</span>
+ <span class="s1">'predictionInput'</span><span class="p">:</span> <span class="n">prediction_input</span>
+ <span class="p">}</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MLEngineHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">check_existing_job</span><span class="p">(</span><span class="n">existing_job</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">existing_job</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'predictionInput'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> <span class="o">==</span> \
+ <span class="bp">self</span><span class="o">.</span><span class="n">prediction_job_request</span><span class="p">[</span><span class="s1">'predictionInput'</span><span class="p">]</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">finished_prediction_job</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_job</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">prediction_job_request</span><span class="p">,</span>
+ <span class="n">check_existing_job</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span><span class="p">:</span>
+ <span class="k">raise</span>
+
+ <span class="k">if</span> <span class="n">finished_prediction_job</span><span class="p">[</span><span class="s1">'state'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'SUCCEEDED'</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
+ <span class="s1">'Batch prediction job failed: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="nb">str</span><span class="p">(</span><span class="n">finished_prediction_job</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="n">finished_prediction_job</span><span class="p">[</span><span class="s1">'errorMessage'</span><span class="p">])</span>
+
+ <span class="k">return</span> <span class="n">finished_prediction_job</span><span class="p">[</span><span class="s1">'predictionOutput'</span><span class="p">]</span></div>
+
+
+<div class="viewcode-block" id="MLEngineModelOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.mlengine_operator.MLEngineModelOperator">[docs]</a><span class="k">class</span> <span class="nc">MLEngineModelOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Operator for managing a Google Cloud ML Engine model.</span>
+
+<span class="sd"> :param project_id: The Google Cloud project name to which MLEngine</span>
+<span class="sd"> model belongs.</span>
+<span class="sd"> :type project_id: string</span>
+
+<span class="sd"> :param model: A dictionary containing the information about the model.</span>
+<span class="sd"> If the `operation` is `create`, then the `model` parameter should</span>
+<span class="sd"> contain all the information about this model such as `name`.</span>
+
+<span class="sd"> If the `operation` is `get`, the `model` parameter</span>
+<span class="sd"> should contain the `name` of the model.</span>
+<span class="sd"> :type model: dict</span>
+
+<span class="sd"> :param operation: The operation to perform. Available operations are:</span>
+<span class="sd"> 'create': Creates a new model as provided by the `model` parameter.</span>
+<span class="sd"> 'get': Gets a particular model where the name is specified in `model`.</span>
+
+<span class="sd"> :param gcp_conn_id: The connection ID to use when fetching connection info.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have</span>
+<span class="sd"> domain-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="s1">'_model'</span><span class="p">,</span>
+ <span class="p">]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span>
+ <span class="n">model</span><span class="p">,</span>
+ <span class="n">operation</span><span class="o">=</span><span class="s1">'create'</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MLEngineModelOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_model</span> <span class="o">=</span> <span class="n">model</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">=</span> <span class="n">operation</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MLEngineHook</span><span class="p">(</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_gcp_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_delegate_to</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">==</span> <span class="s1">'create'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_model</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">==</span> <span class="s1">'get'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_model</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Unknown operation: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_operation</span><span class="p">))</span></div>
+
+
+<div class="viewcode-block" id="MLEngineVersionOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator">[docs]</a><span class="k">class</span> <span class="nc">MLEngineVersionOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Operator for managing a Google Cloud ML Engine version.</span>
+
+<span class="sd"> :param project_id: The Google Cloud project name to which MLEngine</span>
+<span class="sd"> model belongs.</span>
+<span class="sd"> :type project_id: string</span>
+
+<span class="sd"> :param model_name: The name of the Google Cloud ML Engine model that the version</span>
+<span class="sd"> belongs to.</span>
+<span class="sd"> :type model_name: string</span>
+
+<span class="sd"> :param version_name: A name to use for the version being operated upon. If</span>
+<span class="sd"> not None and the `version` argument is None or does not have a value for</span>
+<span class="sd"> the `name` key, then this will be populated in the payload for the</span>
+<span class="sd"> `name` key.</span>
+<span class="sd"> :type version_name: string</span>
+
+<span class="sd"> :param version: A dictionary containing the information about the version.</span>
+<span class="sd"> If the `operation` is `create`, `version` should contain all the</span>
+<span class="sd"> information about this version such as name, and deploymentUrl.</span>
+<span class="sd"> If the `operation` is `get` or `delete`, the `version` parameter</span>
+<span class="sd"> should contain the `name` of the version.</span>
+<span class="sd"> If it is None, the only `operation` possible would be `list`.</span>
+<span class="sd"> :type version: dict</span>
+
+<span class="sd"> :param operation: The operation to perform. Available operations are:</span>
+<span class="sd"> 'create': Creates a new version in the model specified by `model_name`,</span>
+<span class="sd"> in which case the `version` parameter should contain all the</span>
+<span class="sd"> information to create that version</span>
+<span class="sd"> (e.g. `name`, `deploymentUrl`).</span>
+<span class="sd"> 'get': Gets full information of a particular version in the model</span>
+<span class="sd"> specified by `model_name`.</span>
+<span class="sd"> The name of the version should be specified in the `version`</span>
+<span class="sd"> parameter.</span>
+
+<span class="sd"> 'list': Lists all available versions of the model specified</span>
+<span class="sd"> by `model_name`.</span>
+
+<span class="sd"> 'delete': Deletes the version specified in `version` parameter from the</span>
+<span class="sd"> model specified by `model_name`).</span>
+<span class="sd"> The name of the version should be specified in the `version`</span>
+<span class="sd"> parameter.</span>
+<span class="sd"> :type operation: string</span>
+
+<span class="sd"> :param gcp_conn_id: The connection ID to use when fetching connection info.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have</span>
+<span class="sd"> domain-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="s1">'_model_name'</span><span class="p">,</span>
+ <span class="s1">'_version_name'</span><span class="p">,</span>
+ <span class="s1">'_version'</span><span class="p">,</span>
+ <span class="p">]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span>
+ <span class="n">model_name</span><span class="p">,</span>
+ <span class="n">version_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">version</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">operation</span><span class="o">=</span><span class="s1">'create'</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+
+ <span class="nb">super</span><span class="p">(</span><span class="n">MLEngineVersionOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_model_name</span> <span class="o">=</span> <span class="n">model_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_version_name</span> <span class="o">=</span> <span class="n">version_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_version</span> <span class="o">=</span> <span class="n">version</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">=</span> <span class="n">operation</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="s1">'name'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_version</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_version</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_version_name</span>
+
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MLEngineHook</span><span class="p">(</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_gcp_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_delegate_to</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">==</span> <span class="s1">'create'</span><span class="p">:</span>
+ <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">_version</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_version</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model_name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_version</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">==</span> <span class="s1">'set_default'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">set_default_version</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model_name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_version</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">==</span> <span class="s1">'list'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">list_versions</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model_name</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_operation</span> <span class="o">==</span> <span class="s1">'delete'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">delete_version</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_model_name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_version</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Unknown operation: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_operation</span><span class="p">))</span></div>
+
+
+<div class="viewcode-block" id="MLEngineTrainingOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator">[docs]</a><span class="k">class</span> <span class="nc">MLEngineTrainingOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Operator for launching a MLEngine training job.</span>
+
+<span class="sd"> :param project_id: The Google Cloud project name within which MLEngine</span>
+<span class="sd"> training job should run. This field could be templated.</span>
+<span class="sd"> :type project_id: string</span>
+
+<span class="sd"> :param job_id: A unique templated id for the submitted Google MLEngine</span>
+<span class="sd"> training job.</span>
+<span class="sd"> :type job_id: string</span>
+
+<span class="sd"> :param package_uris: A list of package locations for MLEngine training job,</span>
+<span class="sd"> which should include the main training program + any additional</span>
+<span class="sd"> dependencies.</span>
+<span class="sd"> :type package_uris: string</span>
+
+<span class="sd"> :param training_python_module: The Python module name to run within MLEngine</span>
+<span class="sd"> training job after installing 'package_uris' packages.</span>
+<span class="sd"> :type training_python_module: string</span>
+
+<span class="sd"> :param training_args: A list of templated command line arguments to pass to</span>
+<span class="sd"> the MLEngine training program.</span>
+<span class="sd"> :type training_args: string</span>
+
+<span class="sd"> :param region: The Google Compute Engine region to run the MLEngine training</span>
+<span class="sd"> job in. This field could be templated.</span>
+<span class="sd"> :type region: string</span>
+
+<span class="sd"> :param scale_tier: Resource tier for MLEngine training job.</span>
+<span class="sd"> :type scale_tier: string</span>
+
+<span class="sd"> :param gcp_conn_id: The connection ID to use when fetching connection info.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have</span>
+<span class="sd"> domain-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+
+<span class="sd"> :param mode: Can be one of 'DRY_RUN'/'CLOUD'. In 'DRY_RUN' mode, no real</span>
+<span class="sd"> training job will be launched, but the MLEngine training job request</span>
+<span class="sd"> will be printed out. In 'CLOUD' mode, a real MLEngine training job</span>
+<span class="sd"> creation request will be issued.</span>
+<span class="sd"> :type mode: string</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="s1">'_project_id'</span><span class="p">,</span>
+ <span class="s1">'_job_id'</span><span class="p">,</span>
+ <span class="s1">'_package_uris'</span><span class="p">,</span>
+ <span class="s1">'_training_python_module'</span><span class="p">,</span>
+ <span class="s1">'_training_args'</span><span class="p">,</span>
+ <span class="s1">'_region'</span><span class="p">,</span>
+ <span class="s1">'_scale_tier'</span><span class="p">,</span>
+ <span class="p">]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span>
+ <span class="n">job_id</span><span class="p">,</span>
+ <span class="n">package_uris</span><span class="p">,</span>
+ <span class="n">training_python_module</span><span class="p">,</span>
+ <span class="n">training_args</span><span class="p">,</span>
+ <span class="n">region</span><span class="p">,</span>
+ <span class="n">scale_tier</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mode</span><span class="o">=</span><span class="s1">'PRODUCTION'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MLEngineTrainingOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span> <span class="o">=</span> <span class="n">job_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_package_uris</span> <span class="o">=</span> <span class="n">package_uris</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_training_python_module</span> <span class="o">=</span> <span class="n">training_python_module</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_training_args</span> <span class="o">=</span> <span class="n">training_args</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_region</span> <span class="o">=</span> <span class="n">region</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_scale_tier</span> <span class="o">=</span> <span class="n">scale_tier</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_mode</span> <span class="o">=</span> <span class="n">mode</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Google Cloud project id is required.'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s1">'An unique job id is required for Google MLEngine training '</span>
+ <span class="s1">'job.'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">package_uris</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s1">'At least one python package is required for MLEngine '</span>
+ <span class="s1">'Training job.'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">training_python_module</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s1">'Python module name to run after installing required '</span>
+ <span class="s1">'packages is required.'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_region</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Google Compute Engine region is required.'</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">job_id</span> <span class="o">=</span> <span class="n">_normalize_mlengine_job_id</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span><span class="p">)</span>
+ <span class="n">training_request</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'jobId'</span><span class="p">:</span> <span class="n">job_id</span><span class="p">,</span>
+ <span class="s1">'trainingInput'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'scaleTier'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_scale_tier</span><span class="p">,</span>
+ <span class="s1">'packageUris'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_package_uris</span><span class="p">,</span>
+ <span class="s1">'pythonModule'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_training_python_module</span><span class="p">,</span>
+ <span class="s1">'region'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_region</span><span class="p">,</span>
+ <span class="s1">'args'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_training_args</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mode</span> <span class="o">==</span> <span class="s1">'DRY_RUN'</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'In dry_run mode.'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'MLEngine Training job request is: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">training_request</span><span class="p">))</span>
+ <span class="k">return</span>
+
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MLEngineHook</span><span class="p">(</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_gcp_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_delegate_to</span><span class="p">)</span>
+
+ <span class="c1"># Helper method to check if the existing job's training input is the</span>
+ <span class="c1"># same as the request we get here.</span>
+ <span class="k">def</span> <span class="nf">check_existing_job</span><span class="p">(</span><span class="n">existing_job</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">existing_job</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'trainingInput'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> <span class="o">==</span> \
+ <span class="n">training_request</span><span class="p">[</span><span class="s1">'trainingInput'</span><span class="p">]</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">finished_training_job</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_job</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_project_id</span><span class="p">,</span> <span class="n">training_request</span><span class="p">,</span> <span class="n">check_existing_job</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span><span class="p">:</span>
+ <span class="k">raise</span>
+
+ <span class="k">if</span> <span class="n">finished_training_job</span><span class="p">[</span><span class="s1">'state'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'SUCCEEDED'</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'MLEngine training job failed: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="nb">str</span><span class="p">(</span><span class="n">finished_training_job</span><span class="p">)))</span>
+ <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="n">finished_training_job</span><span class="p">[</span><span class="s1">'errorMessage'</span><span class="p">])</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/sensors/wasb_sensor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/sensors/wasb_sensor.html b/_modules/airflow/contrib/sensors/wasb_sensor.html
new file mode 100644
index 0000000..a1a8d95
--- /dev/null
+++ b/_modules/airflow/contrib/sensors/wasb_sensor.html
@@ -0,0 +1,327 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.sensors.wasb_sensor — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.sensors.wasb_sensor</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.sensors.wasb_sensor</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.wasb_hook</span> <span class="k">import</span> <span class="n">WasbHook</span>
+<span class="kn">from</span> <span class="nn">airflow.operators.sensors</span> <span class="k">import</span> <span class="n">BaseSensorOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="WasbBlobSensor"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.sensors.wasb_sensor.WasbBlobSensor">[docs]</a><span class="k">class</span> <span class="nc">WasbBlobSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a blob to arrive on Azure Blob Storage.</span>
+
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param blob_name: Name of the blob.</span>
+<span class="sd"> :type blob_name: str</span>
+<span class="sd"> :param wasb_conn_id: Reference to the wasb connection.</span>
+<span class="sd"> :type wasb_conn_id: str</span>
+<span class="sd"> :param check_options: Optional keyword arguments that</span>
+<span class="sd"> `WasbHook.check_for_blob()` takes.</span>
+<span class="sd"> :type check_options: dict</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'container_name'</span><span class="p">,</span> <span class="s1">'blob_name'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span>
+ <span class="n">wasb_conn_id</span><span class="o">=</span><span class="s1">'wasb_default'</span><span class="p">,</span> <span class="n">check_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">WasbBlobSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">check_options</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">check_options</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wasb_conn_id</span> <span class="o">=</span> <span class="n">wasb_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">container_name</span> <span class="o">=</span> <span class="n">container_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">blob_name</span> <span class="o">=</span> <span class="n">blob_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">check_options</span> <span class="o">=</span> <span class="n">check_options</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for blob: </span><span class="si">{self.blob_name}</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'in wasb://</span><span class="si">{self.container_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">WasbHook</span><span class="p">(</span><span class="n">wasb_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">wasb_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_blob</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container_name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">blob_name</span><span class="p">,</span>
+ <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">check_options</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="WasbPrefixSensor"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.sensors.wasb_sensor.WasbPrefixSensor">[docs]</a><span class="k">class</span> <span class="nc">WasbPrefixSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for blobs matching a prefix to arrive on Azure Blob Storage.</span>
+
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param prefix: Prefix of the blob.</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param wasb_conn_id: Reference to the wasb connection.</span>
+<span class="sd"> :type wasb_conn_id: str</span>
+<span class="sd"> :param check_options: Optional keyword arguments that</span>
+<span class="sd"> `WasbHook.check_for_prefix()` takes.</span>
+<span class="sd"> :type check_options: dict</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'container_name'</span><span class="p">,</span> <span class="s1">'prefix'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">wasb_conn_id</span><span class="o">=</span><span class="s1">'wasb_default'</span><span class="p">,</span>
+ <span class="n">check_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">WasbPrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">check_options</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">check_options</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wasb_conn_id</span> <span class="o">=</span> <span class="n">wasb_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">container_name</span> <span class="o">=</span> <span class="n">container_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">check_options</span> <span class="o">=</span> <span class="n">check_options</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for prefix: </span><span class="si">{self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'in wasb://</span><span class="si">{self.container_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">WasbHook</span><span class="p">(</span><span class="n">wasb_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">wasb_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_prefix</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container_name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">check_options</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[20/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/operators/sensors.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/sensors.html b/_modules/airflow/operators/sensors.html
index 03643fe..7414daf 100644
--- a/_modules/airflow/operators/sensors.html
+++ b/_modules/airflow/operators/sensors.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
@@ -178,23 +183,25 @@
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">print_function</span>
<span class="kn">from</span> <span class="nn">future</span> <span class="k">import</span> <span class="n">standard_library</span>
+
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
<span class="kn">from</span> <span class="nn">time</span> <span class="k">import</span> <span class="n">sleep</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">sys</span>
-<span class="kn">import</span> <span class="nn">airflow</span>
-<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">hooks</span><span class="p">,</span> <span class="n">settings</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">settings</span>
<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span><span class="p">,</span> <span class="n">AirflowSensorTimeout</span><span class="p">,</span> <span class="n">AirflowSkipException</span>
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">TaskInstance</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.hdfs_hook</span> <span class="k">import</span> <span class="n">HDFSHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.http_hook</span> <span class="k">import</span> <span class="n">HttpHook</span>
<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="k">import</span> <span class="n">State</span>
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
@@ -223,7 +230,7 @@
<span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">7</span><span class="p">,</span>
<span class="n">soft_fail</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span> <span class="o">=</span> <span class="n">poke_interval</span>
<span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span> <span class="o">=</span> <span class="n">soft_fail</span>
<span class="bp">self</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timeout</span>
@@ -236,20 +243,20 @@
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Override me.'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">started_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">started_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="k">while</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">poke</span><span class="p">(</span><span class="n">context</span><span class="p">):</span>
- <span class="k">if</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">started_at</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">:</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">started_at</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowSkipException</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowSensorTimeout</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
<span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Success criteria met. Exiting."</span><span class="p">)</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Success criteria met. Exiting."</span><span class="p">)</span></div>
<span class="k">class</span> <span class="nc">SqlSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Runs a sql statement until a criteria is met. It will keep trying until</span>
+<span class="sd"> Runs a sql statement until a criteria is met. It will keep trying while</span>
<span class="sd"> sql returns no row, or if the first cell in (0, '0', '').</span>
<span class="sd"> :param conn_id: The connection to run the sensor against</span>
@@ -265,12 +272,12 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_id</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">conn_id</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="n">records</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">records</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">False</span>
@@ -279,7 +286,6 @@
<span class="k">return</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">True</span>
- <span class="nb">print</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span>
<span class="k">class</span> <span class="nc">MetastorePartitionSensor</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">):</span>
@@ -321,7 +327,7 @@
<span class="c1"># The inheritance model needs to be reworked in order to support overriding args/</span>
<span class="c1"># kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the</span>
<span class="c1"># constructor below and apply_defaults will no longer throw an exception.</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span><span class="p">:</span>
@@ -360,7 +366,7 @@
<span class="sd"> ExternalTaskSensor, but not both.</span>
<span class="sd"> :type execution_delta: datetime.timedelta</span>
<span class="sd"> :param execution_date_fn: function that receives the current execution date</span>
-<span class="sd"> and returns the desired execution date to query. Either execution_delta</span>
+<span class="sd"> and returns the desired execution dates to query. Either execution_delta</span>
<span class="sd"> or execution_date_fn can be passed to ExternalTaskSensor, but not both.</span>
<span class="sd"> :type execution_date_fn: callable</span>
<span class="sd"> """</span>
@@ -375,7 +381,7 @@
<span class="n">execution_delta</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">execution_date_fn</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">ExternalTaskSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">ExternalTaskSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span> <span class="o">=</span> <span class="n">allowed_states</span> <span class="ow">or</span> <span class="p">[</span><span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">]</span>
<span class="k">if</span> <span class="n">execution_delta</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">execution_date_fn</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
@@ -395,11 +401,15 @@
<span class="k">else</span><span class="p">:</span>
<span class="n">dttm</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="n">dttm_filter</span> <span class="o">=</span> <span class="n">dttm</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">dttm</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">dttm</span><span class="p">]</span>
+ <span class="n">serialized_dttm_filter</span> <span class="o">=</span> <span class="s1">','</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">datetime</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span> <span class="k">for</span> <span class="n">datetime</span> <span class="ow">in</span> <span class="n">dttm_filter</span><span class="p">])</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s1">'Poking for '</span>
<span class="s1">'</span><span class="si">{self.external_dag_id}</span><span class="s1">.'</span>
<span class="s1">'</span><span class="si">{self.external_task_id}</span><span class="s1"> on '</span>
- <span class="s1">'</span><span class="si">{dttm}</span><span class="s1"> ... '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="s1">'</span><span class="si">{}</span><span class="s1"> ... '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">serialized_dttm_filter</span><span class="p">,</span> <span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="n">TI</span> <span class="o">=</span> <span class="n">TaskInstance</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
@@ -407,11 +417,11 @@
<span class="n">TI</span><span class="o">.</span><span class="n">dag_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_dag_id</span><span class="p">,</span>
<span class="n">TI</span><span class="o">.</span><span class="n">task_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_task_id</span><span class="p">,</span>
<span class="n">TI</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span><span class="p">),</span>
- <span class="n">TI</span><span class="o">.</span><span class="n">execution_date</span> <span class="o">==</span> <span class="n">dttm</span><span class="p">,</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">execution_date</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">dttm_filter</span><span class="p">),</span>
<span class="p">)</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
<span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">count</span>
+ <span class="k">return</span> <span class="n">count</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">dttm_filter</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">NamedHivePartitionSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
@@ -442,7 +452,7 @@
<span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span> <span class="o">*</span> <span class="mi">3</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">NamedHivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">NamedHivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">poke_interval</span><span class="o">=</span><span class="n">poke_interval</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">partition_names</span><span class="p">,</span> <span class="n">basestring</span><span class="p">):</span>
@@ -462,16 +472,16 @@
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Could not parse '</span> <span class="o">+</span> <span class="n">partition</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
-
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'hook'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HiveMetastoreHook</span><span class="p">(</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveMetastoreHook</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span>
<span class="n">metastore_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke_partition</span><span class="p">(</span><span class="n">partition</span><span class="p">):</span>
<span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_partition_name</span><span class="p">(</span><span class="n">partition</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s1">'Poking for </span><span class="si">{schema}</span><span class="s1">.</span><span class="si">{table}</span><span class="s1">/</span><span class="si">{partition}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
<span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">check_for_named_partition</span><span class="p">(</span>
@@ -507,7 +517,7 @@
<span class="sd"> :type metastore_conn_id: str</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'schema'</span><span class="p">,</span> <span class="s1">'table'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,)</span>
- <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#2b2d42'</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#C5CAE9'</span>
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
@@ -517,7 +527,7 @@
<span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
<span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">3</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">poke_interval</span><span class="o">=</span><span class="n">poke_interval</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">partition</span><span class="p">:</span>
<span class="n">partition</span> <span class="o">=</span> <span class="s2">"ds='{{ ds }}'"</span>
@@ -529,11 +539,12 @@
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s1">'Poking for table </span><span class="si">{self.schema}</span><span class="s1">.</span><span class="si">{self.table}</span><span class="s1">, '</span>
<span class="s1">'partition </span><span class="si">{self.partition}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'hook'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HiveMetastoreHook</span><span class="p">(</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveMetastoreHook</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span>
<span class="n">metastore_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">check_for_partition</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">)</span>
@@ -556,7 +567,7 @@
<span class="n">file_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">hook</span><span class="o">=</span><span class="n">HDFSHook</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span> <span class="o">=</span> <span class="n">hdfs_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file_size</span> <span class="o">=</span> <span class="n">file_size</span>
@@ -568,52 +579,57 @@
<span class="k">def</span> <span class="nf">filter_for_filesize</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Will test the filepath result and test if its size is at least self.filesize</span>
+
<span class="sd"> :param result: a list of dicts returned by Snakebite ls</span>
<span class="sd"> :param size: the file size in MB a file should be at least to trigger True</span>
<span class="sd"> :return: (bool) depending on the matching criteria</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">size</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Filtering for file size >= </span><span class="si">%s</span><span class="s1"> in files: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">))</span>
+ <span class="n">log</span> <span class="o">=</span> <span class="n">LoggingMixin</span><span class="p">()</span><span class="o">.</span><span class="n">log</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Filtering for file size >= </span><span class="si">%s</span><span class="s1"> in files: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">))</span>
<span class="n">size</span> <span class="o">*=</span> <span class="n">settings</span><span class="o">.</span><span class="n">MEGABYTE</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">result</span> <span class="k">if</span> <span class="n">x</span><span class="p">[</span><span class="s1">'length'</span><span class="p">]</span> <span class="o">>=</span> <span class="n">size</span><span class="p">]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after size filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after size filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">return</span> <span class="n">result</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">filter_for_ignored_ext</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">ignored_ext</span><span class="p">,</span> <span class="n">ignore_copying</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Will filter if instructed to do so the result to remove matching criteria</span>
+
<span class="sd"> :param result: (list) of dicts returned by Snakebite ls</span>
-<span class="sd"> :param ignored_ext: (list) of ignored extentions</span>
+<span class="sd"> :param ignored_ext: (list) of ignored extensions</span>
<span class="sd"> :param ignore_copying: (bool) shall we ignore ?</span>
-<span class="sd"> :return:</span>
+<span class="sd"> :return: (list) of dicts which were not removed</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">ignore_copying</span><span class="p">:</span>
+ <span class="n">log</span> <span class="o">=</span> <span class="n">LoggingMixin</span><span class="p">()</span><span class="o">.</span><span class="n">log</span>
<span class="n">regex_builder</span> <span class="o">=</span> <span class="s2">"^.*\.(</span><span class="si">%s</span><span class="s2">$)$"</span> <span class="o">%</span> <span class="s1">'$|'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">ignored_ext</span><span class="p">)</span>
<span class="n">ignored_extentions_regex</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">regex_builder</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Filtering result for ignored extentions: </span><span class="si">%s</span><span class="s1"> in files </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">ignored_extentions_regex</span><span class="o">.</span><span class="n">pattern</span><span class="p">,</span>
- <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">))</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span>
+ <span class="s1">'Filtering result for ignored extensions: </span><span class="si">%s</span><span class="s1"> in files </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">ignored_extentions_regex</span><span class="o">.</span><span class="n">pattern</span><span class="p">,</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">],</span> <span class="n">result</span><span class="p">)</span>
+ <span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">result</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">ignored_extentions_regex</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">x</span><span class="p">[</span><span class="s1">'path'</span><span class="p">])]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after ext filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: after ext filter result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="k">return</span> <span class="n">result</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">sb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"snakebite"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">WARNING</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1"> '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># IMOO it's not right here, as there no raise of any kind.</span>
<span class="c1"># if the filepath is let's say '/data/mydirectory', it's correct but if it is '/data/mydirectory/*',</span>
<span class="c1"># it's not correct as the directory exists and sb does not raise any error</span>
<span class="c1"># here is a quick fix</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">[</span><span class="n">f</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">sb</span><span class="o">.</span><span class="n">ls</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">],</span> <span class="n">include_toplevel</span><span class="o">=</span><span class="kc">False</span><span class="p">)]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'HdfsSensor.poke: result is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">result</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">filter_for_ignored_ext</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">ignored_ext</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">ignore_copying</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">filter_for_filesize</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">file_size</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Caught an exception !: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Caught an exception !: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="k">return</span> <span class="kc">False</span>
@@ -629,14 +645,14 @@
<span class="n">filepath</span><span class="p">,</span>
<span class="n">webhdfs_conn_id</span><span class="o">=</span><span class="s1">'webhdfs_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">WebHdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">WebHdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
<span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span> <span class="o">=</span> <span class="n">webhdfs_conn_id</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">c</span> <span class="o">=</span> <span class="n">airflow</span><span class="o">.</span><span class="n">hooks</span><span class="o">.</span><span class="n">webhdfs_hook</span><span class="o">.</span><span class="n">WebHDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1"> '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.webhdfs_hook</span> <span class="k">import</span> <span class="n">WebHDFSHook</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="n">WebHDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for file </span><span class="si">{self.filepath}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">return</span> <span class="n">c</span><span class="o">.</span><span class="n">check_for_path</span><span class="p">(</span><span class="n">hdfs_path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">)</span>
@@ -654,8 +670,8 @@
<span class="sd"> :param wildcard_match: whether the bucket_key should be interpreted as a</span>
<span class="sd"> Unix wildcard pattern</span>
<span class="sd"> :type wildcard_match: bool</span>
-<span class="sd"> :param s3_conn_id: a reference to the s3 connection</span>
-<span class="sd"> :type s3_conn_id: str</span>
+<span class="sd"> :param aws_conn_id: a reference to the s3 connection</span>
+<span class="sd"> :type aws_conn_id: str</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket_key'</span><span class="p">,</span> <span class="s1">'bucket_name'</span><span class="p">)</span>
@@ -664,9 +680,9 @@
<span class="bp">self</span><span class="p">,</span> <span class="n">bucket_key</span><span class="p">,</span>
<span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">wildcard_match</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
- <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">S3KeySensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3KeySensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="c1"># Parse</span>
<span class="k">if</span> <span class="n">bucket_name</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">parsed_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">bucket_key</span><span class="p">)</span>
@@ -681,13 +697,13 @@
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span> <span class="o">=</span> <span class="n">bucket_key</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="kn">import</span> <span class="nn">airflow.hooks.S3_hook</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">airflow</span><span class="o">.</span><span class="n">hooks</span><span class="o">.</span><span class="n">S3_hook</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
<span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for key : </span><span class="si">{full_url}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for key : </span><span class="si">{full_url}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
<span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
@@ -718,21 +734,21 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span>
<span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">'/'</span><span class="p">,</span>
- <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">S3PrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3PrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="c1"># Parse</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span>
<span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
<span class="bp">self</span><span class="o">.</span><span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="n">bucket_name</span> <span class="o">+</span> <span class="s1">'/'</span> <span class="o">+</span> <span class="n">prefix</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for prefix : </span><span class="si">{self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
- <span class="s1">'in bucket s3://</span><span class="si">{self.bucket_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
- <span class="kn">import</span> <span class="nn">airflow.hooks.S3_hook</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">airflow</span><span class="o">.</span><span class="n">hooks</span><span class="o">.</span><span class="n">S3_hook</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for prefix : </span><span class="si">{self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'in bucket s3://</span><span class="si">{self.bucket_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
<span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_prefix</span><span class="p">(</span>
<span class="n">prefix</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">prefix</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
@@ -750,13 +766,12 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">target_time</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">TimeSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">target_time</span> <span class="o">=</span> <span class="n">target_time</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s1">'Checking if the time (</span><span class="si">{0}</span><span class="s1">) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">target_time</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time (</span><span class="si">%s</span><span class="s1">) has come'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span>
<span class="k">class</span> <span class="nc">TimeDeltaSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
@@ -773,15 +788,15 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">delta</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">TimeDeltaSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeDeltaSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">delta</span> <span class="o">=</span> <span class="n">delta</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">dag</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'dag'</span><span class="p">]</span>
<span class="n">target_dttm</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">following_schedule</span><span class="p">(</span><span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">])</span>
<span class="n">target_dttm</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">delta</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time (</span><span class="si">{0}</span><span class="s1">) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">target_dttm</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">></span> <span class="n">target_dttm</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time (</span><span class="si">%s</span><span class="s1">) has come'</span><span class="p">,</span> <span class="n">target_dttm</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">></span> <span class="n">target_dttm</span>
<span class="k">class</span> <span class="nc">HttpSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
@@ -791,10 +806,12 @@
<span class="sd"> :param http_conn_id: The connection to run the sensor against</span>
<span class="sd"> :type http_conn_id: string</span>
+<span class="sd"> :param method: The HTTP request method to use</span>
+<span class="sd"> :type method: string</span>
<span class="sd"> :param endpoint: The relative part of the full url</span>
<span class="sd"> :type endpoint: string</span>
-<span class="sd"> :param params: The parameters to be added to the GET url</span>
-<span class="sd"> :type params: a dictionary of string key/value pairs</span>
+<span class="sd"> :param request_params: The parameters to be added to the GET url</span>
+<span class="sd"> :type request_params: a dictionary of string key/value pairs</span>
<span class="sd"> :param headers: The HTTP headers to be added to the GET request</span>
<span class="sd"> :type headers: a dictionary of string key/value pairs</span>
<span class="sd"> :param response_check: A check against the 'requests' response object.</span>
@@ -806,31 +823,34 @@
<span class="sd"> depends on the option that's being modified.</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">'request_params'</span><span class="p">)</span>
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">endpoint</span><span class="p">,</span>
<span class="n">http_conn_id</span><span class="o">=</span><span class="s1">'http_default'</span><span class="p">,</span>
- <span class="n">params</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">method</span><span class="o">=</span><span class="s1">'GET'</span><span class="p">,</span>
+ <span class="n">request_params</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">response_check</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">extra_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HttpSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HttpSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="n">params</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">request_params</span> <span class="o">=</span> <span class="n">request_params</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_check</span> <span class="o">=</span> <span class="n">response_check</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">http_hook</span><span class="o">.</span><span class="n">HttpHook</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s1">'GET'</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="n">http_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">HttpHook</span><span class="p">(</span>
+ <span class="n">method</span><span class="o">=</span><span class="n">method</span><span class="p">,</span>
+ <span class="n">http_conn_id</span><span class="o">=</span><span class="n">http_conn_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
- <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span>
+ <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">request_params</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">,</span>
<span class="n">extra_options</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">:</span>
[33/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/bigquery_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/bigquery_hook.html b/_modules/airflow/contrib/hooks/bigquery_hook.html
new file mode 100644
index 0000000..1926c79
--- /dev/null
+++ b/_modules/airflow/contrib/hooks/bigquery_hook.html
@@ -0,0 +1,1279 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.hooks.bigquery_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.hooks.bigquery_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.hooks.bigquery_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="sd">"""</span>
+<span class="sd">This module contains a BigQuery Hook, as well as a very basic PEP 249</span>
+<span class="sd">implementation for BigQuery.</span>
+<span class="sd">"""</span>
+
+<span class="kn">import</span> <span class="nn">time</span>
+
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span><span class="p">,</span> <span class="n">HttpError</span>
+<span class="kn">from</span> <span class="nn">googleapiclient</span> <span class="k">import</span> <span class="n">errors</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">range</span>
+<span class="kn">from</span> <span class="nn">pandas_gbq.gbq</span> <span class="k">import</span> <span class="n">GbqConnector</span><span class="p">,</span> \
+ <span class="n">_parse_data</span> <span class="k">as</span> <span class="n">gbq_parse_data</span><span class="p">,</span> \
+ <span class="n">_check_google_client_version</span> <span class="k">as</span> <span class="n">gbq_check_google_client_version</span><span class="p">,</span> \
+ <span class="n">_test_google_api_imports</span> <span class="k">as</span> <span class="n">gbq_test_google_api_imports</span>
+<span class="kn">from</span> <span class="nn">pandas.tools.merge</span> <span class="k">import</span> <span class="n">concat</span>
+<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="k">import</span> <span class="n">DbApiHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
+
+<div class="viewcode-block" id="BigQueryHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook">[docs]</a><span class="k">class</span> <span class="nc">BigQueryHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">,</span> <span class="n">DbApiHook</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with BigQuery. This hook uses the Google Cloud Platform</span>
+<span class="sd"> connection.</span>
+<span class="sd"> """</span>
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'bigquery_conn_id'</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
+ <span class="n">conn_id</span><span class="o">=</span><span class="n">bigquery_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="n">delegate_to</span><span class="p">)</span>
+
+<div class="viewcode-block" id="BigQueryHook.get_conn"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a BigQuery PEP 249 connection object.</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_service</span><span class="p">()</span>
+ <span class="n">project</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_field</span><span class="p">(</span><span class="s1">'project'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">BigQueryConnection</span><span class="p">(</span><span class="n">service</span><span class="o">=</span><span class="n">service</span><span class="p">,</span> <span class="n">project_id</span><span class="o">=</span><span class="n">project</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="BigQueryHook.get_service"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_service">[docs]</a> <span class="k">def</span> <span class="nf">get_service</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a BigQuery service object.</span>
+<span class="sd"> """</span>
+ <span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'bigquery'</span><span class="p">,</span> <span class="s1">'v2'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="BigQueryHook.insert_rows"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.insert_rows">[docs]</a> <span class="k">def</span> <span class="nf">insert_rows</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">target_fields</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">commit_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Insertion is currently unsupported. Theoretically, you could use</span>
+<span class="sd"> BigQuery's streaming API to insert rows into a table, but this hasn't</span>
+<span class="sd"> been implemented.</span>
+<span class="sd"> """</span>
+ <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="BigQueryHook.get_pandas_df"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="s1">'legacy'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a Pandas DataFrame for the results produced by a BigQuery</span>
+<span class="sd"> query. The DbApiHook method must be overridden because Pandas</span>
+<span class="sd"> doesn't support PEP 249 connections, except for SQLite. See:</span>
+
+<span class="sd"> https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447</span>
+<span class="sd"> https://github.com/pydata/pandas/issues/6900</span>
+
+<span class="sd"> :param bql: The BigQuery SQL to execute.</span>
+<span class="sd"> :type bql: string</span>
+<span class="sd"> :param parameters: The parameters to render the SQL query with (not used, leave to override superclass method)</span>
+<span class="sd"> :type parameters: mapping or iterable</span>
+<span class="sd"> :param dialect: Dialect of BigQuery SQL – legacy SQL or standard SQL</span>
+<span class="sd"> :type dialect: string in {'legacy', 'standard'}, default 'legacy'</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_service</span><span class="p">()</span>
+ <span class="n">project</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_field</span><span class="p">(</span><span class="s1">'project'</span><span class="p">)</span>
+ <span class="n">connector</span> <span class="o">=</span> <span class="n">BigQueryPandasConnector</span><span class="p">(</span><span class="n">project</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
+ <span class="n">schema</span><span class="p">,</span> <span class="n">pages</span> <span class="o">=</span> <span class="n">connector</span><span class="o">.</span><span class="n">run_query</span><span class="p">(</span><span class="n">bql</span><span class="p">)</span>
+ <span class="n">dataframe_list</span> <span class="o">=</span> <span class="p">[]</span>
+
+ <span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">pages</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">page</span> <span class="o">=</span> <span class="n">pages</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
+ <span class="n">dataframe_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">gbq_parse_data</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">page</span><span class="p">))</span>
+
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">dataframe_list</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">concat</span><span class="p">(</span><span class="n">dataframe_list</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">gbq_parse_data</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="p">[])</span></div>
+
+<div class="viewcode-block" id="BigQueryHook.table_exists"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.table_exists">[docs]</a> <span class="k">def</span> <span class="nf">table_exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks for the existence of a table in Google BigQuery.</span>
+
+<span class="sd"> :param project_id: The Google cloud project in which to look for the table. The connection supplied to the hook</span>
+<span class="sd"> must provide access to the specified project.</span>
+<span class="sd"> :type project_id: string</span>
+<span class="sd"> :param dataset_id: The name of the dataset in which to look for the table.</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type dataset_id: string</span>
+<span class="sd"> :param table_id: The name of the table to check the existence of.</span>
+<span class="sd"> :type table_id: string</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_service</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span>
+ <span class="n">tableId</span><span class="o">=</span><span class="n">table_id</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div></div>
+
+
+<span class="k">class</span> <span class="nc">BigQueryPandasConnector</span><span class="p">(</span><span class="n">GbqConnector</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> This connector behaves identically to GbqConnector (from Pandas), except</span>
+<span class="sd"> that it allows the service to be injected, and disables a call to</span>
+<span class="sd"> self.get_credentials(). This allows Airflow to use BigQuery with Pandas</span>
+<span class="sd"> without forcing a three legged OAuth connection. Instead, we can inject</span>
+<span class="sd"> service account credentials into the binding.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">reauth</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="s1">'legacy'</span><span class="p">):</span>
+ <span class="n">gbq_check_google_client_version</span><span class="p">()</span>
+ <span class="n">gbq_test_google_api_imports</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">reauth</span> <span class="o">=</span> <span class="n">reauth</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">service</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">verbose</span> <span class="o">=</span> <span class="n">verbose</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span>
+
+
+<span class="k">class</span> <span class="nc">BigQueryConnection</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> BigQuery does not have a notion of a persistent connection. Thus, these</span>
+<span class="sd"> objects are small stateless factories for cursors, which do all the real</span>
+<span class="sd"> work.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_args</span> <span class="o">=</span> <span class="n">args</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_kwargs</span> <span class="o">=</span> <span class="n">kwargs</span>
+
+ <span class="k">def</span> <span class="nf">close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">""" BigQueryConnection does not have anything to close. """</span>
+ <span class="k">pass</span>
+
+ <span class="k">def</span> <span class="nf">commit</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">""" BigQueryConnection does not support transactions. """</span>
+ <span class="k">pass</span>
+
+ <span class="k">def</span> <span class="nf">cursor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">""" Return a new :py:class:`Cursor` object using the connection. """</span>
+ <span class="k">return</span> <span class="n">BigQueryCursor</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">_args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">_kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">rollback</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span>
+ <span class="s2">"BigQueryConnection does not have transactions"</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">BigQueryBaseCursor</span><span class="p">(</span><span class="n">LoggingMixin</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> The BigQuery base cursor contains helper methods to execute queries against</span>
+<span class="sd"> BigQuery. The methods can be used directly by operators, in cases where a</span>
+<span class="sd"> PEP 249 cursor isn't needed.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">project_id</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">service</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+
+ <span class="k">def</span> <span class="nf">run_query</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">bql</span><span class="p">,</span> <span class="n">destination_dataset_table</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="n">write_disposition</span> <span class="o">=</span> <span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
+ <span class="n">allow_large_results</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">udf_config</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="n">use_legacy_sql</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">maximum_billing_tier</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
+ <span class="n">query_params</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a BigQuery SQL query. Optionally persists results in a BigQuery</span>
+<span class="sd"> table. See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs</span>
+
+<span class="sd"> For more details about these parameters.</span>
+
+<span class="sd"> :param bql: The BigQuery SQL to execute.</span>
+<span class="sd"> :type bql: string</span>
+<span class="sd"> :param destination_dataset_table: The dotted <dataset>.<table></span>
+<span class="sd"> BigQuery table to save the query results.</span>
+<span class="sd"> :param write_disposition: What to do if the table already exists in</span>
+<span class="sd"> BigQuery.</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param create_disposition: Specifies whether the job is allowed to create new tables.</span>
+<span class="sd"> :type create_disposition: string</span>
+<span class="sd"> :param allow_large_results: Whether to allow large results.</span>
+<span class="sd"> :type allow_large_results: boolean</span>
+<span class="sd"> :param udf_config: The User Defined Function configuration for the query.</span>
+<span class="sd"> See https://cloud.google.com/bigquery/user-defined-functions for details.</span>
+<span class="sd"> :type udf_config: list</span>
+<span class="sd"> :param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).</span>
+<span class="sd"> :type use_legacy_sql: boolean</span>
+<span class="sd"> :param maximum_billing_tier: Positive integer that serves as a multiplier of the basic price.</span>
+<span class="sd"> :type maximum_billing_tier: integer</span>
+<span class="sd"> """</span>
+ <span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'query'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'query'</span><span class="p">:</span> <span class="n">bql</span><span class="p">,</span>
+ <span class="s1">'useLegacySql'</span><span class="p">:</span> <span class="n">use_legacy_sql</span><span class="p">,</span>
+ <span class="s1">'maximumBillingTier'</span><span class="p">:</span> <span class="n">maximum_billing_tier</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="n">destination_dataset_table</span><span class="p">:</span>
+ <span class="k">assert</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">destination_dataset_table</span><span class="p">,</span> <span class="p">(</span>
+ <span class="s1">'Expected destination_dataset_table in the format of '</span>
+ <span class="s1">'<dataset>.<table>. Got: </span><span class="si">{}</span><span class="s1">'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">destination_dataset_table</span><span class="p">)</span>
+ <span class="n">destination_project</span><span class="p">,</span> <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">destination_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">)</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'query'</span><span class="p">]</span><span class="o">.</span><span class="n">update</span><span class="p">({</span>
+ <span class="s1">'allowLargeResults'</span><span class="p">:</span> <span class="n">allow_large_results</span><span class="p">,</span>
+ <span class="s1">'writeDisposition'</span><span class="p">:</span> <span class="n">write_disposition</span><span class="p">,</span>
+ <span class="s1">'createDisposition'</span><span class="p">:</span> <span class="n">create_disposition</span><span class="p">,</span>
+ <span class="s1">'destinationTable'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="n">destination_project</span><span class="p">,</span>
+ <span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">destination_dataset</span><span class="p">,</span>
+ <span class="s1">'tableId'</span><span class="p">:</span> <span class="n">destination_table</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="p">})</span>
+ <span class="k">if</span> <span class="n">udf_config</span><span class="p">:</span>
+ <span class="k">assert</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">udf_config</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'query'</span><span class="p">]</span><span class="o">.</span><span class="n">update</span><span class="p">({</span>
+ <span class="s1">'userDefinedFunctionResources'</span><span class="p">:</span> <span class="n">udf_config</span>
+ <span class="p">})</span>
+
+ <span class="k">if</span> <span class="n">query_params</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'query'</span><span class="p">][</span><span class="s1">'queryParameters'</span><span class="p">]</span> <span class="o">=</span> <span class="n">query_params</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_with_configuration</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">run_extract</span><span class="p">(</span> <span class="c1"># noqa</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">source_project_dataset_table</span><span class="p">,</span> <span class="n">destination_cloud_storage_uris</span><span class="p">,</span>
+ <span class="n">compression</span><span class="o">=</span><span class="s1">'NONE'</span><span class="p">,</span> <span class="n">export_format</span><span class="o">=</span><span class="s1">'CSV'</span><span class="p">,</span> <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">print_header</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a BigQuery extract command to copy data from BigQuery to</span>
+<span class="sd"> Google Cloud Storage. See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs</span>
+
+<span class="sd"> For more details about these parameters.</span>
+
+<span class="sd"> :param source_project_dataset_table: The dotted <dataset>.<table></span>
+<span class="sd"> BigQuery table to use as the source data.</span>
+<span class="sd"> :type source_project_dataset_table: string</span>
+<span class="sd"> :param destination_cloud_storage_uris: The destination Google Cloud</span>
+<span class="sd"> Storage URI (e.g. gs://some-bucket/some-file.txt). Follows</span>
+<span class="sd"> convention defined here:</span>
+<span class="sd"> https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple</span>
+<span class="sd"> :type destination_cloud_storage_uris: list</span>
+<span class="sd"> :param compression: Type of compression to use.</span>
+<span class="sd"> :type compression: string</span>
+<span class="sd"> :param export_format: File format to export.</span>
+<span class="sd"> :type export_format: string</span>
+<span class="sd"> :param field_delimiter: The delimiter to use when extracting to a CSV.</span>
+<span class="sd"> :type field_delimiter: string</span>
+<span class="sd"> :param print_header: Whether to print a header for a CSV file extract.</span>
+<span class="sd"> :type print_header: boolean</span>
+<span class="sd"> """</span>
+
+ <span class="n">source_project</span><span class="p">,</span> <span class="n">source_dataset</span><span class="p">,</span> <span class="n">source_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">source_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">var_name</span><span class="o">=</span><span class="s1">'source_project_dataset_table'</span><span class="p">)</span>
+
+ <span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'extract'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'sourceTable'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="n">source_project</span><span class="p">,</span>
+ <span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">source_dataset</span><span class="p">,</span>
+ <span class="s1">'tableId'</span><span class="p">:</span> <span class="n">source_table</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="s1">'compression'</span><span class="p">:</span> <span class="n">compression</span><span class="p">,</span>
+ <span class="s1">'destinationUris'</span><span class="p">:</span> <span class="n">destination_cloud_storage_uris</span><span class="p">,</span>
+ <span class="s1">'destinationFormat'</span><span class="p">:</span> <span class="n">export_format</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="n">export_format</span> <span class="o">==</span> <span class="s1">'CSV'</span><span class="p">:</span>
+ <span class="c1"># Only set fieldDelimiter and printHeader fields if using CSV.</span>
+ <span class="c1"># Google does not like it if you set these fields for other export</span>
+ <span class="c1"># formats.</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'extract'</span><span class="p">][</span><span class="s1">'fieldDelimiter'</span><span class="p">]</span> <span class="o">=</span> <span class="n">field_delimiter</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'extract'</span><span class="p">][</span><span class="s1">'printHeader'</span><span class="p">]</span> <span class="o">=</span> <span class="n">print_header</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_with_configuration</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">run_copy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">source_project_dataset_tables</span><span class="p">,</span>
+ <span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a BigQuery copy command to copy data from one BigQuery table</span>
+<span class="sd"> to another. See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy</span>
+
+<span class="sd"> For more details about these parameters.</span>
+
+<span class="sd"> :param source_project_dataset_tables: One or more dotted</span>
+<span class="sd"> (project:|project.)<dataset>.<table></span>
+<span class="sd"> BigQuery tables to use as the source data. Use a list if there are</span>
+<span class="sd"> multiple source tables.</span>
+<span class="sd"> If <project> is not included, project will be the project defined</span>
+<span class="sd"> in the connection json.</span>
+<span class="sd"> :type source_project_dataset_tables: list|string</span>
+<span class="sd"> :param destination_project_dataset_table: The destination BigQuery</span>
+<span class="sd"> table. Format is: (project:|project.)<dataset>.<table></span>
+<span class="sd"> :type destination_project_dataset_table: string</span>
+<span class="sd"> :param write_disposition: The write disposition if the table already exists.</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param create_disposition: The create disposition if the table doesn't exist.</span>
+<span class="sd"> :type create_disposition: string</span>
+<span class="sd"> """</span>
+ <span class="n">source_project_dataset_tables</span> <span class="o">=</span> <span class="p">(</span>
+ <span class="p">[</span><span class="n">source_project_dataset_tables</span><span class="p">]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source_project_dataset_tables</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
+ <span class="k">else</span> <span class="n">source_project_dataset_tables</span><span class="p">)</span>
+
+ <span class="n">source_project_dataset_tables_fixup</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="k">for</span> <span class="n">source_project_dataset_table</span> <span class="ow">in</span> <span class="n">source_project_dataset_tables</span><span class="p">:</span>
+ <span class="n">source_project</span><span class="p">,</span> <span class="n">source_dataset</span><span class="p">,</span> <span class="n">source_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">source_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">var_name</span><span class="o">=</span><span class="s1">'source_project_dataset_table'</span><span class="p">)</span>
+ <span class="n">source_project_dataset_tables_fixup</span><span class="o">.</span><span class="n">append</span><span class="p">({</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="n">source_project</span><span class="p">,</span>
+ <span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">source_dataset</span><span class="p">,</span>
+ <span class="s1">'tableId'</span><span class="p">:</span> <span class="n">source_table</span>
+ <span class="p">})</span>
+
+ <span class="n">destination_project</span><span class="p">,</span> <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">)</span>
+ <span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'copy'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'createDisposition'</span><span class="p">:</span> <span class="n">create_disposition</span><span class="p">,</span>
+ <span class="s1">'writeDisposition'</span><span class="p">:</span> <span class="n">write_disposition</span><span class="p">,</span>
+ <span class="s1">'sourceTables'</span><span class="p">:</span> <span class="n">source_project_dataset_tables_fixup</span><span class="p">,</span>
+ <span class="s1">'destinationTable'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="n">destination_project</span><span class="p">,</span>
+ <span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">destination_dataset</span><span class="p">,</span>
+ <span class="s1">'tableId'</span><span class="p">:</span> <span class="n">destination_table</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_with_configuration</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">run_load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">schema_fields</span><span class="p">,</span> <span class="n">source_uris</span><span class="p">,</span>
+ <span class="n">source_format</span><span class="o">=</span><span class="s1">'CSV'</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
+ <span class="n">skip_leading_rows</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
+ <span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
+ <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">max_bad_records</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
+ <span class="n">quote_character</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">allow_quoted_newlines</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">allow_jagged_rows</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">schema_update_options</span><span class="o">=</span><span class="p">(),</span>
+ <span class="n">src_fmt_configs</span><span class="o">=</span><span class="p">{}):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a BigQuery load command to load data from Google Cloud Storage</span>
+<span class="sd"> to BigQuery. See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs</span>
+
+<span class="sd"> For more details about these parameters.</span>
+
+<span class="sd"> :param destination_project_dataset_table:</span>
+<span class="sd"> The dotted (<project>.|<project>:)<dataset>.<table> BigQuery table to load</span>
+<span class="sd"> data into. If <project> is not included, project will be the project defined</span>
+<span class="sd"> in the connection json.</span>
+<span class="sd"> :type destination_project_dataset_table: string</span>
+<span class="sd"> :param schema_fields: The schema field list as defined here:</span>
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load</span>
+<span class="sd"> :type schema_fields: list</span>
+<span class="sd"> :param source_uris: The source Google Cloud</span>
+<span class="sd"> Storage URI (e.g. gs://some-bucket/some-file.txt). A single wild</span>
+<span class="sd"> per-object name can be used.</span>
+<span class="sd"> :type source_uris: list</span>
+<span class="sd"> :param source_format: File format to export.</span>
+<span class="sd"> :type source_format: string</span>
+<span class="sd"> :param create_disposition: The create disposition if the table doesn't exist.</span>
+<span class="sd"> :type create_disposition: string</span>
+<span class="sd"> :param skip_leading_rows: Number of rows to skip when loading from a CSV.</span>
+<span class="sd"> :type skip_leading_rows: int</span>
+<span class="sd"> :param write_disposition: The write disposition if the table already exists.</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param field_delimiter: The delimiter to use when loading from a CSV.</span>
+<span class="sd"> :type field_delimiter: string</span>
+<span class="sd"> :param max_bad_records: The maximum number of bad records that BigQuery can</span>
+<span class="sd"> ignore when running the job.</span>
+<span class="sd"> :type max_bad_records: int</span>
+<span class="sd"> :param quote_character: The value that is used to quote data sections in a CSV file.</span>
+<span class="sd"> :type quote_character: string</span>
+<span class="sd"> :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not (false).</span>
+<span class="sd"> :type allow_quoted_newlines: boolean</span>
+<span class="sd"> :param allow_jagged_rows: Accept rows that are missing trailing optional columns.</span>
+<span class="sd"> The missing values are treated as nulls. If false, records with missing trailing columns</span>
+<span class="sd"> are treated as bad records, and if there are too many bad records, an invalid error is</span>
+<span class="sd"> returned in the job result. Only applicable when soure_format is CSV.</span>
+<span class="sd"> :type allow_jagged_rows: bool</span>
+<span class="sd"> :param schema_update_options: Allows the schema of the desitination</span>
+<span class="sd"> table to be updated as a side effect of the load job.</span>
+<span class="sd"> :type schema_update_options: list</span>
+<span class="sd"> :param src_fmt_configs: configure optional fields specific to the source format</span>
+<span class="sd"> :type src_fmt_configs: dict</span>
+<span class="sd"> """</span>
+
+ <span class="c1"># bigquery only allows certain source formats</span>
+ <span class="c1"># we check to make sure the passed source format is valid</span>
+ <span class="c1"># if it's not, we raise a ValueError</span>
+ <span class="c1"># Refer to this link for more details:</span>
+ <span class="c1"># https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat</span>
+ <span class="n">source_format</span> <span class="o">=</span> <span class="n">source_format</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
+ <span class="n">allowed_formats</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"CSV"</span><span class="p">,</span> <span class="s2">"NEWLINE_DELIMITED_JSON"</span><span class="p">,</span> <span class="s2">"AVRO"</span><span class="p">,</span> <span class="s2">"GOOGLE_SHEETS"</span><span class="p">,</span> <span class="s2">"DATASTORE_BACKUP"</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">source_format</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">allowed_formats</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"</span><span class="si">{0}</span><span class="s2"> is not a valid source format. "</span>
+ <span class="s2">"Please use one of the following types: </span><span class="si">{1}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">source_format</span><span class="p">,</span> <span class="n">allowed_formats</span><span class="p">))</span>
+
+ <span class="c1"># bigquery also allows you to define how you want a table's schema to change</span>
+ <span class="c1"># as a side effect of a load</span>
+ <span class="c1"># for more details:</span>
+ <span class="c1"># https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schemaUpdateOptions</span>
+ <span class="n">allowed_schema_update_options</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="s1">'ALLOW_FIELD_ADDITION'</span><span class="p">,</span>
+ <span class="s2">"ALLOW_FIELD_RELAXATION"</span>
+ <span class="p">]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">set</span><span class="p">(</span><span class="n">allowed_schema_update_options</span><span class="p">)</span><span class="o">.</span><span class="n">issuperset</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">schema_update_options</span><span class="p">)):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s2">"</span><span class="si">{0}</span><span class="s2"> contains invalid schema update options. "</span>
+ <span class="s2">"Please only use one or more of the following options: </span><span class="si">{1}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">schema_update_options</span><span class="p">,</span> <span class="n">allowed_schema_update_options</span><span class="p">)</span>
+ <span class="p">)</span>
+
+ <span class="n">destination_project</span><span class="p">,</span> <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">var_name</span><span class="o">=</span><span class="s1">'destination_project_dataset_table'</span><span class="p">)</span>
+
+ <span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'load'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'createDisposition'</span><span class="p">:</span> <span class="n">create_disposition</span><span class="p">,</span>
+ <span class="s1">'destinationTable'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="n">destination_project</span><span class="p">,</span>
+ <span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">destination_dataset</span><span class="p">,</span>
+ <span class="s1">'tableId'</span><span class="p">:</span> <span class="n">destination_table</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="s1">'sourceFormat'</span><span class="p">:</span> <span class="n">source_format</span><span class="p">,</span>
+ <span class="s1">'sourceUris'</span><span class="p">:</span> <span class="n">source_uris</span><span class="p">,</span>
+ <span class="s1">'writeDisposition'</span><span class="p">:</span> <span class="n">write_disposition</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+ <span class="k">if</span> <span class="n">schema_fields</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'schema'</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'fields'</span><span class="p">:</span> <span class="n">schema_fields</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="n">schema_update_options</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">write_disposition</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">"WRITE_APPEND"</span><span class="p">,</span> <span class="s2">"WRITE_TRUNCATE"</span><span class="p">]:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s2">"schema_update_options is only "</span>
+ <span class="s2">"allowed if write_disposition is "</span>
+ <span class="s2">"'WRITE_APPEND' or 'WRITE_TRUNCATE'."</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Adding experimental "</span>
+ <span class="s2">"'schemaUpdateOptions': </span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">schema_update_options</span><span class="p">)</span>
+ <span class="p">)</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'schemaUpdateOptions'</span><span class="p">]</span> <span class="o">=</span> <span class="n">schema_update_options</span>
+
+ <span class="k">if</span> <span class="n">max_bad_records</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'maxBadRecords'</span><span class="p">]</span> <span class="o">=</span> <span class="n">max_bad_records</span>
+
+ <span class="c1"># if following fields are not specified in src_fmt_configs,</span>
+ <span class="c1"># honor the top-level params for backward-compatibility</span>
+ <span class="k">if</span> <span class="s1">'skipLeadingRows'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">src_fmt_configs</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'skipLeadingRows'</span><span class="p">]</span> <span class="o">=</span> <span class="n">skip_leading_rows</span>
+ <span class="k">if</span> <span class="s1">'fieldDelimiter'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">src_fmt_configs</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'fieldDelimiter'</span><span class="p">]</span> <span class="o">=</span> <span class="n">field_delimiter</span>
+ <span class="k">if</span> <span class="n">quote_character</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'quote'</span><span class="p">]</span> <span class="o">=</span> <span class="n">quote_character</span>
+ <span class="k">if</span> <span class="n">allow_quoted_newlines</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'allowQuotedNewlines'</span><span class="p">]</span> <span class="o">=</span> <span class="n">allow_quoted_newlines</span>
+
+ <span class="n">src_fmt_to_configs_mapping</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'CSV'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'allowJaggedRows'</span><span class="p">,</span> <span class="s1">'allowQuotedNewlines'</span><span class="p">,</span> <span class="s1">'autodetect'</span><span class="p">,</span>
+ <span class="s1">'fieldDelimiter'</span><span class="p">,</span> <span class="s1">'skipLeadingRows'</span><span class="p">,</span> <span class="s1">'ignoreUnknownValues'</span><span class="p">,</span>
+ <span class="s1">'nullMarker'</span><span class="p">,</span> <span class="s1">'quote'</span><span class="p">],</span>
+ <span class="s1">'DATASTORE_BACKUP'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'projectionFields'</span><span class="p">],</span>
+ <span class="s1">'NEWLINE_DELIMITED_JSON'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'autodetect'</span><span class="p">,</span> <span class="s1">'ignoreUnknownValues'</span><span class="p">],</span>
+ <span class="s1">'AVRO'</span><span class="p">:</span> <span class="p">[],</span>
+ <span class="p">}</span>
+ <span class="n">valid_configs</span> <span class="o">=</span> <span class="n">src_fmt_to_configs_mapping</span><span class="p">[</span><span class="n">source_format</span><span class="p">]</span>
+ <span class="n">src_fmt_configs</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">src_fmt_configs</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">valid_configs</span><span class="p">}</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">]</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">src_fmt_configs</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">allow_jagged_rows</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'allowJaggedRows'</span><span class="p">]</span> <span class="o">=</span> <span class="n">allow_jagged_rows</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_with_configuration</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">run_with_configuration</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">configuration</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a BigQuery SQL query. See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs</span>
+
+<span class="sd"> For more details about the configuration parameter.</span>
+
+<span class="sd"> :param configuration: The configuration parameter maps directly to</span>
+<span class="sd"> BigQuery's configuration field in the job object. See</span>
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs for</span>
+<span class="sd"> details.</span>
+<span class="sd"> """</span>
+ <span class="n">jobs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">jobs</span><span class="p">()</span>
+ <span class="n">job_data</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'configuration'</span><span class="p">:</span> <span class="n">configuration</span>
+ <span class="p">}</span>
+
+ <span class="c1"># Send query and wait for reply.</span>
+ <span class="n">query_reply</span> <span class="o">=</span> <span class="n">jobs</span> \
+ <span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">job_data</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="n">job_id</span> <span class="o">=</span> <span class="n">query_reply</span><span class="p">[</span><span class="s1">'jobReference'</span><span class="p">][</span><span class="s1">'jobId'</span><span class="p">]</span>
+
+ <span class="c1"># Wait for query to finish.</span>
+ <span class="n">keep_polling_job</span> <span class="o">=</span> <span class="kc">True</span>
+ <span class="k">while</span> <span class="p">(</span><span class="n">keep_polling_job</span><span class="p">):</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">job</span> <span class="o">=</span> <span class="n">jobs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">jobId</span><span class="o">=</span><span class="n">job_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'state'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'DONE'</span><span class="p">):</span>
+ <span class="n">keep_polling_job</span> <span class="o">=</span> <span class="kc">False</span>
+ <span class="c1"># Check if job had errors.</span>
+ <span class="k">if</span> <span class="s1">'errorResult'</span> <span class="ow">in</span> <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'BigQuery job failed. Final error was: </span><span class="si">{}</span><span class="s1">. The job was: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'errorResult'</span><span class="p">],</span> <span class="n">job</span>
+ <span class="p">)</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Waiting for job to complete : </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+
+ <span class="k">except</span> <span class="n">HttpError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">err</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">500</span><span class="p">,</span> <span class="mi">503</span><span class="p">]:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1">: Retryable error, waiting for job to complete: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">err</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'BigQuery job status check failed. Final error was: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">err</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">job_id</span>
+
+ <span class="k">def</span> <span class="nf">get_schema</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get the schema for a given datset.table.</span>
+<span class="sd"> see https://cloud.google.com/bigquery/docs/reference/v2/tables#resource</span>
+
+<span class="sd"> :param dataset_id: the dataset ID of the requested table</span>
+<span class="sd"> :param table_id: the table ID of the requested table</span>
+<span class="sd"> :return: a table schema</span>
+<span class="sd"> """</span>
+ <span class="n">tables_resource</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span> <span class="n">tableId</span><span class="o">=</span><span class="n">table_id</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">tables_resource</span><span class="p">[</span><span class="s1">'schema'</span><span class="p">]</span>
+
+ <span class="k">def</span> <span class="nf">get_tabledata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">,</span>
+ <span class="n">max_results</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">page_token</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">start_index</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get the data of a given dataset.table.</span>
+<span class="sd"> see https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list</span>
+
+<span class="sd"> :param dataset_id: the dataset ID of the requested table.</span>
+<span class="sd"> :param table_id: the table ID of the requested table.</span>
+<span class="sd"> :param max_results: the maximum results to return.</span>
+<span class="sd"> :param page_token: page token, returned from a previous call,</span>
+<span class="sd"> identifying the result set.</span>
+<span class="sd"> :param start_index: zero based index of the starting row to read.</span>
+<span class="sd"> :return: map containing the requested rows.</span>
+<span class="sd"> """</span>
+ <span class="n">optional_params</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="k">if</span> <span class="n">max_results</span><span class="p">:</span>
+ <span class="n">optional_params</span><span class="p">[</span><span class="s1">'maxResults'</span><span class="p">]</span> <span class="o">=</span> <span class="n">max_results</span>
+ <span class="k">if</span> <span class="n">page_token</span><span class="p">:</span>
+ <span class="n">optional_params</span><span class="p">[</span><span class="s1">'pageToken'</span><span class="p">]</span> <span class="o">=</span> <span class="n">page_token</span>
+ <span class="k">if</span> <span class="n">start_index</span><span class="p">:</span>
+ <span class="n">optional_params</span><span class="p">[</span><span class="s1">'startIndex'</span><span class="p">]</span> <span class="o">=</span> <span class="n">start_index</span>
+ <span class="k">return</span> <span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tabledata</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span>
+ <span class="n">tableId</span><span class="o">=</span><span class="n">table_id</span><span class="p">,</span> <span class="o">**</span><span class="n">optional_params</span><span class="p">)</span>
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">run_table_delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">deletion_dataset_table</span><span class="p">,</span> <span class="n">ignore_if_missing</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Delete an existing table from the dataset;</span>
+<span class="sd"> If the table does not exist, return an error unless ignore_if_missing</span>
+<span class="sd"> is set to True.</span>
+
+<span class="sd"> :param deletion_dataset_table: A dotted</span>
+<span class="sd"> (<project>.|<project>:)<dataset>.<table> that indicates which table</span>
+<span class="sd"> will be deleted.</span>
+<span class="sd"> :type deletion_dataset_table: str</span>
+<span class="sd"> :param ignore_if_missing: if True, then return success even if the</span>
+<span class="sd"> requested table does not exist.</span>
+<span class="sd"> :type ignore_if_missing: boolean</span>
+<span class="sd"> :return:</span>
+<span class="sd"> """</span>
+
+ <span class="k">assert</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">deletion_dataset_table</span><span class="p">,</span> <span class="p">(</span>
+ <span class="s1">'Expected deletion_dataset_table in the format of '</span>
+ <span class="s1">'<dataset>.<table>. Got: </span><span class="si">{}</span><span class="s1">'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">deletion_dataset_table</span><span class="p">)</span>
+ <span class="n">deletion_project</span><span class="p">,</span> <span class="n">deletion_dataset</span><span class="p">,</span> <span class="n">deletion_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">deletion_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">)</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">tables_resource</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">deletion_project</span><span class="p">,</span>
+ <span class="n">datasetId</span><span class="o">=</span><span class="n">deletion_dataset</span><span class="p">,</span>
+ <span class="n">tableId</span><span class="o">=</span><span class="n">deletion_table</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Deleted table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1">.'</span><span class="p">,</span>
+ <span class="n">deletion_project</span><span class="p">,</span> <span class="n">deletion_dataset</span><span class="p">,</span> <span class="n">deletion_table</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">HttpError</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">ignore_if_missing</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'Table deletion failed. Table does not exist.'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Table does not exist. Skipping.'</span><span class="p">)</span>
+
+
+ <span class="k">def</span> <span class="nf">run_table_upsert</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_resource</span><span class="p">,</span> <span class="n">project_id</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> creates a new, empty table in the dataset;</span>
+<span class="sd"> If the table already exists, update the existing table.</span>
+<span class="sd"> Since BigQuery does not natively allow table upserts, this is not an</span>
+<span class="sd"> atomic operation.</span>
+
+<span class="sd"> :param dataset_id: the dataset to upsert the table into.</span>
+<span class="sd"> :type dataset_id: str</span>
+<span class="sd"> :param table_resource: a table resource. see</span>
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/tables#resource</span>
+<span class="sd"> :type table_resource: dict</span>
+<span class="sd"> :param project_id: the project to upsert the table into. If None,</span>
+<span class="sd"> project will be self.project_id.</span>
+<span class="sd"> :return:</span>
+<span class="sd"> """</span>
+ <span class="c1"># check to see if the table exists</span>
+ <span class="n">table_id</span> <span class="o">=</span> <span class="n">table_resource</span><span class="p">[</span><span class="s1">'tableReference'</span><span class="p">][</span><span class="s1">'tableId'</span><span class="p">]</span>
+ <span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span> <span class="k">if</span> <span class="n">project_id</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span>
+ <span class="n">tables_list_resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">table</span> <span class="ow">in</span> <span class="n">tables_list_resp</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tables'</span><span class="p">,</span> <span class="p">[]):</span>
+ <span class="k">if</span> <span class="n">table</span><span class="p">[</span><span class="s1">'tableReference'</span><span class="p">][</span><span class="s1">'tableId'</span><span class="p">]</span> <span class="o">==</span> <span class="n">table_id</span><span class="p">:</span>
+ <span class="c1"># found the table, do update</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> exists, updating.'</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span>
+ <span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span>
+ <span class="n">tableId</span><span class="o">=</span><span class="n">table_id</span><span class="p">,</span>
+ <span class="n">body</span><span class="o">=</span><span class="n">table_resource</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="c1"># If there is a next page, we need to check the next page.</span>
+ <span class="k">if</span> <span class="s1">'nextPageToken'</span> <span class="ow">in</span> <span class="n">tables_list_resp</span><span class="p">:</span>
+
<TRUNCATED>
[32/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/datastore_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/datastore_hook.html b/_modules/airflow/contrib/hooks/datastore_hook.html
new file mode 100644
index 0000000..21363c3
--- /dev/null
+++ b/_modules/airflow/contrib/hooks/datastore_hook.html
@@ -0,0 +1,410 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.hooks.datastore_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.hooks.datastore_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.hooks.datastore_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">import</span> <span class="nn">json</span>
+<span class="kn">import</span> <span class="nn">time</span>
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
+
+
+<div class="viewcode-block" id="DatastoreHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook">[docs]</a><span class="k">class</span> <span class="nc">DatastoreHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with Google Cloud Datastore. This hook uses the Google Cloud Platform</span>
+<span class="sd"> connection.</span>
+
+<span class="sd"> This object is not threads safe. If you want to make multiple requests</span>
+<span class="sd"> simultaniously, you will need to create a hook per thread.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">datastore_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_datastore_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DatastoreHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">datastore_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">admin_connection</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">(</span><span class="s1">'v1beta1'</span><span class="p">)</span>
+
+<div class="viewcode-block" id="DatastoreHook.get_conn"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="s1">'v1'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a Google Cloud Storage service object.</span>
+<span class="sd"> """</span>
+ <span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'datastore'</span><span class="p">,</span> <span class="n">version</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.allocate_ids"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.allocate_ids">[docs]</a> <span class="k">def</span> <span class="nf">allocate_ids</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">partialKeys</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Allocate IDs for incomplete keys.</span>
+<span class="sd"> see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds</span>
+
+<span class="sd"> :param partialKeys: a list of partial keys</span>
+<span class="sd"> :return: a list of full keys.</span>
+<span class="sd"> """</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">allocateIds</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s1">'keys'</span><span class="p">:</span> <span class="n">partialKeys</span><span class="p">})</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span><span class="p">[</span><span class="s1">'keys'</span><span class="p">]</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.begin_transaction"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.begin_transaction">[docs]</a> <span class="k">def</span> <span class="nf">begin_transaction</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a new transaction handle</span>
+<span class="sd"> see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction</span>
+
+<span class="sd"> :return: a transaction handle</span>
+<span class="sd"> """</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">beginTransaction</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="p">{})</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span><span class="p">[</span><span class="s1">'transaction'</span><span class="p">]</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.commit"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.commit">[docs]</a> <span class="k">def</span> <span class="nf">commit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Commit a transaction, optionally creating, deleting or modifying some entities.</span>
+<span class="sd"> see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit</span>
+
+<span class="sd"> :param body: the body of the commit request</span>
+<span class="sd"> :return: the response body of the commit request</span>
+<span class="sd"> """</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">commit</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.lookup"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.lookup">[docs]</a> <span class="k">def</span> <span class="nf">lookup</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">keys</span><span class="p">,</span> <span class="n">read_consistency</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">transaction</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lookup some entities by key</span>
+<span class="sd"> see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/lookup</span>
+<span class="sd"> :param keys: the keys to lookup</span>
+<span class="sd"> :param read_consistency: the read consistency to use. default, strong or eventual.</span>
+<span class="sd"> Cannot be used with a transaction.</span>
+<span class="sd"> :param transaction: the transaction to use, if any.</span>
+<span class="sd"> :return: the response body of the lookup request.</span>
+<span class="sd"> """</span>
+ <span class="n">body</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'keys'</span><span class="p">:</span> <span class="n">keys</span><span class="p">}</span>
+ <span class="k">if</span> <span class="n">read_consistency</span><span class="p">:</span>
+ <span class="n">body</span><span class="p">[</span><span class="s1">'readConsistency'</span><span class="p">]</span> <span class="o">=</span> <span class="n">read_consistency</span>
+ <span class="k">if</span> <span class="n">transaction</span><span class="p">:</span>
+ <span class="n">body</span><span class="p">[</span><span class="s1">'transaction'</span><span class="p">]</span> <span class="o">=</span> <span class="n">transaction</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">lookup</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.rollback"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.rollback">[docs]</a> <span class="k">def</span> <span class="nf">rollback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">transaction</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Roll back a transaction</span>
+<span class="sd"> see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback</span>
+<span class="sd"> :param transaction: the transaction to roll back</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">rollback</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="p">{</span><span class="s1">'transaction'</span><span class="p">:</span> <span class="n">transaction</span><span class="p">})</span>\
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.run_query"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.run_query">[docs]</a> <span class="k">def</span> <span class="nf">run_query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">body</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Run a query for entities.</span>
+<span class="sd"> see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery</span>
+<span class="sd"> :param body: the body of the query request</span>
+<span class="sd"> :return: the batch of query results.</span>
+<span class="sd"> """</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">runQuery</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span><span class="p">[</span><span class="s1">'batch'</span><span class="p">]</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.get_operation"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.get_operation">[docs]</a> <span class="k">def</span> <span class="nf">get_operation</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Gets the latest state of a long-running operation</span>
+
+<span class="sd"> :param name: the name of the operation resource</span>
+<span class="sd"> """</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">operations</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.delete_operation"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.delete_operation">[docs]</a> <span class="k">def</span> <span class="nf">delete_operation</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Deletes the long-running operation</span>
+
+<span class="sd"> :param name: the name of the operation resource</span>
+<span class="sd"> """</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">operations</span><span class="p">()</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.poll_operation_until_done"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.poll_operation_until_done">[docs]</a> <span class="k">def</span> <span class="nf">poll_operation_until_done</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">polling_interval_in_seconds</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Poll backup operation state until it's completed</span>
+<span class="sd"> """</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_operation</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
+ <span class="n">state</span> <span class="o">=</span> <span class="n">result</span><span class="p">[</span><span class="s1">'metadata'</span><span class="p">][</span><span class="s1">'common'</span><span class="p">][</span><span class="s1">'state'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">state</span> <span class="o">==</span> <span class="s1">'PROCESSING'</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Operation is processing. Re-polling state in </span><span class="si">{}</span><span class="s1"> seconds'</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">polling_interval_in_seconds</span><span class="p">))</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">polling_interval_in_seconds</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">result</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.export_to_storage_bucket"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.export_to_storage_bucket">[docs]</a> <span class="k">def</span> <span class="nf">export_to_storage_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="n">namespace</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">entity_filter</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">labels</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Export entities from Cloud Datastore to Cloud Storage for backup</span>
+<span class="sd"> """</span>
+ <span class="n">output_uri_prefix</span> <span class="o">=</span> <span class="s1">'gs://'</span> <span class="o">+</span> <span class="p">(</span><span class="s1">'/'</span><span class="p">)</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="p">[</span><span class="n">bucket</span><span class="p">,</span> <span class="n">namespace</span><span class="p">]))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">entity_filter</span><span class="p">:</span>
+ <span class="n">entity_filter</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">labels</span><span class="p">:</span>
+ <span class="n">labels</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="n">body</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'outputUrlPrefix'</span><span class="p">:</span> <span class="n">output_uri_prefix</span><span class="p">,</span>
+ <span class="s1">'entityFilter'</span><span class="p">:</span> <span class="n">entity_filter</span><span class="p">,</span>
+ <span class="s1">'labels'</span><span class="p">:</span> <span class="n">labels</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">admin_connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">export</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span></div>
+
+<div class="viewcode-block" id="DatastoreHook.import_from_storage_bucket"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.import_from_storage_bucket">[docs]</a> <span class="k">def</span> <span class="nf">import_from_storage_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="n">file</span><span class="p">,</span> <span class="n">namespace</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">entity_filter</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">labels</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Import a backup from Cloud Storage to Cloud Datastore</span>
+<span class="sd"> """</span>
+ <span class="n">input_url</span> <span class="o">=</span> <span class="s1">'gs://'</span> <span class="o">+</span> <span class="p">(</span><span class="s1">'/'</span><span class="p">)</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="p">[</span><span class="n">bucket</span><span class="p">,</span> <span class="n">namespace</span><span class="p">,</span> <span class="n">file</span><span class="p">]))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">entity_filter</span><span class="p">:</span>
+ <span class="n">entity_filter</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">labels</span><span class="p">:</span>
+ <span class="n">labels</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="n">body</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'inputUrl'</span><span class="p">:</span> <span class="n">input_url</span><span class="p">,</span>
+ <span class="s1">'entityFilter'</span><span class="p">:</span> <span class="n">entity_filter</span><span class="p">,</span>
+ <span class="s1">'labels'</span><span class="p">:</span> <span class="n">labels</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="n">resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">admin_connection</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">import_</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">body</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">resp</span></div></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/emr_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/emr_hook.html b/_modules/airflow/contrib/hooks/emr_hook.html
new file mode 100644
index 0000000..cc158a2
--- /dev/null
+++ b/_modules/airflow/contrib/hooks/emr_hook.html
@@ -0,0 +1,295 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.hooks.emr_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.hooks.emr_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.hooks.emr_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.aws_hook</span> <span class="k">import</span> <span class="n">AwsHook</span>
+
+
+<div class="viewcode-block" id="EmrHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.emr_hook.EmrHook">[docs]</a><span class="k">class</span> <span class="nc">EmrHook</span><span class="p">(</span><span class="n">AwsHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with AWS EMR. emr_conn_id is only neccessary for using the create_job_flow method.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">emr_conn_id</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">emr_conn_id</span> <span class="o">=</span> <span class="n">emr_conn_id</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">EmrHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_client_type</span><span class="p">(</span><span class="s1">'emr'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>
+
+ <span class="k">def</span> <span class="nf">create_job_flow</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">job_flow_overrides</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Creates a job flow using the config from the EMR connection.</span>
+<span class="sd"> Keys of the json extra hash may have the arguments of the boto3 run_job_flow method.</span>
+<span class="sd"> Overrides for this config may be passed as the job_flow_overrides.</span>
+<span class="sd"> """</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">emr_conn_id</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'emr_conn_id must be present to use create_job_flow'</span><span class="p">)</span>
+
+ <span class="n">emr_conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">emr_conn_id</span><span class="p">)</span>
+
+ <span class="n">config</span> <span class="o">=</span> <span class="n">emr_conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
+ <span class="n">config</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">job_flow_overrides</span><span class="p">)</span>
+
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">run_job_flow</span><span class="p">(</span>
+ <span class="n">Name</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Name'</span><span class="p">),</span>
+ <span class="n">LogUri</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'LogUri'</span><span class="p">),</span>
+ <span class="n">ReleaseLabel</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'ReleaseLabel'</span><span class="p">),</span>
+ <span class="n">Instances</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Instances'</span><span class="p">),</span>
+ <span class="n">Steps</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Steps'</span><span class="p">,</span> <span class="p">[]),</span>
+ <span class="n">BootstrapActions</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'BootstrapActions'</span><span class="p">,</span> <span class="p">[]),</span>
+ <span class="n">Applications</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Applications'</span><span class="p">),</span>
+ <span class="n">Configurations</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Configurations'</span><span class="p">,</span> <span class="p">[]),</span>
+ <span class="n">VisibleToAllUsers</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'VisibleToAllUsers'</span><span class="p">),</span>
+ <span class="n">JobFlowRole</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'JobFlowRole'</span><span class="p">),</span>
+ <span class="n">ServiceRole</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'ServiceRole'</span><span class="p">),</span>
+ <span class="n">Tags</span><span class="o">=</span><span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Tags'</span><span class="p">)</span>
+ <span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">response</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/gcp_dataflow_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/gcp_dataflow_hook.html b/_modules/airflow/contrib/hooks/gcp_dataflow_hook.html
new file mode 100644
index 0000000..82e3ccd
--- /dev/null
+++ b/_modules/airflow/contrib/hooks/gcp_dataflow_hook.html
@@ -0,0 +1,395 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.hooks.gcp_dataflow_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.hooks.gcp_dataflow_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.hooks.gcp_dataflow_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">import</span> <span class="nn">select</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">import</span> <span class="nn">time</span>
+<span class="kn">import</span> <span class="nn">uuid</span>
+
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
+
+<span class="k">class</span> <span class="nc">_DataflowJob</span><span class="p">(</span><span class="n">LoggingMixin</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">,</span> <span class="n">project_number</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_dataflow</span> <span class="o">=</span> <span class="n">dataflow</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_project_number</span> <span class="o">=</span> <span class="n">project_number</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job_name</span> <span class="o">=</span> <span class="n">name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_job</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">_get_job_id_from_name</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">jobs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_dataflow</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">jobs</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_number</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">job</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">[</span><span class="s1">'jobs'</span><span class="p">]:</span>
+ <span class="k">if</span> <span class="n">job</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job_name</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span> <span class="o">=</span> <span class="n">job</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]</span>
+ <span class="k">return</span> <span class="n">job</span>
+ <span class="k">return</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">_get_job</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">job</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_job_id_from_name</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">job</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_dataflow</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">jobs</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_project_number</span><span class="p">,</span>
+ <span class="n">jobId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">if</span> <span class="s1">'currentState'</span> <span class="ow">in</span> <span class="n">job</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Google Cloud DataFlow job </span><span class="si">%s</span><span class="s1"> is </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">job</span><span class="p">[</span><span class="s1">'name'</span><span class="p">],</span> <span class="n">job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Google Cloud DataFlow with job_id </span><span class="si">%s</span><span class="s1"> has name </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job_id</span><span class="p">,</span> <span class="n">job</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span>
+ <span class="p">)</span>
+ <span class="k">return</span> <span class="n">job</span>
+
+ <span class="k">def</span> <span class="nf">wait_for_done</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="k">if</span> <span class="s1">'currentState'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">:</span>
+ <span class="k">if</span> <span class="s1">'JOB_STATE_DONE'</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]:</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">elif</span> <span class="s1">'JOB_STATE_FAILED'</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s2">"Google Cloud Dataflow job </span><span class="si">{}</span><span class="s2"> has failed."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]))</span>
+ <span class="k">elif</span> <span class="s1">'JOB_STATE_CANCELLED'</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s2">"Google Cloud Dataflow job </span><span class="si">{}</span><span class="s2"> was cancelled."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]))</span>
+ <span class="k">elif</span> <span class="s1">'JOB_STATE_RUNNING'</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]:</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="s1">'JOB_STATE_PENDING'</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]:</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s2">"Google Cloud Dataflow job </span><span class="si">{}</span><span class="s2"> was unknown state: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'name'</span><span class="p">],</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span><span class="p">[</span><span class="s1">'currentState'</span><span class="p">]))</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">_job</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_job</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_job</span>
+
+
+<span class="k">class</span> <span class="nc">_Dataflow</span><span class="p">(</span><span class="n">LoggingMixin</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cmd</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
+ <span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_line</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fd</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">fd</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">fileno</span><span class="p">():</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">line</span>
+ <span class="k">if</span> <span class="n">fd</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">fileno</span><span class="p">():</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">line</span>
+
+ <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">_extract_job</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">line</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"Submitted job: "</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">line</span><span class="p">[</span><span class="mi">15</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+
+ <span class="k">def</span> <span class="nf">wait_for_done</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">reads</span> <span class="o">=</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">fileno</span><span class="p">(),</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">fileno</span><span class="p">()]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Start waiting for DataFlow process to complete."</span><span class="p">)</span>
+ <span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">poll</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">ret</span> <span class="o">=</span> <span class="n">select</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">reads</span><span class="p">,</span> <span class="p">[],</span> <span class="p">[],</span> <span class="mi">5</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">ret</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">fd</span> <span class="ow">in</span> <span class="n">ret</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_line</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="n">line</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Waiting for DataFlow process to complete."</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">returncode</span> <span class="ow">is</span> <span class="ow">not</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s2">"DataFlow failed with return code </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_proc</span><span class="o">.</span><span class="n">returncode</span><span class="p">))</span>
+
+
+<div class="viewcode-block" id="DataFlowHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook">[docs]</a><span class="k">class</span> <span class="nc">DataFlowHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataFlowHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+
+<div class="viewcode-block" id="DataFlowHook.get_conn"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a Google Cloud Storage service object.</span>
+<span class="sd"> """</span>
+ <span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'dataflow'</span><span class="p">,</span> <span class="s1">'v1b3'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">_start_dataflow</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">command_prefix</span><span class="p">):</span>
+ <span class="n">cmd</span> <span class="o">=</span> <span class="n">command_prefix</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">_build_cmd</span><span class="p">(</span><span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">)</span>
+ <span class="n">_Dataflow</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span><span class="o">.</span><span class="n">wait_for_done</span><span class="p">()</span>
+ <span class="n">_DataflowJob</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">(),</span> <span class="n">variables</span><span class="p">[</span><span class="s1">'project'</span><span class="p">],</span> <span class="n">name</span><span class="p">)</span><span class="o">.</span><span class="n">wait_for_done</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">start_java_dataflow</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">):</span>
+ <span class="n">name</span> <span class="o">=</span> <span class="n">task_id</span> <span class="o">+</span> <span class="s2">"-"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid1</span><span class="p">())[:</span><span class="mi">8</span><span class="p">]</span>
+ <span class="n">variables</span><span class="p">[</span><span class="s1">'jobName'</span><span class="p">]</span> <span class="o">=</span> <span class="n">name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_start_dataflow</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="p">[</span><span class="s2">"java"</span><span class="p">,</span> <span class="s2">"-jar"</span><span class="p">])</span>
+
+ <span class="k">def</span> <span class="nf">start_python_dataflow</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">,</span> <span class="n">py_options</span><span class="p">):</span>
+ <span class="n">name</span> <span class="o">=</span> <span class="n">task_id</span> <span class="o">+</span> <span class="s2">"-"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid1</span><span class="p">())[:</span><span class="mi">8</span><span class="p">]</span>
+ <span class="n">variables</span><span class="p">[</span><span class="s2">"job_name"</span><span class="p">]</span> <span class="o">=</span> <span class="n">name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_start_dataflow</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="p">[</span><span class="s2">"python"</span><span class="p">]</span> <span class="o">+</span> <span class="n">py_options</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_build_cmd</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_id</span><span class="p">,</span> <span class="n">variables</span><span class="p">,</span> <span class="n">dataflow</span><span class="p">):</span>
+ <span class="n">command</span> <span class="o">=</span> <span class="p">[</span><span class="n">dataflow</span><span class="p">,</span> <span class="s2">"--runner=DataflowRunner"</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">variables</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">attr</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">variables</span><span class="o">.</span><span class="n">iteritems</span><span class="p">():</span>
+ <span class="n">command</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s2">"--"</span> <span class="o">+</span> <span class="n">attr</span> <span class="o">+</span> <span class="s2">"="</span> <span class="o">+</span> <span class="n">value</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">command</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[07/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/code.html
----------------------------------------------------------------------
diff --git a/code.html b/code.html
index e37b111..65e03ee 100644
--- a/code.html
+++ b/code.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -207,82 +212,77 @@ method at a specified <code class="docutils literal"><span class="pre">poke_inte
<h3>BaseOperator<a class="headerlink" href="#baseoperator" title="Permalink to this headline">¶</a></h3>
<p>All operators are derived from <code class="docutils literal"><span class="pre">BaseOperator</span></code> and acquire much
functionality through inheritance. Since this is the core of the engine,
-it’s worth taking the time to understand the parameters of <code class="docutils literal"><span class="pre">BaseOperator</span></code>
+it’s worth taking the time to understand the parameters of <code class="docutils literal"><span class="pre">BaseOperator</span></code>
to understand the primitive features that can be leveraged in your
DAGs.</p>
<dl class="class">
<dt id="airflow.models.BaseOperator">
-<em class="property">class </em><code class="descclassname">airflow.models.</code><code class="descname">BaseOperator</code><span class="sig-paren">(</span><em>task_id</em>, <em>owner='Airflow'</em>, <em>email=None</em>, <em>email_on_retry=True</em>, <em>email_on_failure=True</em>, <em>retries=0</em>, <em>retry_delay=datetime.timedelta(0</em>, <em>300)</em>, <em>retry_exponential_backoff=False</em>, <em>max_retry_delay=None</em>, <em>start_date=None</em>, <em>end_date=None</em>, <em>schedule_interval=None</em>, <em>depends_on_past=False</em>, <em>wait_for_downstream=False</em>, <em>dag=None</em>, <em>params=None</em>, <em>default_args=None</em>, <em>adhoc=False</em>, <em>priority_weight=1</em>, <em>queue='default'</em>, <em>pool=None</em>, <em>sla=None</em>, <em>execution_timeout=None</em>, <em>on_failure_callback=None</em>, <em>on_success_callback=None</em>, <em>on_retry_callback=None</em>, <em>trigger_rule=u'all_success'</em>, <em>resources=None</em>, <em>run_as_user=None</em>, <e
m>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/models.html#BaseOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.models.BaseOperator" title="Permalink to this definition">¶</a></dt>
+<em class="property">class </em><code class="descclassname">airflow.models.</code><code class="descname">BaseOperator</code><span class="sig-paren">(</span><em>task_id</em>, <em>owner='Airflow'</em>, <em>email=None</em>, <em>email_on_retry=True</em>, <em>email_on_failure=True</em>, <em>retries=0</em>, <em>retry_delay=datetime.timedelta(0</em>, <em>300)</em>, <em>retry_exponential_backoff=False</em>, <em>max_retry_delay=None</em>, <em>start_date=None</em>, <em>end_date=None</em>, <em>schedule_interval=None</em>, <em>depends_on_past=False</em>, <em>wait_for_downstream=False</em>, <em>dag=None</em>, <em>params=None</em>, <em>default_args=None</em>, <em>adhoc=False</em>, <em>priority_weight=1</em>, <em>queue='default'</em>, <em>pool=None</em>, <em>sla=None</em>, <em>execution_timeout=None</em>, <em>on_failure_callback=None</em>, <em>on_success_callback=None</em>, <em>on_retry_callback=None</em>, <em>trigger_rule=u'all_success'</em>, <em>resources=None</em>, <em>run_as_user=None</em>, <e
m>task_concurrency=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/models.html#BaseOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.models.BaseOperator" title="Permalink to this definition">¶</a></dt>
<dd><p>Abstract base class for all operators. Since operators create objects that
become node in the dag, BaseOperator contains many recursive methods for
dag crawling behavior. To derive this class, you are expected to override
-the constructor as well as the ‘execute’ method.</p>
-<p>Operators derived from this task should perform or trigger certain tasks
+the constructor as well as the ‘execute’ method.</p>
+<p>Operators derived from this class should perform or trigger certain tasks
synchronously (wait for completion). Example of operators could be an
operator the runs a Pig job (PigOperator), a sensor operator that
waits for a partition to land in Hive (HiveSensorOperator), or one that
moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these
operators (tasks) target specific operations, running specific scripts,
functions or data transfers.</p>
-<p>This class is abstract and shouldn’t be instantiated. Instantiating a
+<p>This class is abstract and shouldn’t be instantiated. Instantiating a
class derived from this one results in the creation of a task object,
which ultimately becomes a node in DAG objects. Task dependencies should
be set by using the set_upstream and/or set_downstream methods.</p>
-<p>Note that this class is derived from SQLAlchemy’s Base class, which
-allows us to push metadata regarding tasks to the database. Deriving this
-classes needs to implement the polymorphic specificities documented in
-SQLAlchemy. This should become clear while reading the code for other
-operators.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>task_id</strong> (<em>string</em>) – a unique, meaningful id for the task</li>
-<li><strong>owner</strong> (<em>string</em>) – the owner of the task, using the unix username is recommended</li>
-<li><strong>retries</strong> (<em>int</em>) – the number of retries that should be performed before
+<li><strong>task_id</strong> (<em>string</em>) – a unique, meaningful id for the task</li>
+<li><strong>owner</strong> (<em>string</em>) – the owner of the task, using the unix username is recommended</li>
+<li><strong>retries</strong> (<em>int</em>) – the number of retries that should be performed before
failing the task</li>
-<li><strong>retry_delay</strong> (<em>timedelta</em>) – delay between retries</li>
-<li><strong>retry_exponential_backoff</strong> (<em>bool</em>) – allow progressive longer waits between
+<li><strong>retry_delay</strong> (<em>timedelta</em>) – delay between retries</li>
+<li><strong>retry_exponential_backoff</strong> (<em>bool</em>) – allow progressive longer waits between
retries by using exponential backoff algorithm on retry delay (delay
will be converted into seconds)</li>
-<li><strong>max_retry_delay</strong> (<em>timedelta</em>) – maximum delay interval between retries</li>
-<li><strong>start_date</strong> (<em>datetime</em>) – The <code class="docutils literal"><span class="pre">start_date</span></code> for the task, determines
+<li><strong>max_retry_delay</strong> (<em>timedelta</em>) – maximum delay interval between retries</li>
+<li><strong>start_date</strong> (<em>datetime</em>) – The <code class="docutils literal"><span class="pre">start_date</span></code> for the task, determines
the <code class="docutils literal"><span class="pre">execution_date</span></code> for the first task instance. The best practice
is to have the start_date rounded
-to your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
+to your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
some day at 00:00:00, hourly jobs have their start_date at 00:00
of a specific hour. Note that Airflow simply looks at the latest
<code class="docutils literal"><span class="pre">execution_date</span></code> and adds the <code class="docutils literal"><span class="pre">schedule_interval</span></code> to determine
the next <code class="docutils literal"><span class="pre">execution_date</span></code>. It is also very important
-to note that different tasks’ dependencies
+to note that different tasks’ dependencies
need to line up in time. If task A depends on task B and their
-start_date are offset in a way that their execution_date don’t line
-up, A’s dependencies will never be met. If you are looking to delay
+start_date are offset in a way that their execution_date don’t line
+up, A’s dependencies will never be met. If you are looking to delay
a task, for example running a daily task at 2AM, look into the
<code class="docutils literal"><span class="pre">TimeSensor</span></code> and <code class="docutils literal"><span class="pre">TimeDeltaSensor</span></code>. We advise against using
dynamic <code class="docutils literal"><span class="pre">start_date</span></code> and recommend using fixed ones. Read the
FAQ entry about start_date for more information.</li>
-<li><strong>end_date</strong> (<em>datetime</em>) – if specified, the scheduler won’t go beyond this date</li>
-<li><strong>depends_on_past</strong> (<em>bool</em>) – when set to true, task instances will run
-sequentially while relying on the previous task’s schedule to
+<li><strong>end_date</strong> (<em>datetime</em>) – if specified, the scheduler won’t go beyond this date</li>
+<li><strong>depends_on_past</strong> (<em>bool</em>) – when set to true, task instances will run
+sequentially while relying on the previous task’s schedule to
succeed. The task instance for the start_date is allowed to run.</li>
-<li><strong>wait_for_downstream</strong> (<em>bool</em>) – when set to true, an instance of task
+<li><strong>wait_for_downstream</strong> (<em>bool</em>) – when set to true, an instance of task
X will wait for tasks immediately downstream of the previous instance
of task X to finish successfully before it runs. This is useful if the
different instances of a task X alter the same asset, and this asset
is used by tasks downstream of task X. Note that depends_on_past
is forced to True wherever wait_for_downstream is used.</li>
-<li><strong>queue</strong> (<em>str</em>) – which queue to target when running this job. Not
+<li><strong>queue</strong> (<em>str</em>) – which queue to target when running this job. Not
all executors implement queue management, the CeleryExecutor
does support targeting specific queues.</li>
-<li><strong>dag</strong> (<a class="reference internal" href="#airflow.models.DAG" title="airflow.models.DAG"><em>DAG</em></a>) – a reference to the dag the task is attached to (if any)</li>
-<li><strong>priority_weight</strong> (<em>int</em>) – priority weight of this task against other task.
+<li><strong>dag</strong> (<a class="reference internal" href="#airflow.models.DAG" title="airflow.models.DAG"><em>DAG</em></a>) – a reference to the dag the task is attached to (if any)</li>
+<li><strong>priority_weight</strong> (<em>int</em>) – priority weight of this task against other task.
This allows the executor to trigger higher priority tasks before
others when things get backed up.</li>
-<li><strong>pool</strong> (<em>str</em>) – the slot pool this task should run in, slot pools are a
+<li><strong>pool</strong> (<em>str</em>) – the slot pool this task should run in, slot pools are a
way to limit concurrency for certain tasks</li>
-<li><strong>sla</strong> (<em>datetime.timedelta</em>) – time by which the job is expected to succeed. Note that
+<li><strong>sla</strong> (<em>datetime.timedelta</em>) – time by which the job is expected to succeed. Note that
this represents the <code class="docutils literal"><span class="pre">timedelta</span></code> after the period is closed. For
example if you set an SLA of 1 hour, the scheduler would send dan email
soon after 1:00AM on the <code class="docutils literal"><span class="pre">2016-01-02</span></code> if the <code class="docutils literal"><span class="pre">2016-01-01</span></code> instance
@@ -293,27 +293,29 @@ emails for sla misses. SLA misses are also recorded in the database
for future reference. All tasks that share the same SLA time
get bundled in a single email, sent soon after that time. SLA
notification are sent once and only once for each task instance.</li>
-<li><strong>execution_timeout</strong> (<em>datetime.timedelta</em>) – max time allowed for the execution of
+<li><strong>execution_timeout</strong> (<em>datetime.timedelta</em>) – max time allowed for the execution of
this task instance, if it goes beyond it will raise and fail.</li>
-<li><strong>on_failure_callback</strong> (<em>callable</em>) – a function to be called when a task instance
+<li><strong>on_failure_callback</strong> (<em>callable</em>) – a function to be called when a task instance
of this task fails. a context dictionary is passed as a single
parameter to this function. Context contains references to related
objects to the task instance and is documented under the macros
section of the API.</li>
-<li><strong>on_retry_callback</strong> – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> excepts
+<li><strong>on_retry_callback</strong> – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> except
that it is executed when retries occur.</li>
-<li><strong>on_success_callback</strong> (<em>callable</em>) – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> excepts
+<li><strong>on_success_callback</strong> (<em>callable</em>) – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> except
that it is executed when the task succeeds.</li>
-<li><strong>trigger_rule</strong> (<em>str</em>) – defines the rule by which dependencies are applied
+<li><strong>trigger_rule</strong> (<em>str</em>) – defines the rule by which dependencies are applied
for the task to get triggered. Options are:
<code class="docutils literal"><span class="pre">{</span> <span class="pre">all_success</span> <span class="pre">|</span> <span class="pre">all_failed</span> <span class="pre">|</span> <span class="pre">all_done</span> <span class="pre">|</span> <span class="pre">one_success</span> <span class="pre">|</span>
<span class="pre">one_failed</span> <span class="pre">|</span> <span class="pre">dummy}</span></code>
default is <code class="docutils literal"><span class="pre">all_success</span></code>. Options can be set as string or
using the constants defined in the static class
<code class="docutils literal"><span class="pre">airflow.utils.TriggerRule</span></code></li>
-<li><strong>resources</strong> (<em>dict</em>) – A map of resource parameter names (the argument names of the
+<li><strong>resources</strong> (<em>dict</em>) – A map of resource parameter names (the argument names of the
Resources constructor) to their values.</li>
-<li><strong>run_as_user</strong> (<em>str</em>) – unix username to impersonate while running the task</li>
+<li><strong>run_as_user</strong> (<em>str</em>) – unix username to impersonate while running the task</li>
+<li><strong>task_concurrency</strong> (<em>int</em>) – When set, a task will be able to limit the concurrent
+runs across execution_dates</li>
</ul>
</td>
</tr>
@@ -340,10 +342,10 @@ attributes.</p>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>soft_fail</strong> (<em>bool</em>) – Set to true to mark the task as SKIPPED on failure</li>
-<li><strong>poke_interval</strong> (<em>int</em>) – Time in seconds that the job should wait in
+<li><strong>soft_fail</strong> (<em>bool</em>) – Set to true to mark the task as SKIPPED on failure</li>
+<li><strong>poke_interval</strong> (<em>int</em>) – Time in seconds that the job should wait in
between each tries</li>
-<li><strong>timeout</strong> (<em>int</em>) – Time, in seconds before the task times out and fails.</li>
+<li><strong>timeout</strong> (<em>int</em>) – Time, in seconds before the task times out and fails.</li>
</ul>
</td>
</tr>
@@ -374,11 +376,11 @@ required to support attribute-based usage:</p>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>bash_command</strong> (<em>string</em>) – The command, set of commands or reference to a
-bash script (must be ‘.sh’) to be executed.</li>
-<li><strong>xcom_push</strong> (<em>bool</em>) – If xcom_push is True, the last line written to stdout
+<li><strong>bash_command</strong> (<em>string</em>) – The command, set of commands or reference to a
+bash script (must be ‘.sh’) to be executed.</li>
+<li><strong>xcom_push</strong> (<em>bool</em>) – If xcom_push is True, the last line written to stdout
will also be pushed to an XCom when the bash command completes.</li>
-<li><strong>env</strong> (<em>dict</em>) – If env is not None, it must be a mapping that defines the
+<li><strong>env</strong> (<em>dict</em>) – If env is not None, it must be a mapping that defines the
environment variables for the new process; these are used instead
of inheriting the current process environment, which is the default
behavior. (templated)</li>
@@ -399,15 +401,15 @@ which will be cleaned afterwards</p>
<dl class="class">
<dt id="airflow.operators.BranchPythonOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">BranchPythonOperator</code><span class="sig-paren">(</span><em>python_callable</em>, <em>op_args=None</em>, <em>op_kwargs=None</em>, <em>provide_context=False</em>, <em>templates_dict=None</em>, <em>templates_exts=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/python_operator.html#BranchPythonOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.BranchPythonOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">python_operator.PythonOperator</span></code></p>
-<p>Allows a workflow to “branch” or follow a single path following the
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">python_operator.PythonOperator</span></code>, <code class="xref py py-class docutils literal"><span class="pre">airflow.models.SkipMixin</span></code></p>
+<p>Allows a workflow to “branch” or follow a single path following the
execution of this task.</p>
<p>It derives the PythonOperator and expects a Python function that returns
the task_id to follow. The task_id returned should point to a task
-directly downstream from {self}. All other “branches” or
+directly downstream from {self}. All other “branches” or
directly downstream tasks are marked with a state of <code class="docutils literal"><span class="pre">skipped</span></code> so that
-these paths can’t move forward. The <code class="docutils literal"><span class="pre">skipped</span></code> states are propageted
-downstream to allow for the DAG state to fill up and the DAG run’s state
+these paths can’t move forward. The <code class="docutils literal"><span class="pre">skipped</span></code> states are propageted
+downstream to allow for the DAG state to fill up and the DAG run’s state
to be inferred.</p>
<p>Note that using tasks with <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> downstream from
<code class="docutils literal"><span class="pre">BranchPythonOperator</span></code> is logically unsound as <code class="docutils literal"><span class="pre">skipped</span></code> status
@@ -426,8 +428,8 @@ will invariably lead to block tasks that depend on their past successes.
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>trigger_dag_id</strong> (<em>str</em>) – the dag_id to trigger</li>
-<li><strong>python_callable</strong> (<em>python callable</em>) – a reference to a python function that will be
+<li><strong>trigger_dag_id</strong> (<em>str</em>) – the dag_id to trigger</li>
+<li><strong>python_callable</strong> (<em>python callable</em>) – a reference to a python function that will be
called while passing it the <code class="docutils literal"><span class="pre">context</span></code> object and a placeholder
object <code class="docutils literal"><span class="pre">obj</span></code> for your callable to fill and return if you want
a DagRun created. This <code class="docutils literal"><span class="pre">obj</span></code> object contains a <code class="docutils literal"><span class="pre">run_id</span></code> and
@@ -461,13 +463,13 @@ DAG.</p>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>to</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em><em></em>) – list of emails to send the email to</li>
-<li><strong>subject</strong> (<em>string</em>) – subject line for the email (templated)</li>
-<li><strong>html_content</strong> (<em>string</em>) – content of the email (templated), html markup
+<li><strong>to</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of emails to send the email to</li>
+<li><strong>subject</strong> (<em>string</em>) – subject line for the email (templated)</li>
+<li><strong>html_content</strong> (<em>string</em>) – content of the email (templated), html markup
is allowed</li>
-<li><strong>files</strong> (<em>list</em>) – file names to attach in email</li>
-<li><strong>cc</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em><em></em>) – list of recipients to be added in CC field</li>
-<li><strong>bcc</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em><em></em>) – list of recipients to be added in BCC field</li>
+<li><strong>files</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – file names to attach in email</li>
+<li><strong>cc</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of recipients to be added in CC field</li>
+<li><strong>bcc</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of recipients to be added in BCC field</li>
</ul>
</td>
</tr>
@@ -485,18 +487,18 @@ is allowed</li>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>external_dag_id</strong> (<em>string</em>) – The dag_id that contains the task you want to
+<li><strong>external_dag_id</strong> (<em>string</em>) – The dag_id that contains the task you want to
wait for</li>
-<li><strong>external_task_id</strong> (<em>string</em>) – The task_id that contains the task you want to
+<li><strong>external_task_id</strong> (<em>string</em>) – The task_id that contains the task you want to
wait for</li>
-<li><strong>allowed_states</strong> (<em>list</em>) – list of allowed states, default is <code class="docutils literal"><span class="pre">['success']</span></code></li>
-<li><strong>execution_delta</strong> (<em>datetime.timedelta</em>) – time difference with the previous execution to
+<li><strong>allowed_states</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – list of allowed states, default is <code class="docutils literal"><span class="pre">['success']</span></code></li>
+<li><strong>execution_delta</strong> (<em>datetime.timedelta</em>) – time difference with the previous execution to
look at, the default is the same execution_date as the current task.
For yesterday, use [positive!] datetime.timedelta(days=1). Either
execution_delta or execution_date_fn can be passed to
ExternalTaskSensor, but not both.</li>
-<li><strong>execution_date_fn</strong> (<em>callable</em>) – function that receives the current execution date
-and returns the desired execution date to query. Either execution_delta
+<li><strong>execution_date_fn</strong> (<em>callable</em>) – function that receives the current execution date
+and returns the desired execution dates to query. Either execution_delta
or execution_date_fn can be passed to ExternalTaskSensor, but not both.</li>
</ul>
</td>
@@ -513,17 +515,17 @@ or execution_date_fn can be passed to ExternalTaskSensor, but not both.</li>
provide the required methods in their respective hooks. The source hook
needs to expose a <cite>get_records</cite> method, and the destination a
<cite>insert_rows</cite> method.</p>
-<p>This is mean to be used on small-ish datasets that fit in memory.</p>
+<p>This is meant to be used on small-ish datasets that fit in memory.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the source database</li>
-<li><strong>destination_table</strong> (<em>str</em>) – target table</li>
-<li><strong>source_conn_id</strong> (<em>str</em>) – source connection</li>
-<li><strong>destination_conn_id</strong> (<em>str</em>) – source connection</li>
-<li><strong>preoperator</strong> (<em>str</em><em> or </em><em>list of str</em>) – sql statement or list of statements to be
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the source database</li>
+<li><strong>destination_table</strong> (<em>str</em>) – target table</li>
+<li><strong>source_conn_id</strong> (<em>str</em>) – source connection</li>
+<li><strong>destination_conn_id</strong> (<em>str</em>) – source connection</li>
+<li><strong>preoperator</strong> (<em>str</em><em> or </em><em>list of str</em>) – sql statement or list of statements to be
executed prior to loading the data</li>
</ul>
</td>
@@ -540,44 +542,99 @@ executed prior to loading the data</li>
<dl class="staticmethod">
<dt id="airflow.operators.HdfsSensor.filter_for_filesize">
<em class="property">static </em><code class="descname">filter_for_filesize</code><span class="sig-paren">(</span><em>result</em>, <em>size=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HdfsSensor.filter_for_filesize"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HdfsSensor.filter_for_filesize" title="Permalink to this definition">¶</a></dt>
-<dd><p>Will test the filepath result and test if its size is at least self.filesize
-:param result: a list of dicts returned by Snakebite ls
-:param size: the file size in MB a file should be at least to trigger True
-:return: (bool) depending on the matching criteria</p>
+<dd><p>Will test the filepath result and test if its size is at least self.filesize</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
+<li><strong>result</strong> – a list of dicts returned by Snakebite ls</li>
+<li><strong>size</strong> – the file size in MB a file should be at least to trigger True</li>
+</ul>
+</td>
+</tr>
+<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">(bool) depending on the matching criteria</p>
+</td>
+</tr>
+</tbody>
+</table>
</dd></dl>
<dl class="staticmethod">
<dt id="airflow.operators.HdfsSensor.filter_for_ignored_ext">
<em class="property">static </em><code class="descname">filter_for_ignored_ext</code><span class="sig-paren">(</span><em>result</em>, <em>ignored_ext</em>, <em>ignore_copying</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HdfsSensor.filter_for_ignored_ext"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HdfsSensor.filter_for_ignored_ext" title="Permalink to this definition">¶</a></dt>
-<dd><p>Will filter if instructed to do so the result to remove matching criteria
-:param result: (list) of dicts returned by Snakebite ls
-:param ignored_ext: (list) of ignored extentions
-:param ignore_copying: (bool) shall we ignore ?
-:return:</p>
+<dd><p>Will filter if instructed to do so the result to remove matching criteria</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
+<li><strong>result</strong> – (list) of dicts returned by Snakebite ls</li>
+<li><strong>ignored_ext</strong> – (list) of ignored extensions</li>
+<li><strong>ignore_copying</strong> – (bool) shall we ignore ?</li>
+</ul>
+</td>
+</tr>
+<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">(list) of dicts which were not removed</p>
+</td>
+</tr>
+</tbody>
+</table>
</dd></dl>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.HiveOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HiveOperator</code><span class="sig-paren">(</span><em>hql</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>schema='default'</em>, <em>hiveconf_jinja_translate=False</em>, <em>script_begin_tag=None</em>, <em>run_as_owner=False</em>, <em>mapred_queue=None</em>, <em>mapred_queue_priority=None</em>, <em>mapred_job_name=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_operator.html#HiveOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes hql code in a specific Hive database.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>hql</strong> (<em>string</em>) – the hql to be executed</li>
+<li><strong>hive_cli_conn_id</strong> (<em>string</em>) – reference to the Hive database</li>
+<li><strong>hiveconf_jinja_translate</strong> (<em>boolean</em>) – when True, hiveconf-type templating
+${var} gets translated into jinja-type templating {{ var }}. Note that
+you may want to use this along with the
+<code class="docutils literal"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
+object documentation for more details.</li>
+<li><strong>script_begin_tag</strong> (<em>str</em>) – If defined, the operator will get rid of the
+part of the script before the first occurrence of <cite>script_begin_tag</cite></li>
+<li><strong>mapred_queue</strong> (<em>string</em>) – queue used by the Hadoop CapacityScheduler</li>
+<li><strong>mapred_queue_priority</strong> (<em>string</em>) – priority within CapacityScheduler queue.
+Possible settings include: VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW</li>
+<li><strong>mapred_job_name</strong> (<em>string</em>) – This name will appear in the jobtracker.
+This can make monitoring easier.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.HivePartitionSensor">
<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HivePartitionSensor</code><span class="sig-paren">(</span><em>table</em>, <em>partition="ds='{{ ds }}'"</em>, <em>metastore_conn_id='metastore_default'</em>, <em>schema='default'</em>, <em>poke_interval=180</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HivePartitionSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HivePartitionSensor" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
<p>Waits for a partition to show up in Hive.</p>
<p>Note: Because <code class="docutils literal"><span class="pre">partition</span></code> supports general logical operators, it
can be inefficient. Consider using NamedHivePartitionSensor instead if
-you don’t need the full flexibility of HivePartitionSensor.</p>
+you don’t need the full flexibility of HivePartitionSensor.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>table</strong> (<em>string</em>) – The name of the table to wait for, supports the dot
+<li><strong>table</strong> (<em>string</em>) – The name of the table to wait for, supports the dot
notation (my_database.my_table)</li>
-<li><strong>partition</strong> (<em>string</em>) – The partition clause to wait for. This is passed as
+<li><strong>partition</strong> (<em>string</em>) – The partition clause to wait for. This is passed as
is to the metastore Thrift client <code class="docutils literal"><span class="pre">get_partitions_by_filter</span></code> method,
and apparently supports SQL like notation as in <code class="docutils literal"><span class="pre">ds='2015-01-01'</span>
<span class="pre">AND</span> <span class="pre">type='value'</span></code> and comparison operators as in <code class="docutils literal"><span class="pre">"ds>=2015-01-01"</span></code></li>
-<li><strong>metastore_conn_id</strong> (<em>str</em>) – reference to the metastore thrift service
+<li><strong>metastore_conn_id</strong> (<em>str</em>) – reference to the metastore thrift service
connection id</li>
</ul>
</td>
@@ -587,6 +644,89 @@ connection id</li>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.HiveToDruidTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HiveToDruidTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>druid_datasource</em>, <em>ts_dim</em>, <em>metric_spec=None</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>druid_ingest_conn_id='druid_ingest_default'</em>, <em>metastore_conn_id='metastore_default'</em>, <em>hadoop_dependency_coordinates=None</em>, <em>intervals=None</em>, <em>num_shards=-1</em>, <em>target_partition_size=-1</em>, <em>query_granularity='NONE'</em>, <em>segment_granularity='DAY'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_to_druid.html#HiveToDruidTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveToDruidTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Hive to Druid, [del]note that for now the data is loaded
+into memory before being pushed to Druid, so this operator should
+be used for smallish amount of data.[/del]</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the Druid database</li>
+<li><strong>druid_datasource</strong> (<em>str</em>) – the datasource you want to ingest into in druid</li>
+<li><strong>ts_dim</strong> (<em>str</em>) – the timestamp dimension</li>
+<li><strong>metric_spec</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – the metrics you want to define for your data</li>
+<li><strong>hive_cli_conn_id</strong> (<em>str</em>) – the hive connection id</li>
+<li><strong>druid_ingest_conn_id</strong> (<em>str</em>) – the druid ingest connection id</li>
+<li><strong>metastore_conn_id</strong> (<em>str</em>) – the metastore connection id</li>
+<li><strong>hadoop_dependency_coordinates</strong> (<em>list of str</em>) – list of coordinates to squeeze
+int the ingest json</li>
+<li><strong>intervals</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – list of time intervals that defines segments, this
+is passed as is to the json object</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+<dl class="method">
+<dt id="airflow.operators.HiveToDruidTransfer.construct_ingest_query">
+<code class="descname">construct_ingest_query</code><span class="sig-paren">(</span><em>static_path</em>, <em>columns</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_to_druid.html#HiveToDruidTransfer.construct_ingest_query"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveToDruidTransfer.construct_ingest_query" title="Permalink to this definition">¶</a></dt>
+<dd><p>Builds an ingest query for an HDFS TSV load.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>static_path</strong> (<em>str</em>) – The path on hdfs where the data is</li>
+<li><strong>columns</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – List of all the columns that are available</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HiveToMySqlTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HiveToMySqlTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_table</em>, <em>hiveserver2_conn_id='hiveserver2_default'</em>, <em>mysql_conn_id='mysql_default'</em>, <em>mysql_preoperator=None</em>, <em>mysql_postoperator=None</em>, <em>bulk_load=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_to_mysql.html#HiveToMySqlTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveToMySqlTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Hive to MySQL, note that for now the data is loaded
+into memory before being pushed to MySQL, so this operator should
+be used for smallish amount of data.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the MySQL database</li>
+<li><strong>mysql_table</strong> (<em>str</em>) – target MySQL table, use dot notation to target a
+specific database</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – source mysql connection</li>
+<li><strong>hiveserver2_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>mysql_preoperator</strong> (<em>str</em>) – sql statement to run against mysql prior to
+import, typically use to truncate of delete in place of the data
+coming in, allowing the task to be idempotent (running the task
+twice won’t double load data)</li>
+<li><strong>mysql_postoperator</strong> (<em>str</em>) – sql statement to run against mysql after the
+import, typically used to move data from staging to production
+and issue cleanup commands.</li>
+<li><strong>bulk_load</strong> (<em>bool</em>) – flag to use bulk_load option. This loads mysql directly
+from a tab-delimited text file using the LOAD DATA LOCAL INFILE command.
+This option requires an extra connection parameter for the
+destination MySQL connection: {‘local_infile’: true}.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.SimpleHttpOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SimpleHttpOperator</code><span class="sig-paren">(</span><em>endpoint</em>, <em>method='POST'</em>, <em>data=None</em>, <em>headers=None</em>, <em>response_check=None</em>, <em>extra_options=None</em>, <em>xcom_push=False</em>, <em>http_conn_id='http_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/http_operator.html#SimpleHttpOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SimpleHttpOperator" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
@@ -596,18 +736,18 @@ connection id</li>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>http_conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
-<li><strong>endpoint</strong> (<em>string</em>) – The relative part of the full url</li>
-<li><strong>method</strong> (<em>string</em>) – The HTTP method to use, default = “POST”</li>
+<li><strong>http_conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
+<li><strong>endpoint</strong> (<em>string</em>) – The relative part of the full url</li>
+<li><strong>method</strong> (<em>string</em>) – The HTTP method to use, default = “POST”</li>
<li><strong>data</strong> (<em>For POST/PUT</em><em>, </em><em>depends on the content-type parameter</em><em>,
-</em><em>for GET a dictionary of key/value string pairs</em>) – The data to pass. POST-data in POST/PUT and params
+</em><em>for GET a dictionary of key/value string pairs</em>) – The data to pass. POST-data in POST/PUT and params
in the URL for a GET request.</li>
-<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) – The HTTP headers to be added to the GET request</li>
-<li><strong>response_check</strong> (<em>A lambda</em><em> or </em><em>defined function.</em>) – A check against the ‘requests’ response object.
-Returns True for ‘pass’ and False otherwise.</li>
+<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) – The HTTP headers to be added to the GET request</li>
+<li><strong>response_check</strong> (<em>A lambda</em><em> or </em><em>defined function.</em>) – A check against the ‘requests’ response object.
+Returns True for ‘pass’ and False otherwise.</li>
<li><strong>extra_options</strong> (<em>A dictionary of options</em><em>, </em><em>where key is string and value
-depends on the option that's being modified.</em>) – Extra options for the ‘requests’ library, see the
-‘requests’ documentation (options to modify timeout, ssl, etc.)</li>
+depends on the option that's being modified.</em>) – Extra options for the ‘requests’ library, see the
+‘requests’ documentation (options to modify timeout, ssl, etc.)</li>
</ul>
</td>
</tr>
@@ -617,7 +757,7 @@ depends on the option that's being modified.</em>) – Extra options for the
<dl class="class">
<dt id="airflow.operators.HttpSensor">
-<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HttpSensor</code><span class="sig-paren">(</span><em>endpoint</em>, <em>http_conn_id='http_default'</em>, <em>params=None</em>, <em>headers=None</em>, <em>response_check=None</em>, <em>extra_options=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HttpSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HttpSensor" title="Permalink to this definition">¶</a></dt>
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HttpSensor</code><span class="sig-paren">(</span><em>endpoint</em>, <em>http_conn_id='http_default'</em>, <em>method='GET'</em>, <em>request_params=None</em>, <em>headers=None</em>, <em>response_check=None</em>, <em>extra_options=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HttpSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HttpSensor" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
<dl class="docutils">
<dt>Executes a HTTP get statement and returns False on failure:</dt>
@@ -628,15 +768,16 @@ depends on the option that's being modified.</em>) – Extra options for the
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>http_conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
-<li><strong>endpoint</strong> (<em>string</em>) – The relative part of the full url</li>
-<li><strong>params</strong> (<em>a dictionary of string key/value pairs</em>) – The parameters to be added to the GET url</li>
-<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) – The HTTP headers to be added to the GET request</li>
-<li><strong>response_check</strong> (<em>A lambda</em><em> or </em><em>defined function.</em>) – A check against the ‘requests’ response object.
-Returns True for ‘pass’ and False otherwise.</li>
+<li><strong>http_conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
+<li><strong>method</strong> (<em>string</em>) – The HTTP request method to use</li>
+<li><strong>endpoint</strong> (<em>string</em>) – The relative part of the full url</li>
+<li><strong>request_params</strong> (<em>a dictionary of string key/value pairs</em>) – The parameters to be added to the GET url</li>
+<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) – The HTTP headers to be added to the GET request</li>
+<li><strong>response_check</strong> (<em>A lambda</em><em> or </em><em>defined function.</em>) – A check against the ‘requests’ response object.
+Returns True for ‘pass’ and False otherwise.</li>
<li><strong>extra_options</strong> (<em>A dictionary of options</em><em>, </em><em>where key is string and value
-depends on the option that's being modified.</em>) – Extra options for the ‘requests’ library, see the
-‘requests’ documentation (options to modify timeout, ssl, etc.)</li>
+depends on the option that's being modified.</em>) – Extra options for the ‘requests’ library, see the
+‘requests’ documentation (options to modify timeout, ssl, etc.)</li>
</ul>
</td>
</tr>
@@ -651,20 +792,20 @@ depends on the option that's being modified.</em>) – Extra options for the
<p>An alternative to the HivePartitionSensor that talk directly to the
MySQL db. This was created as a result of observing sub optimal
queries generated by the Metastore thrift service when hitting
-subpartitioned tables. The Thrift service’s queries were written in a
-way that wouldn’t leverage the indexes.</p>
+subpartitioned tables. The Thrift service’s queries were written in a
+way that wouldn’t leverage the indexes.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>schema</strong> (<em>str</em>) – the schema</li>
-<li><strong>table</strong> (<em>str</em>) – the table</li>
-<li><strong>partition_name</strong> (<em>str</em>) – the partition name, as defined in the PARTITIONS
+<li><strong>schema</strong> (<em>str</em>) – the schema</li>
+<li><strong>table</strong> (<em>str</em>) – the table</li>
+<li><strong>partition_name</strong> (<em>str</em>) – the partition name, as defined in the PARTITIONS
table of the Metastore. Order of the fields does matter.
Examples: <code class="docutils literal"><span class="pre">ds=2016-01-01</span></code> or
<code class="docutils literal"><span class="pre">ds=2016-01-01/sub=foo</span></code> for a sub partitioned table</li>
-<li><strong>mysql_conn_id</strong> (<em>str</em>) – a reference to the MySQL conn_id for the metastore</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – a reference to the MySQL conn_id for the metastore</li>
</ul>
</td>
</tr>
@@ -674,7 +815,7 @@ Examples: <code class="docutils literal"><span class="pre">ds=2016-01-01</span><
<dl class="class">
<dt id="airflow.operators.MySqlOperator">
-<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MySqlOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_conn_id='mysql_default'</em>, <em>parameters=None</em>, <em>autocommit=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mysql_operator.html#MySqlOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MySqlOperator" title="Permalink to this definition">¶</a></dt>
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MySqlOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_conn_id='mysql_default'</em>, <em>parameters=None</em>, <em>autocommit=False</em>, <em>database=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mysql_operator.html#MySqlOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MySqlOperator" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific MySQL database</p>
<table class="docutils field-list" frame="void" rules="none">
@@ -682,10 +823,50 @@ Examples: <code class="docutils literal"><span class="pre">ds=2016-01-01</span><
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>mysql_conn_id</strong> (<em>string</em>) – reference to a specific mysql database</li>
+<li><strong>mysql_conn_id</strong> (<em>string</em>) – reference to a specific mysql database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
-</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em></em><em>, or </em><em>reference to a template file.
-Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+<li><strong>database</strong> (<em>string</em>) – name of database which overwrite defined one in connection</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.MySqlToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MySqlToHiveTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>hive_table</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>delimiter=u'x01'</em>, <em>mysql_conn_id='mysql_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>tblproperties=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mysql_to_hive.html#MySqlToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MySqlToHiveTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from MySql to Hive. The operator runs your query against
+MySQL, stores the file locally before loading it into a Hive table.
+If the <code class="docutils literal"><span class="pre">create</span></code> or <code class="docutils literal"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal"><span class="pre">True</span></code>,
+a <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata. Note that the
+table generated in Hive uses <code class="docutils literal"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the table gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the MySQL database</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a
+specific database</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every
+execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns
+and values</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – source mysql connection</li>
+<li><strong>hive_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>tblproperties</strong> (<em>dict</em>) – TBLPROPERTIES of the hive table being created</li>
</ul>
</td>
</tr>
@@ -703,14 +884,14 @@ Template reference are recognized by str ending in '.sql'</em>) – the sql
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>partition_names</strong> (<em>list of strings</em>) – List of fully qualified names of the
+<li><strong>partition_names</strong> (<em>list of strings</em>) – List of fully qualified names of the
partitions to wait for. A fully qualified name is of the
form <code class="docutils literal"><span class="pre">schema.table/pk1=pv1/pk2=pv2</span></code>, for example,
default.users/ds=2016-01-01. This is passed as is to the metastore
Thrift client <code class="docutils literal"><span class="pre">get_partitions_by_name</span></code> method. Note that
you cannot use logical or comparison operators as in
HivePartitionSensor.</li>
-<li><strong>metastore_conn_id</strong> (<em>str</em>) – reference to the metastore thrift service
+<li><strong>metastore_conn_id</strong> (<em>str</em>) – reference to the metastore thrift service
connection id</li>
</ul>
</td>
@@ -720,6 +901,28 @@ connection id</li>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.PostgresOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PostgresOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>postgres_conn_id='postgres_default'</em>, <em>autocommit=False</em>, <em>parameters=None</em>, <em>database=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/postgres_operator.html#PostgresOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PostgresOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes sql code in a specific Postgres database</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>postgres_conn_id</strong> (<em>string</em>) – reference to a specific postgres database</li>
+<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
+</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+<li><strong>database</strong> (<em>string</em>) – name of database which overwrite defined one in connection</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.PrestoCheckOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PrestoCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>presto_conn_id='presto_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/presto_check_operator.html#PrestoCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PrestoCheckOperator" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">airflow.operators.check_operator.CheckOperator</span></code></p>
@@ -738,8 +941,8 @@ values return <code class="docutils literal"><span class="pre">False</span></cod
<p>Given a query like <code class="docutils literal"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
the count <code class="docutils literal"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
-the source table upstream, or that the count of today’s partition is
-greater than yesterday’s partition, or that a set of metrics are less
+the source table upstream, or that the count of today’s partition is
+greater than yesterday’s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.</p>
<p>This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
@@ -751,8 +954,8 @@ without stopping the progress of the DAG.</p>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
-<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
+<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
+<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
</ul>
</td>
</tr>
@@ -771,11 +974,11 @@ a certain tolerance of the ones from days_back before.</p>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>table</strong> (<em>str</em>) – the table name</li>
-<li><strong>days_back</strong> (<em>int</em>) – number of days between ds and the ds we want to check
+<li><strong>table</strong> (<em>str</em>) – the table name</li>
+<li><strong>days_back</strong> (<em>int</em>) – number of days between ds and the ds we want to check
against. Defaults to 7 days</li>
-<li><strong>metrics_threshold</strong> (<em>dict</em>) – a dictionary of ratios indexed by metrics</li>
-<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
+<li><strong>metrics_threshold</strong> (<em>dict</em>) – a dictionary of ratios indexed by metrics</li>
+<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
</ul>
</td>
</tr>
@@ -793,8 +996,8 @@ against. Defaults to 7 days</li>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
-<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
+<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
+<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
</ul>
</td>
</tr>
@@ -812,21 +1015,21 @@ against. Defaults to 7 days</li>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>python_callable</strong> (<em>python callable</em>) – A reference to an object that is callable</li>
-<li><strong>op_kwargs</strong> (<em>dict</em>) – a dictionary of keyword arguments that will get unpacked
+<li><strong>python_callable</strong> (<em>python callable</em>) – A reference to an object that is callable</li>
+<li><strong>op_kwargs</strong> (<em>dict</em>) – a dictionary of keyword arguments that will get unpacked
in your function</li>
-<li><strong>op_args</strong> (<em>list</em>) – a list of positional arguments that will get unpacked when
+<li><strong>op_args</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – a list of positional arguments that will get unpacked when
calling your callable</li>
-<li><strong>provide_context</strong> (<em>bool</em>) – if set to true, Airflow will pass a set of
+<li><strong>provide_context</strong> (<em>bool</em>) – if set to true, Airflow will pass a set of
keyword arguments that can be used in your function. This set of
kwargs correspond exactly to what you can use in your jinja
templates. For this to work, you need to define <cite>**kwargs</cite> in your
function header.</li>
-<li><strong>templates_dict</strong> (<em>dict of str</em>) – a dictionary where the values are templates that
+<li><strong>templates_dict</strong> (<em>dict of str</em>) – a dictionary where the values are templates that
will get templated by the Airflow engine sometime between
<code class="docutils literal"><span class="pre">__init__</span></code> and <code class="docutils literal"><span class="pre">execute</span></code> takes place and are made available
-in your callable’s context after the template has been applied</li>
-<li><strong>templates_exts</strong> – a list of file extensions to resolve while
+in your callable’s context after the template has been applied</li>
+<li><strong>templates_exts</strong> – a list of file extensions to resolve while
processing templated fields, for examples <code class="docutils literal"><span class="pre">['.sql',</span> <span class="pre">'.hql']</span></code></li>
</ul>
</td>
@@ -837,7 +1040,7 @@ processing templated fields, for examples <code class="docutils literal"><span c
<dl class="class">
<dt id="airflow.operators.S3KeySensor">
-<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">S3KeySensor</code><span class="sig-paren">(</span><em>bucket_key</em>, <em>bucket_name=None</em>, <em>wildcard_match=False</em>, <em>s3_conn_id='s3_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#S3KeySensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.S3KeySensor" title="Permalink to this definition">¶</a></dt>
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">S3KeySensor</code><span class="sig-paren">(</span><em>bucket_key</em>, <em>bucket_name=None</em>, <em>wildcard_match=False</em>, <em>aws_conn_id='aws_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#S3KeySensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.S3KeySensor" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
<p>Waits for a key (a file-like instance on S3) to be present in a S3 bucket.
S3 being a key/value it does not support folders. The path is just a key
@@ -847,12 +1050,61 @@ a resource.</p>
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>bucket_key</strong> (<em>str</em>) – The key being waited on. Supports full s3:// style url
+<li><strong>bucket_key</strong> (<em>str</em>) – The key being waited on. Supports full s3:// style url
or relative path from root level.</li>
-<li><strong>bucket_name</strong> (<em>str</em>) – Name of the S3 bucket</li>
-<li><strong>wildcard_match</strong> (<em>bool</em>) – whether the bucket_key should be interpreted as a
+<li><strong>bucket_name</strong> (<em>str</em>) – Name of the S3 bucket</li>
+<li><strong>wildcard_match</strong> (<em>bool</em>) – whether the bucket_key should be interpreted as a
Unix wildcard pattern</li>
-<li><strong>s3_conn_id</strong> (<em>str</em>) – a reference to the s3 connection</li>
+<li><strong>aws_conn_id</strong> (<em>str</em>) – a reference to the s3 connection</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.S3ToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">S3ToHiveTransfer</code><span class="sig-paren">(</span><em>s3_key</em>, <em>field_dict</em>, <em>hive_table</em>, <em>delimiter='</em>, <em>'</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>headers=False</em>, <em>check_headers=False</em>, <em>wildcard_match=False</em>, <em>aws_conn_id='aws_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>input_compressed=False</em>, <em>tblproperties=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/s3_to_hive_operator.html#S3ToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.S3ToHiveTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from S3 to Hive. The operator downloads a file from S3,
+stores the file locally before loading it into a Hive table.
+If the <code class="docutils literal"><span class="pre">create</span></code> or <code class="docutils literal"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal"><span class="pre">True</span></code>,
+a <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata from.</p>
+<p>Note that the table generated in Hive uses <code class="docutils literal"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the tables gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>s3_key</strong> (<em>str</em>) – The key to be retrieved from S3</li>
+<li><strong>field_dict</strong> (<em>dict</em>) – A dictionary of the fields name in the file
+as keys and their Hive types as values</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a
+specific database</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every
+execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns
+and values</li>
+<li><strong>headers</strong> (<em>bool</em>) – whether the file contains column names on the first
+line</li>
+<li><strong>check_headers</strong> (<em>bool</em>) – whether the column names on the first line should be
+checked against the keys of field_dict</li>
+<li><strong>wildcard_match</strong> (<em>bool</em>) – whether the s3_key should be interpreted as a Unix
+wildcard pattern</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>aws_conn_id</strong> (<em>str</em>) – source s3 connection</li>
+<li><strong>hive_cli_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>input_compressed</strong> (<em>bool</em>) – Boolean to determine if file decompression is
+required to process headers</li>
+<li><strong>tblproperties</strong> (<em>dict</em>) – TBLPROPERTIES of the hive table being created</li>
</ul>
</td>
</tr>
@@ -863,29 +1115,89 @@ Unix wildcard pattern</li>
<dl class="class">
<dt id="airflow.operators.ShortCircuitOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">ShortCircuitOperator</code><span class="sig-paren">(</span><em>python_callable</em>, <em>op_args=None</em>, <em>op_kwargs=None</em>, <em>provide_context=False</em>, <em>templates_dict=None</em>, <em>templates_exts=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/python_operator.html#ShortCircuitOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.ShortCircuitOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">python_operator.PythonOperator</span></code></p>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">python_operator.PythonOperator</span></code>, <code class="xref py py-class docutils literal"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Allows a workflow to continue only if a condition is met. Otherwise, the
-workflow “short-circuits” and downstream tasks are skipped.</p>
+workflow “short-circuits” and downstream tasks are skipped.</p>
<p>The ShortCircuitOperator is derived from the PythonOperator. It evaluates a
condition and short-circuits the workflow if the condition is False. Any
-downstream tasks are marked with a state of “skipped”. If the condition is
+downstream tasks are marked with a state of “skipped”. If the condition is
True, downstream tasks proceed as normal.</p>
<p>The condition is determined by the result of <cite>python_callable</cite>.</p>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.SlackAPIOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SlackAPIOperator</code><span class="sig-paren">(</span><em>token='unset'</em>, <em>method='unset'</em>, <em>api_params=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Base Slack Operator
+The SlackAPIPostOperator is derived from this operator.
+In the future additional Slack API Operators will be derived from this class as well</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>token</strong> (<em>string</em>) – Slack API token (<a class="reference external" href="https://api.slack.com/web">https://api.slack.com/web</a>)</li>
+<li><strong>method</strong> (<em>string</em>) – The Slack API Method to Call (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</li>
+<li><strong>api_params</strong> (<em>dict</em>) – API Method call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+<dl class="method">
+<dt id="airflow.operators.SlackAPIOperator.construct_api_call_params">
+<code class="descname">construct_api_call_params</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIOperator.construct_api_call_params"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIOperator.construct_api_call_params" title="Permalink to this definition">¶</a></dt>
+<dd><p>Used by the execute function. Allows templating on the source fields of the api_call_params dict before construction</p>
+<p>Override in child classes.
+Each SlackAPIOperator child class is responsible for having a construct_api_call_params function
+which sets self.api_call_params with a dict of API call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.operators.SlackAPIOperator.execute">
+<code class="descname">execute</code><span class="sig-paren">(</span><em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIOperator.execute"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIOperator.execute" title="Permalink to this definition">¶</a></dt>
+<dd><p>SlackAPIOperator calls will not fail even if the call is not unsuccessful.
+It should not prevent a DAG from completing in success</p>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.SlackAPIPostOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SlackAPIPostOperator</code><span class="sig-paren">(</span><em>channel='#general'</em>, <em>username='Airflow'</em>, <em>text='No message has been set.nHere is a cat video insteadnhttps://www.youtube.com/watch?v=J---aiyznGQ'</em>, <em>icon_url='https://raw.githubusercontent.com/airbnb/airflow/master/airflow/www/static/pin_100.png'</em>, <em>attachments=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIPostOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIPostOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">slack_operator.SlackAPIOperator</span></code></p>
+<p>Posts messages to a slack channel</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>channel</strong> (<em>string</em>) – channel in which to post message on slack name (#general) or ID (C12318391)</li>
+<li><strong>username</strong> (<em>string</em>) – Username that airflow will be posting to Slack as</li>
+<li><strong>text</strong> (<em>string</em>) – message to send to slack</li>
+<li><strong>icon_url</strong> (<em>string</em>) – url to icon used for this message</li>
+<li><strong>attachments</strong> (<em>array of hashes</em>) – extra formatting details - see <a class="reference external" href="https://api.slack.com/docs/attachments">https://api.slack.com/docs/attachments</a></li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.SqlSensor">
<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SqlSensor</code><span class="sig-paren">(</span><em>conn_id</em>, <em>sql</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#SqlSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SqlSensor" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
-<p>Runs a sql statement until a criteria is met. It will keep trying until
-sql returns no row, or if the first cell in (0, ‘0’, ‘’).</p>
+<p>Runs a sql statement until a criteria is met. It will keep trying while
+sql returns no row, or if the first cell in (0, ‘0’, ‘’).</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
-<li><strong>sql</strong> – The sql to run. To pass, it needs to return at least one cell
+<li><strong>conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
+<li><strong>sql</strong> – The sql to run. To pass, it needs to return at least one cell
that contains a non-zero / empty string value.</li>
</ul>
</td>
@@ -903,7 +1215,7 @@ that contains a non-zero / empty string value.</li>
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
-<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>target_time</strong> (<em>datetime.time</em>) – time after which the job succeeds</td>
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>target_time</strong> (<em>datetime.time</em>) – time after which the job succeeds</td>
</tr>
</tbody>
</table>
@@ -916,6 +1228,60 @@ that contains a non-zero / empty string value.</li>
<p>Waits for a file or folder to land in HDFS</p>
</dd></dl>
+<dl class="class">
+<dt id="airflow.operators.docker_operator.DockerOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.docker_operator.</code><code class="descname">DockerOperator</code><span class="sig-paren">(</span><em>image</em>, <em>api_version=None</em>, <em>command=None</em>, <em>cpus=1.0</em>, <em>docker_url='unix://var/run/docker.sock'</em>, <em>environment=None</em>, <em>force_pull=False</em>, <em>mem_limit=None</em>, <em>network_mode=None</em>, <em>tls_ca_cert=None</em>, <em>tls_client_cert=None</em>, <em>tls_client_key=None</em>, <em>tls_hostname=None</em>, <em>tls_ssl_version=None</em>, <em>tmp_dir='/tmp/airflow'</em>, <em>user=None</em>, <em>volumes=None</em>, <em>working_dir=None</em>, <em>xcom_push=False</em>, <em>xcom_all=False</em>, <em>docker_conn_id=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/docker_operator.html#DockerOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflo
w.operators.docker_operator.DockerOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Execute a command inside a docker container.</p>
+<p>A temporary directory is created on the host and mounted into a container to allow storing files
+that together exceed the default disk size of 10GB in a container. The path to the mounted
+directory can be accessed via the environment variable <code class="docutils literal"><span class="pre">AIRFLOW_TMP_DIR</span></code>.</p>
+<p>If a login to a private registry is required prior to pulling the image, a
+Docker connection needs to be configured in Airflow and the connection ID
+be provided with the parameter <code class="docutils literal"><span class="pre">docker_conn_id</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>image</strong> (<em>str</em>) – Docker image from which to create the container.</li>
+<li><strong>api_version</strong> (<em>str</em>) – Remote API version. Set to <code class="docutils literal"><span class="pre">auto</span></code> to automatically
+detect the server’s version.</li>
+<li><strong>command</strong> (<em>str</em><em> or </em><a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – Command to be run in the container.</li>
+<li><strong>cpus</strong> (<em>float</em>) – Number of CPUs to assign to the container.
+This value gets multiplied with 1024. See
+<a class="reference external" href="https://docs.docker.com/engine/reference/run/#cpu-share-constraint">https://docs.docker.com/engine/reference/run/#cpu-share-constraint</a></li>
+<li><strong>docker_url</strong> (<em>str</em>) – URL of the host running the docker daemon.
+Default is unix://var/run/docker.sock</li>
+<li><strong>environment</strong> (<em>dict</em>) – Environment variables to set in the container.</li>
+<li><strong>force_pull</strong> (<em>bool</em>) – Pull the docker image on every run. Default is false.</li>
+<li><strong>mem_limit</strong> (<em>float</em><em> or </em><em>str</em>) – Maximum amount of memory the container can use. Either a float value, which
+represents the limit in bytes, or a string like <code class="docutils literal"><span class="pre">128m</span></code> or <code class="docutils literal"><span class="pre">1g</span></code>.</li>
+<li><strong>network_mode</strong> (<em>str</em>) – Network mode for the container.</li>
+<li><strong
<TRUNCATED>
[18/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/bigquery_hook.html
----------------------------------------------------------------------
diff --git a/_modules/bigquery_hook.html b/_modules/bigquery_hook.html
index 1f8170a..e58a93b 100644
--- a/_modules/bigquery_hook.html
+++ b/_modules/bigquery_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>bigquery_hook</li>
+
+ <li>bigquery_hook</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -157,25 +187,24 @@
<span class="sd">implementation for BigQuery.</span>
<span class="sd">"""</span>
-<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">range</span>
-<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="kn">import</span> <span class="nb">basestring</span>
-
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">time</span>
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="kn">import</span> <span class="n">GoogleCloudBaseHook</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
-<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="kn">import</span> <span class="n">build</span>
-<span class="kn">from</span> <span class="nn">pandas.io.gbq</span> <span class="kn">import</span> <span class="n">GbqConnector</span><span class="p">,</span> \
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span><span class="p">,</span> <span class="n">HttpError</span>
+<span class="kn">from</span> <span class="nn">googleapiclient</span> <span class="k">import</span> <span class="n">errors</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">range</span>
+<span class="kn">from</span> <span class="nn">pandas_gbq.gbq</span> <span class="k">import</span> <span class="n">GbqConnector</span><span class="p">,</span> \
<span class="n">_parse_data</span> <span class="k">as</span> <span class="n">gbq_parse_data</span><span class="p">,</span> \
<span class="n">_check_google_client_version</span> <span class="k">as</span> <span class="n">gbq_check_google_client_version</span><span class="p">,</span> \
<span class="n">_test_google_api_imports</span> <span class="k">as</span> <span class="n">gbq_test_google_api_imports</span>
-<span class="kn">from</span> <span class="nn">pandas.tools.merge</span> <span class="kn">import</span> <span class="n">concat</span>
+<span class="kn">from</span> <span class="nn">pandas.tools.merge</span> <span class="k">import</span> <span class="n">concat</span>
+<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
-<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"bigquery"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="k">import</span> <span class="n">DbApiHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
-<div class="viewcode-block" id="BigQueryHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook">[docs]</a><span class="k">class</span> <span class="nc">BigQueryHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">,</span> <span class="n">DbApiHook</span><span class="p">):</span>
+<div class="viewcode-block" id="BigQueryHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook">[docs]</a><span class="k">class</span> <span class="nc">BigQueryHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">,</span> <span class="n">DbApiHook</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Interact with BigQuery. This hook uses the Google Cloud Platform</span>
<span class="sd"> connection.</span>
@@ -184,8 +213,8 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
- <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">conn_id</span><span class="o">=</span><span class="n">bigquery_conn_id</span><span class="p">,</span>
<span class="n">delegate_to</span><span class="o">=</span><span class="n">delegate_to</span><span class="p">)</span>
@@ -204,7 +233,7 @@
<span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
<span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'bigquery'</span><span class="p">,</span> <span class="s1">'v2'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
-<div class="viewcode-block" id="BigQueryHook.insert_rows"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook.insert_rows">[docs]</a> <span class="k">def</span> <span class="nf">insert_rows</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">target_fields</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">commit_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
+<div class="viewcode-block" id="BigQueryHook.insert_rows"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook.insert_rows">[docs]</a> <span class="k">def</span> <span class="nf">insert_rows</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">target_fields</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">commit_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Insertion is currently unsupported. Theoretically, you could use</span>
<span class="sd"> BigQuery's streaming API to insert rows into a table, but this hasn't</span>
@@ -212,7 +241,7 @@
<span class="sd"> """</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span></div>
-<div class="viewcode-block" id="BigQueryHook.get_pandas_df"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+<div class="viewcode-block" id="BigQueryHook.get_pandas_df"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="s1">'legacy'</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a Pandas DataFrame for the results produced by a BigQuery</span>
<span class="sd"> query. The DbApiHook method must be overridden because Pandas</span>
@@ -223,10 +252,14 @@
<span class="sd"> :param bql: The BigQuery SQL to execute.</span>
<span class="sd"> :type bql: string</span>
+<span class="sd"> :param parameters: The parameters to render the SQL query with (not used, leave to override superclass method)</span>
+<span class="sd"> :type parameters: mapping or iterable</span>
+<span class="sd"> :param dialect: Dialect of BigQuery SQL – legacy SQL or standard SQL</span>
+<span class="sd"> :type dialect: string in {'legacy', 'standard'}, default 'legacy'</span>
<span class="sd"> """</span>
<span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_service</span><span class="p">()</span>
<span class="n">project</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_field</span><span class="p">(</span><span class="s1">'project'</span><span class="p">)</span>
- <span class="n">connector</span> <span class="o">=</span> <span class="n">BigQueryPandasConnector</span><span class="p">(</span><span class="n">project</span><span class="p">,</span> <span class="n">service</span><span class="p">)</span>
+ <span class="n">connector</span> <span class="o">=</span> <span class="n">BigQueryPandasConnector</span><span class="p">(</span><span class="n">project</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="n">dialect</span><span class="p">)</span>
<span class="n">schema</span><span class="p">,</span> <span class="n">pages</span> <span class="o">=</span> <span class="n">connector</span><span class="o">.</span><span class="n">run_query</span><span class="p">(</span><span class="n">bql</span><span class="p">)</span>
<span class="n">dataframe_list</span> <span class="o">=</span> <span class="p">[]</span>
@@ -235,9 +268,35 @@
<span class="n">dataframe_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">gbq_parse_data</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">page</span><span class="p">))</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">dataframe_list</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">concat</span><span class="p">(</span><span class="n">dataframe_list</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">concat</span><span class="p">(</span><span class="n">dataframe_list</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">gbq_parse_data</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="p">[])</span></div></div>
+ <span class="k">return</span> <span class="n">gbq_parse_data</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="p">[])</span></div>
+
+<div class="viewcode-block" id="BigQueryHook.table_exists"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.BigQueryHook.table_exists">[docs]</a> <span class="k">def</span> <span class="nf">table_exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks for the existence of a table in Google BigQuery.</span>
+
+<span class="sd"> :param project_id: The Google cloud project in which to look for the table. The connection supplied to the hook</span>
+<span class="sd"> must provide access to the specified project.</span>
+<span class="sd"> :type project_id: string</span>
+<span class="sd"> :param dataset_id: The name of the dataset in which to look for the table.</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type dataset_id: string</span>
+<span class="sd"> :param table_id: The name of the table to check the existence of.</span>
+<span class="sd"> :type table_id: string</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_service</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span>
+ <span class="n">tableId</span><span class="o">=</span><span class="n">table_id</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div></div>
<span class="k">class</span> <span class="nc">BigQueryPandasConnector</span><span class="p">(</span><span class="n">GbqConnector</span><span class="p">):</span>
@@ -248,13 +307,14 @@
<span class="sd"> without forcing a three legged OAuth connection. Instead, we can inject</span>
<span class="sd"> service account credentials into the binding.</span>
<span class="sd"> """</span>
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">reauth</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">reauth</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">dialect</span><span class="o">=</span><span class="s1">'legacy'</span><span class="p">):</span>
<span class="n">gbq_check_google_client_version</span><span class="p">()</span>
<span class="n">gbq_test_google_api_imports</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reauth</span> <span class="o">=</span> <span class="n">reauth</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">service</span>
<span class="bp">self</span><span class="o">.</span><span class="n">verbose</span> <span class="o">=</span> <span class="n">verbose</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dialect</span> <span class="o">=</span> <span class="n">dialect</span>
<span class="k">class</span> <span class="nc">BigQueryConnection</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
@@ -285,22 +345,25 @@
<span class="s2">"BigQueryConnection does not have transactions"</span><span class="p">)</span>
-<span class="k">class</span> <span class="nc">BigQueryBaseCursor</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+<span class="k">class</span> <span class="nc">BigQueryBaseCursor</span><span class="p">(</span><span class="n">LoggingMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> The BigQuery base cursor contains helper methods to execute queries against</span>
<span class="sd"> BigQuery. The methods can be used directly by operators, in cases where a</span>
<span class="sd"> PEP 249 cursor isn't needed.</span>
<span class="sd"> """</span>
-
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">project_id</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service</span> <span class="o">=</span> <span class="n">service</span>
<span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
<span class="k">def</span> <span class="nf">run_query</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span> <span class="n">bql</span><span class="p">,</span> <span class="n">destination_dataset_table</span> <span class="o">=</span> <span class="bp">False</span><span class="p">,</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">bql</span><span class="p">,</span> <span class="n">destination_dataset_table</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
<span class="n">write_disposition</span> <span class="o">=</span> <span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
- <span class="n">allow_large_results</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">udf_config</span> <span class="o">=</span> <span class="bp">False</span><span class="p">):</span>
+ <span class="n">allow_large_results</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">udf_config</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="n">use_legacy_sql</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">maximum_billing_tier</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
+ <span class="n">query_params</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Executes a BigQuery SQL query. Optionally persists results in a BigQuery</span>
<span class="sd"> table. See here:</span>
@@ -315,29 +378,40 @@
<span class="sd"> BigQuery table to save the query results.</span>
<span class="sd"> :param write_disposition: What to do if the table already exists in</span>
<span class="sd"> BigQuery.</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param create_disposition: Specifies whether the job is allowed to create new tables.</span>
+<span class="sd"> :type create_disposition: string</span>
<span class="sd"> :param allow_large_results: Whether to allow large results.</span>
<span class="sd"> :type allow_large_results: boolean</span>
<span class="sd"> :param udf_config: The User Defined Function configuration for the query.</span>
<span class="sd"> See https://cloud.google.com/bigquery/user-defined-functions for details.</span>
<span class="sd"> :type udf_config: list</span>
+<span class="sd"> :param use_legacy_sql: Whether to use legacy SQL (true) or standard SQL (false).</span>
+<span class="sd"> :type use_legacy_sql: boolean</span>
+<span class="sd"> :param maximum_billing_tier: Positive integer that serves as a multiplier of the basic price.</span>
+<span class="sd"> :type maximum_billing_tier: integer</span>
<span class="sd"> """</span>
<span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'query'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'query'</span><span class="p">:</span> <span class="n">bql</span><span class="p">,</span>
+ <span class="s1">'useLegacySql'</span><span class="p">:</span> <span class="n">use_legacy_sql</span><span class="p">,</span>
+ <span class="s1">'maximumBillingTier'</span><span class="p">:</span> <span class="n">maximum_billing_tier</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="n">destination_dataset_table</span><span class="p">:</span>
<span class="k">assert</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">destination_dataset_table</span><span class="p">,</span> <span class="p">(</span>
<span class="s1">'Expected destination_dataset_table in the format of '</span>
- <span class="s1">'<dataset>.<table>. Got: {}'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">destination_dataset_table</span><span class="p">)</span>
- <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
- <span class="n">destination_dataset_table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="s1">'<dataset>.<table>. Got: </span><span class="si">{}</span><span class="s1">'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">destination_dataset_table</span><span class="p">)</span>
+ <span class="n">destination_project</span><span class="p">,</span> <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">destination_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">)</span>
<span class="n">configuration</span><span class="p">[</span><span class="s1">'query'</span><span class="p">]</span><span class="o">.</span><span class="n">update</span><span class="p">({</span>
<span class="s1">'allowLargeResults'</span><span class="p">:</span> <span class="n">allow_large_results</span><span class="p">,</span>
<span class="s1">'writeDisposition'</span><span class="p">:</span> <span class="n">write_disposition</span><span class="p">,</span>
+ <span class="s1">'createDisposition'</span><span class="p">:</span> <span class="n">create_disposition</span><span class="p">,</span>
<span class="s1">'destinationTable'</span><span class="p">:</span> <span class="p">{</span>
- <span class="s1">'projectId'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="n">destination_project</span><span class="p">,</span>
<span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">destination_dataset</span><span class="p">,</span>
<span class="s1">'tableId'</span><span class="p">:</span> <span class="n">destination_table</span><span class="p">,</span>
<span class="p">}</span>
@@ -348,12 +422,15 @@
<span class="s1">'userDefinedFunctionResources'</span><span class="p">:</span> <span class="n">udf_config</span>
<span class="p">})</span>
+ <span class="k">if</span> <span class="n">query_params</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'query'</span><span class="p">][</span><span class="s1">'queryParameters'</span><span class="p">]</span> <span class="o">=</span> <span class="n">query_params</span>
+
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_with_configuration</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run_extract</span><span class="p">(</span> <span class="c1"># noqa</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">source_project_dataset_table</span><span class="p">,</span> <span class="n">destination_cloud_storage_uris</span><span class="p">,</span>
<span class="n">compression</span><span class="o">=</span><span class="s1">'NONE'</span><span class="p">,</span> <span class="n">export_format</span><span class="o">=</span><span class="s1">'CSV'</span><span class="p">,</span> <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
- <span class="n">print_header</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="n">print_header</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Executes a BigQuery extract command to copy data from BigQuery to</span>
<span class="sd"> Google Cloud Storage. See here:</span>
@@ -379,9 +456,12 @@
<span class="sd"> :param print_header: Whether to print a header for a CSV file extract.</span>
<span class="sd"> :type print_header: boolean</span>
<span class="sd"> """</span>
+
<span class="n">source_project</span><span class="p">,</span> <span class="n">source_dataset</span><span class="p">,</span> <span class="n">source_table</span> <span class="o">=</span> \
- <span class="bp">self</span><span class="o">.</span><span class="n">_split_project_dataset_table_input</span><span class="p">(</span>
- <span class="s1">'source_project_dataset_table'</span><span class="p">,</span> <span class="n">source_project_dataset_table</span><span class="p">)</span>
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">source_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">var_name</span><span class="o">=</span><span class="s1">'source_project_dataset_table'</span><span class="p">)</span>
+
<span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'extract'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'sourceTable'</span><span class="p">:</span> <span class="p">{</span>
@@ -418,14 +498,14 @@
<span class="sd"> For more details about these parameters.</span>
<span class="sd"> :param source_project_dataset_tables: One or more dotted</span>
-<span class="sd"> (<project>.)<dataset>.<table></span>
+<span class="sd"> (project:|project.)<dataset>.<table></span>
<span class="sd"> BigQuery tables to use as the source data. Use a list if there are</span>
<span class="sd"> multiple source tables.</span>
<span class="sd"> If <project> is not included, project will be the project defined</span>
<span class="sd"> in the connection json.</span>
<span class="sd"> :type source_project_dataset_tables: list|string</span>
<span class="sd"> :param destination_project_dataset_table: The destination BigQuery</span>
-<span class="sd"> table. Format is: <project>.<dataset>.<table></span>
+<span class="sd"> table. Format is: (project:|project.)<dataset>.<table></span>
<span class="sd"> :type destination_project_dataset_table: string</span>
<span class="sd"> :param write_disposition: The write disposition if the table already exists.</span>
<span class="sd"> :type write_disposition: string</span>
@@ -440,21 +520,18 @@
<span class="n">source_project_dataset_tables_fixup</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">source_project_dataset_table</span> <span class="ow">in</span> <span class="n">source_project_dataset_tables</span><span class="p">:</span>
<span class="n">source_project</span><span class="p">,</span> <span class="n">source_dataset</span><span class="p">,</span> <span class="n">source_table</span> <span class="o">=</span> \
- <span class="bp">self</span><span class="o">.</span><span class="n">_split_project_dataset_table_input</span><span class="p">(</span>
- <span class="s1">'source_project_dataset_table'</span><span class="p">,</span> <span class="n">source_project_dataset_table</span><span class="p">)</span>
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">source_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">var_name</span><span class="o">=</span><span class="s1">'source_project_dataset_table'</span><span class="p">)</span>
<span class="n">source_project_dataset_tables_fixup</span><span class="o">.</span><span class="n">append</span><span class="p">({</span>
<span class="s1">'projectId'</span><span class="p">:</span> <span class="n">source_project</span><span class="p">,</span>
<span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">source_dataset</span><span class="p">,</span>
<span class="s1">'tableId'</span><span class="p">:</span> <span class="n">source_table</span>
<span class="p">})</span>
- <span class="k">assert</span> <span class="mi">3</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">destination_project_dataset_table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)),</span> <span class="p">(</span>
- <span class="s1">'Expected destination_project_dataset_table in the format of '</span>
- <span class="s1">'<project>.<dataset>.<table>. '</span>
- <span class="s1">'Got: {}'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">destination_project_dataset_table</span><span class="p">)</span>
-
<span class="n">destination_project</span><span class="p">,</span> <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
- <span class="n">destination_project_dataset_table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">)</span>
<span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'copy'</span><span class="p">:</span> <span class="p">{</span>
<span class="s1">'createDisposition'</span><span class="p">:</span> <span class="n">create_disposition</span><span class="p">,</span>
@@ -477,7 +554,13 @@
<span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
<span class="n">skip_leading_rows</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
<span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
- <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">):</span>
+ <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">max_bad_records</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
+ <span class="n">quote_character</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">allow_quoted_newlines</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">allow_jagged_rows</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">schema_update_options</span><span class="o">=</span><span class="p">(),</span>
+ <span class="n">src_fmt_configs</span><span class="o">=</span><span class="p">{}):</span>
<span class="sd">"""</span>
<span class="sd"> Executes a BigQuery load command to load data from Google Cloud Storage</span>
<span class="sd"> to BigQuery. See here:</span>
@@ -487,9 +570,9 @@
<span class="sd"> For more details about these parameters.</span>
<span class="sd"> :param destination_project_dataset_table:</span>
-<span class="sd"> The dotted (<project>.)<dataset>.<table> BigQuery table to load data into.</span>
-<span class="sd"> If <project> is not included, project will be the project defined in</span>
-<span class="sd"> the connection json.</span>
+<span class="sd"> The dotted (<project>.|<project>:)<dataset>.<table> BigQuery table to load</span>
+<span class="sd"> data into. If <project> is not included, project will be the project defined</span>
+<span class="sd"> in the connection json.</span>
<span class="sd"> :type destination_project_dataset_table: string</span>
<span class="sd"> :param schema_fields: The schema field list as defined here:</span>
<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load</span>
@@ -508,10 +591,56 @@
<span class="sd"> :type write_disposition: string</span>
<span class="sd"> :param field_delimiter: The delimiter to use when loading from a CSV.</span>
<span class="sd"> :type field_delimiter: string</span>
+<span class="sd"> :param max_bad_records: The maximum number of bad records that BigQuery can</span>
+<span class="sd"> ignore when running the job.</span>
+<span class="sd"> :type max_bad_records: int</span>
+<span class="sd"> :param quote_character: The value that is used to quote data sections in a CSV file.</span>
+<span class="sd"> :type quote_character: string</span>
+<span class="sd"> :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not (false).</span>
+<span class="sd"> :type allow_quoted_newlines: boolean</span>
+<span class="sd"> :param allow_jagged_rows: Accept rows that are missing trailing optional columns.</span>
+<span class="sd"> The missing values are treated as nulls. If false, records with missing trailing columns</span>
+<span class="sd"> are treated as bad records, and if there are too many bad records, an invalid error is</span>
+<span class="sd"> returned in the job result. Only applicable when soure_format is CSV.</span>
+<span class="sd"> :type allow_jagged_rows: bool</span>
+<span class="sd"> :param schema_update_options: Allows the schema of the desitination</span>
+<span class="sd"> table to be updated as a side effect of the load job.</span>
+<span class="sd"> :type schema_update_options: list</span>
+<span class="sd"> :param src_fmt_configs: configure optional fields specific to the source format</span>
+<span class="sd"> :type src_fmt_configs: dict</span>
<span class="sd"> """</span>
+
+ <span class="c1"># bigquery only allows certain source formats</span>
+ <span class="c1"># we check to make sure the passed source format is valid</span>
+ <span class="c1"># if it's not, we raise a ValueError</span>
+ <span class="c1"># Refer to this link for more details:</span>
+ <span class="c1"># https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query.tableDefinitions.(key).sourceFormat</span>
+ <span class="n">source_format</span> <span class="o">=</span> <span class="n">source_format</span><span class="o">.</span><span class="n">upper</span><span class="p">()</span>
+ <span class="n">allowed_formats</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"CSV"</span><span class="p">,</span> <span class="s2">"NEWLINE_DELIMITED_JSON"</span><span class="p">,</span> <span class="s2">"AVRO"</span><span class="p">,</span> <span class="s2">"GOOGLE_SHEETS"</span><span class="p">,</span> <span class="s2">"DATASTORE_BACKUP"</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">source_format</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">allowed_formats</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"</span><span class="si">{0}</span><span class="s2"> is not a valid source format. "</span>
+ <span class="s2">"Please use one of the following types: </span><span class="si">{1}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">source_format</span><span class="p">,</span> <span class="n">allowed_formats</span><span class="p">))</span>
+
+ <span class="c1"># bigquery also allows you to define how you want a table's schema to change</span>
+ <span class="c1"># as a side effect of a load</span>
+ <span class="c1"># for more details:</span>
+ <span class="c1"># https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.load.schemaUpdateOptions</span>
+ <span class="n">allowed_schema_update_options</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="s1">'ALLOW_FIELD_ADDITION'</span><span class="p">,</span>
+ <span class="s2">"ALLOW_FIELD_RELAXATION"</span>
+ <span class="p">]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">set</span><span class="p">(</span><span class="n">allowed_schema_update_options</span><span class="p">)</span><span class="o">.</span><span class="n">issuperset</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">schema_update_options</span><span class="p">)):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s2">"</span><span class="si">{0}</span><span class="s2"> contains invalid schema update options. "</span>
+ <span class="s2">"Please only use one or more of the following options: </span><span class="si">{1}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">schema_update_options</span><span class="p">,</span> <span class="n">allowed_schema_update_options</span><span class="p">)</span>
+ <span class="p">)</span>
+
<span class="n">destination_project</span><span class="p">,</span> <span class="n">destination_dataset</span><span class="p">,</span> <span class="n">destination_table</span> <span class="o">=</span> \
- <span class="bp">self</span><span class="o">.</span><span class="n">_split_project_dataset_table_input</span><span class="p">(</span>
- <span class="s1">'destination_project_dataset_table'</span><span class="p">,</span> <span class="n">destination_project_dataset_table</span><span class="p">)</span>
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">var_name</span><span class="o">=</span><span class="s1">'destination_project_dataset_table'</span><span class="p">)</span>
<span class="n">configuration</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'load'</span><span class="p">:</span> <span class="p">{</span>
@@ -521,43 +650,62 @@
<span class="s1">'datasetId'</span><span class="p">:</span> <span class="n">destination_dataset</span><span class="p">,</span>
<span class="s1">'tableId'</span><span class="p">:</span> <span class="n">destination_table</span><span class="p">,</span>
<span class="p">},</span>
- <span class="s1">'schema'</span><span class="p">:</span> <span class="p">{</span>
- <span class="s1">'fields'</span><span class="p">:</span> <span class="n">schema_fields</span>
- <span class="p">},</span>
<span class="s1">'sourceFormat'</span><span class="p">:</span> <span class="n">source_format</span><span class="p">,</span>
<span class="s1">'sourceUris'</span><span class="p">:</span> <span class="n">source_uris</span><span class="p">,</span>
<span class="s1">'writeDisposition'</span><span class="p">:</span> <span class="n">write_disposition</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">}</span>
+ <span class="k">if</span> <span class="n">schema_fields</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'schema'</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'fields'</span><span class="p">:</span> <span class="n">schema_fields</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="n">schema_update_options</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">write_disposition</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">"WRITE_APPEND"</span><span class="p">,</span> <span class="s2">"WRITE_TRUNCATE"</span><span class="p">]:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s2">"schema_update_options is only "</span>
+ <span class="s2">"allowed if write_disposition is "</span>
+ <span class="s2">"'WRITE_APPEND' or 'WRITE_TRUNCATE'."</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Adding experimental "</span>
+ <span class="s2">"'schemaUpdateOptions': </span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">schema_update_options</span><span class="p">)</span>
+ <span class="p">)</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'schemaUpdateOptions'</span><span class="p">]</span> <span class="o">=</span> <span class="n">schema_update_options</span>
+
+ <span class="k">if</span> <span class="n">max_bad_records</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'maxBadRecords'</span><span class="p">]</span> <span class="o">=</span> <span class="n">max_bad_records</span>
+
+ <span class="c1"># if following fields are not specified in src_fmt_configs,</span>
+ <span class="c1"># honor the top-level params for backward-compatibility</span>
+ <span class="k">if</span> <span class="s1">'skipLeadingRows'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">src_fmt_configs</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'skipLeadingRows'</span><span class="p">]</span> <span class="o">=</span> <span class="n">skip_leading_rows</span>
+ <span class="k">if</span> <span class="s1">'fieldDelimiter'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">src_fmt_configs</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'fieldDelimiter'</span><span class="p">]</span> <span class="o">=</span> <span class="n">field_delimiter</span>
+ <span class="k">if</span> <span class="n">quote_character</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'quote'</span><span class="p">]</span> <span class="o">=</span> <span class="n">quote_character</span>
+ <span class="k">if</span> <span class="n">allow_quoted_newlines</span><span class="p">:</span>
+ <span class="n">src_fmt_configs</span><span class="p">[</span><span class="s1">'allowQuotedNewlines'</span><span class="p">]</span> <span class="o">=</span> <span class="n">allow_quoted_newlines</span>
+
+ <span class="n">src_fmt_to_configs_mapping</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'CSV'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'allowJaggedRows'</span><span class="p">,</span> <span class="s1">'allowQuotedNewlines'</span><span class="p">,</span> <span class="s1">'autodetect'</span><span class="p">,</span>
+ <span class="s1">'fieldDelimiter'</span><span class="p">,</span> <span class="s1">'skipLeadingRows'</span><span class="p">,</span> <span class="s1">'ignoreUnknownValues'</span><span class="p">,</span>
+ <span class="s1">'nullMarker'</span><span class="p">,</span> <span class="s1">'quote'</span><span class="p">],</span>
+ <span class="s1">'DATASTORE_BACKUP'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'projectionFields'</span><span class="p">],</span>
+ <span class="s1">'NEWLINE_DELIMITED_JSON'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'autodetect'</span><span class="p">,</span> <span class="s1">'ignoreUnknownValues'</span><span class="p">],</span>
+ <span class="s1">'AVRO'</span><span class="p">:</span> <span class="p">[],</span>
+ <span class="p">}</span>
+ <span class="n">valid_configs</span> <span class="o">=</span> <span class="n">src_fmt_to_configs_mapping</span><span class="p">[</span><span class="n">source_format</span><span class="p">]</span>
+ <span class="n">src_fmt_configs</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">src_fmt_configs</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">valid_configs</span><span class="p">}</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">]</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">src_fmt_configs</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">source_format</span> <span class="o">==</span> <span class="s1">'CSV'</span><span class="p">:</span>
- <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'skipLeadingRows'</span><span class="p">]</span> <span class="o">=</span> <span class="n">skip_leading_rows</span>
- <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'fieldDelimiter'</span><span class="p">]</span> <span class="o">=</span> <span class="n">field_delimiter</span>
+ <span class="k">if</span> <span class="n">allow_jagged_rows</span><span class="p">:</span>
+ <span class="n">configuration</span><span class="p">[</span><span class="s1">'load'</span><span class="p">][</span><span class="s1">'allowJaggedRows'</span><span class="p">]</span> <span class="o">=</span> <span class="n">allow_jagged_rows</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_with_configuration</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">_split_project_dataset_table_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">var_name</span><span class="p">,</span> <span class="n">project_dataset_table</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> :param var_name: the name of the variable input, for logging and erroring purposes.</span>
-<span class="sd"> :type var_name: str</span>
-<span class="sd"> :param project_dataset_table: input string in (<project>.)<dataset>.<project> format.</span>
-<span class="sd"> if project is not included in the string, self.project_id will be returned in the tuple.</span>
-<span class="sd"> :type project_dataset_table: str</span>
-<span class="sd"> :return: (project, dataset, table) tuple</span>
-<span class="sd"> """</span>
- <span class="n">table_split</span> <span class="o">=</span> <span class="n">project_dataset_table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
- <span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">table_split</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> <span class="ow">or</span> <span class="nb">len</span><span class="p">(</span><span class="n">table_split</span><span class="p">)</span> <span class="o">==</span> <span class="mi">3</span><span class="p">,</span> <span class="p">(</span>
- <span class="s1">'Expected {var} in the format of (<project.)<dataset>.<table>, '</span>
- <span class="s1">'got {input}'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">var</span><span class="o">=</span><span class="n">var_name</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">project_dataset_table</span><span class="p">)</span>
-
- <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">table_split</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'project not included in {var}: {input}; using project "{project}"'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">var</span><span class="o">=</span><span class="n">var_name</span><span class="p">,</span> <span class="nb">input</span><span class="o">=</span><span class="n">project_dataset_table</span><span class="p">,</span> <span class="n">project</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">))</span>
- <span class="n">dataset</span><span class="p">,</span> <span class="n">table</span> <span class="o">=</span> <span class="n">table_split</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">dataset</span><span class="p">,</span> <span class="n">table</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">project</span><span class="p">,</span> <span class="n">dataset</span><span class="p">,</span> <span class="n">table</span> <span class="o">=</span> <span class="n">table_split</span>
- <span class="k">return</span> <span class="n">project</span><span class="p">,</span> <span class="n">dataset</span><span class="p">,</span> <span class="n">table</span>
-
<span class="k">def</span> <span class="nf">run_with_configuration</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">configuration</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Executes a BigQuery SQL query. See here:</span>
@@ -581,18 +729,32 @@
<span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">job_data</span><span class="p">)</span> \
<span class="o">.</span><span class="n">execute</span><span class="p">()</span>
<span class="n">job_id</span> <span class="o">=</span> <span class="n">query_reply</span><span class="p">[</span><span class="s1">'jobReference'</span><span class="p">][</span><span class="s1">'jobId'</span><span class="p">]</span>
- <span class="n">job</span> <span class="o">=</span> <span class="n">jobs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">jobId</span><span class="o">=</span><span class="n">job_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
<span class="c1"># Wait for query to finish.</span>
- <span class="k">while</span> <span class="ow">not</span> <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'state'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'DONE'</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Waiting for job to complete: </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
- <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
- <span class="n">job</span> <span class="o">=</span> <span class="n">jobs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">jobId</span><span class="o">=</span><span class="n">job_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
-
- <span class="c1"># Check if job had errors.</span>
- <span class="k">if</span> <span class="s1">'errorResult'</span> <span class="ow">in</span> <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]:</span>
- <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
- <span class="s1">'BigQuery job failed. Final error was: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'errorResult'</span><span class="p">])</span>
+ <span class="n">keep_polling_job</span> <span class="o">=</span> <span class="kc">True</span>
+ <span class="k">while</span> <span class="p">(</span><span class="n">keep_polling_job</span><span class="p">):</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">job</span> <span class="o">=</span> <span class="n">jobs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">jobId</span><span class="o">=</span><span class="n">job_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'state'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'DONE'</span><span class="p">):</span>
+ <span class="n">keep_polling_job</span> <span class="o">=</span> <span class="kc">False</span>
+ <span class="c1"># Check if job had errors.</span>
+ <span class="k">if</span> <span class="s1">'errorResult'</span> <span class="ow">in</span> <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'BigQuery job failed. Final error was: </span><span class="si">{}</span><span class="s1">. The job was: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">job</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'errorResult'</span><span class="p">],</span> <span class="n">job</span>
+ <span class="p">)</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Waiting for job to complete : </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+
+ <span class="k">except</span> <span class="n">HttpError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">err</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">500</span><span class="p">,</span> <span class="mi">503</span><span class="p">]:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1">: Retryable error, waiting for job to complete: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">err</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'BigQuery job status check failed. Final error was: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">err</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span><span class="p">)</span>
<span class="k">return</span> <span class="n">job_id</span>
@@ -611,7 +773,7 @@
<span class="k">return</span> <span class="n">tables_resource</span><span class="p">[</span><span class="s1">'schema'</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get_tabledata</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">,</span>
- <span class="n">max_results</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">page_token</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">start_index</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="n">max_results</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">page_token</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">start_index</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get the data of a given dataset.table.</span>
<span class="sd"> see https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list</span>
@@ -639,15 +801,56 @@
<span class="o">.</span><span class="n">execute</span><span class="p">()</span>
<span class="p">)</span>
- <span class="k">def</span> <span class="nf">run_table_upsert</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_resource</span><span class="p">,</span> <span class="n">project_id</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">run_table_delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">deletion_dataset_table</span><span class="p">,</span> <span class="n">ignore_if_missing</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Delete an existing table from the dataset;</span>
+<span class="sd"> If the table does not exist, return an error unless ignore_if_missing</span>
+<span class="sd"> is set to True.</span>
+
+<span class="sd"> :param deletion_dataset_table: A dotted</span>
+<span class="sd"> (<project>.|<project>:)<dataset>.<table> that indicates which table</span>
+<span class="sd"> will be deleted.</span>
+<span class="sd"> :type deletion_dataset_table: str</span>
+<span class="sd"> :param ignore_if_missing: if True, then return success even if the</span>
+<span class="sd"> requested table does not exist.</span>
+<span class="sd"> :type ignore_if_missing: boolean</span>
+<span class="sd"> :return:</span>
+<span class="sd"> """</span>
+
+ <span class="k">assert</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">deletion_dataset_table</span><span class="p">,</span> <span class="p">(</span>
+ <span class="s1">'Expected deletion_dataset_table in the format of '</span>
+ <span class="s1">'<dataset>.<table>. Got: </span><span class="si">{}</span><span class="s1">'</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">deletion_dataset_table</span><span class="p">)</span>
+ <span class="n">deletion_project</span><span class="p">,</span> <span class="n">deletion_dataset</span><span class="p">,</span> <span class="n">deletion_table</span> <span class="o">=</span> \
+ <span class="n">_split_tablename</span><span class="p">(</span><span class="n">table_input</span><span class="o">=</span><span class="n">deletion_dataset_table</span><span class="p">,</span>
+ <span class="n">default_project_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">)</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">tables_resource</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">deletion_project</span><span class="p">,</span>
+ <span class="n">datasetId</span><span class="o">=</span><span class="n">deletion_dataset</span><span class="p">,</span>
+ <span class="n">tableId</span><span class="o">=</span><span class="n">deletion_table</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Deleted table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1">.'</span><span class="p">,</span>
+ <span class="n">deletion_project</span><span class="p">,</span> <span class="n">deletion_dataset</span><span class="p">,</span> <span class="n">deletion_table</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">HttpError</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">ignore_if_missing</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'Table deletion failed. Table does not exist.'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Table does not exist. Skipping.'</span><span class="p">)</span>
+
+
+ <span class="k">def</span> <span class="nf">run_table_upsert</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_resource</span><span class="p">,</span> <span class="n">project_id</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> creates a new, empty table in the dataset;</span>
<span class="sd"> If the table already exists, update the existing table.</span>
<span class="sd"> Since BigQuery does not natively allow table upserts, this is not an</span>
<span class="sd"> atomic operation.</span>
+
<span class="sd"> :param dataset_id: the dataset to upsert the table into.</span>
<span class="sd"> :type dataset_id: str</span>
-<span class="sd"> :param table_resource: a table resource. see https://cloud.google.com/bigquery/docs/reference/v2/tables#resource</span>
+<span class="sd"> :param table_resource: a table resource. see</span>
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/tables#resource</span>
<span class="sd"> :type table_resource: dict</span>
<span class="sd"> :param project_id: the project to upsert the table into. If None,</span>
<span class="sd"> project will be self.project_id.</span>
@@ -655,15 +858,17 @@
<span class="sd"> """</span>
<span class="c1"># check to see if the table exists</span>
<span class="n">table_id</span> <span class="o">=</span> <span class="n">table_resource</span><span class="p">[</span><span class="s1">'tableReference'</span><span class="p">][</span><span class="s1">'tableId'</span><span class="p">]</span>
- <span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span> <span class="k">if</span> <span class="n">project_id</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="k">else</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span>
+ <span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span> <span class="k">if</span> <span class="n">project_id</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span>
<span class="n">tables_list_resp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
<span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
- <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">for</span> <span class="n">table</span> <span class="ow">in</span> <span class="n">tables_list_resp</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tables'</span><span class="p">,</span> <span class="p">[]):</span>
<span class="k">if</span> <span class="n">table</span><span class="p">[</span><span class="s1">'tableReference'</span><span class="p">][</span><span class="s1">'tableId'</span><span class="p">]</span> <span class="o">==</span> <span class="n">table_id</span><span class="p">:</span>
<span class="c1"># found the table, do update</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> exists, updating.'</span><span class="p">,</span>
- <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> exists, updating.'</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span>
+ <span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
<span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span>
<span class="n">tableId</span><span class="o">=</span><span class="n">table_id</span><span class="p">,</span>
@@ -678,8 +883,10 @@
<span class="c1"># If there is no next page, then the table doesn't exist.</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># do insert</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> does not exist. creating.'</span><span class="p">,</span>
- <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> does not exist. creating.'</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">dataset_id</span><span class="p">,</span> <span class="n">table_id</span>
+ <span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">tables</span><span class="p">()</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">projectId</span><span class="o">=</span><span class="n">project_id</span><span class="p">,</span>
<span class="n">datasetId</span><span class="o">=</span><span class="n">dataset_id</span><span class="p">,</span>
<span class="n">body</span><span class="o">=</span><span class="n">table_resource</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
@@ -688,12 +895,13 @@
<span class="n">source_dataset</span><span class="p">,</span>
<span class="n">view_dataset</span><span class="p">,</span>
<span class="n">view_table</span><span class="p">,</span>
- <span class="n">source_project</span> <span class="o">=</span> <span class="bp">None</span><span class="p">,</span>
- <span class="n">view_project</span> <span class="o">=</span> <span class="bp">None</span><span class="p">):</span>
+ <span class="n">source_project</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
+ <span class="n">view_project</span> <span class="o">=</span> <span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Grant authorized view access of a dataset to a view table.</span>
<span class="sd"> If this view has already been granted access to the dataset, do nothing.</span>
<span class="sd"> This method is not atomic. Running it may clobber a simultaneous update.</span>
+
<span class="sd"> :param source_dataset: the source dataset</span>
<span class="sd"> :type source_dataset: str</span>
<span class="sd"> :param view_dataset: the dataset that the view is in</span>
@@ -723,18 +931,20 @@
<span class="s1">'tableId'</span><span class="p">:</span> <span class="n">view_table</span><span class="p">}}</span>
<span class="c1"># check to see if the view we want to add already exists.</span>
<span class="k">if</span> <span class="n">view_access</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">access</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'granting table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> authorized view access to </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1"> dataset.'</span><span class="p">,</span>
- <span class="n">view_project</span><span class="p">,</span> <span class="n">view_dataset</span><span class="p">,</span> <span class="n">view_table</span><span class="p">,</span>
- <span class="n">source_project</span><span class="p">,</span> <span class="n">source_dataset</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Granting table </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1"> authorized view access to </span><span class="si">%s</span><span class="s1">:</span><span class="si">%s</span><span class="s1"> dataset.'</span><span class="p">,</span>
+ <span class="n">view_proje
<TRUNCATED>
[34/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/executors/mesos_executor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/executors/mesos_executor.html b/_modules/airflow/contrib/executors/mesos_executor.html
index 311d1a8..f23df55 100644
--- a/_modules/airflow/contrib/executors/mesos_executor.html
+++ b/_modules/airflow/contrib/executors/mesos_executor.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
@@ -177,9 +182,12 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">future</span> <span class="k">import</span> <span class="n">standard_library</span>
+
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+<span class="kn">from</span> <span class="nn">airflow.www.utils</span> <span class="k">import</span> <span class="n">LoginMixin</span>
+
<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">queue</span> <span class="k">import</span> <span class="n">Queue</span>
<span class="kn">import</span> <span class="nn">mesos.interface</span>
@@ -205,7 +213,7 @@
<span class="c1"># AirflowMesosScheduler, implements Mesos Scheduler interface</span>
<span class="c1"># To schedule airflow jobs on mesos</span>
-<span class="k">class</span> <span class="nc">AirflowMesosScheduler</span><span class="p">(</span><span class="n">mesos</span><span class="o">.</span><span class="n">interface</span><span class="o">.</span><span class="n">Scheduler</span><span class="p">):</span>
+<span class="k">class</span> <span class="nc">AirflowMesosScheduler</span><span class="p">(</span><span class="n">mesos</span><span class="o">.</span><span class="n">interface</span><span class="o">.</span><span class="n">Scheduler</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Airflow Mesos scheduler implements mesos scheduler interface</span>
<span class="sd"> to schedule airflow tasks on mesos.</span>
@@ -213,7 +221,6 @@
<span class="sd"> 'airflow run <dag_id> <task_instance_id> <start_date> --local -p=<pickle>'</span>
<span class="sd"> to run on a mesos slave.</span>
<span class="sd"> """</span>
-
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">task_queue</span><span class="p">,</span>
<span class="n">result_queue</span><span class="p">,</span>
@@ -227,7 +234,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">task_key_map</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">def</span> <span class="nf">registered</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">frameworkId</span><span class="p">,</span> <span class="n">masterInfo</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler registered to mesos with framework ID </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">frameworkId</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler registered to Mesos with framework ID </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">frameworkId</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'CHECKPOINT'</span><span class="p">)</span> <span class="ow">and</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'FAILOVER_TIMEOUT'</span><span class="p">):</span>
<span class="c1"># Import here to work around a circular import error</span>
@@ -248,25 +255,25 @@
<span class="n">Session</span><span class="o">.</span><span class="n">remove</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">reregistered</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">masterInfo</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler re-registered to mesos"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler re-registered to mesos"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">disconnected</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler disconnected from mesos"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler disconnected from mesos"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">offerRescinded</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">offerId</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler offer </span><span class="si">%s</span><span class="s2"> rescinded"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">offerId</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler offer </span><span class="si">%s</span><span class="s2"> rescinded"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">offerId</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">frameworkMessage</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">executorId</span><span class="p">,</span> <span class="n">slaveId</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler received framework message </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"AirflowScheduler received framework message </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">executorLost</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">executorId</span><span class="p">,</span> <span class="n">slaveId</span><span class="p">,</span> <span class="n">status</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"AirflowScheduler executor </span><span class="si">%s</span><span class="s2"> lost"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">executorId</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"AirflowScheduler executor </span><span class="si">%s</span><span class="s2"> lost"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">executorId</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">slaveLost</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">slaveId</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"AirflowScheduler slave </span><span class="si">%s</span><span class="s2"> lost"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">slaveId</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"AirflowScheduler slave </span><span class="si">%s</span><span class="s2"> lost"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">slaveId</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">error</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">message</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"AirflowScheduler driver aborted </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"AirflowScheduler driver aborted </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"AirflowScheduler driver aborted </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">message</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">resourceOffers</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">offers</span><span class="p">):</span>
@@ -280,7 +287,7 @@
<span class="k">elif</span> <span class="n">resource</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="s2">"mem"</span><span class="p">:</span>
<span class="n">offerMem</span> <span class="o">+=</span> <span class="n">resource</span><span class="o">.</span><span class="n">scalar</span><span class="o">.</span><span class="n">value</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Received offer </span><span class="si">%s</span><span class="s2"> with cpus: </span><span class="si">%s</span><span class="s2"> and mem: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">offer</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">offerCpus</span><span class="p">,</span> <span class="n">offerMem</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Received offer </span><span class="si">%s</span><span class="s2"> with cpus: </span><span class="si">%s</span><span class="s2"> and mem: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">offer</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">offerCpus</span><span class="p">,</span> <span class="n">offerMem</span><span class="p">)</span>
<span class="n">remainingCpus</span> <span class="o">=</span> <span class="n">offerCpus</span>
<span class="n">remainingMem</span> <span class="o">=</span> <span class="n">offerMem</span>
@@ -293,7 +300,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">task_counter</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_key_map</span><span class="p">[</span><span class="nb">str</span><span class="p">(</span><span class="n">tid</span><span class="p">)]</span> <span class="o">=</span> <span class="n">key</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Launching task </span><span class="si">%d</span><span class="s2"> using offer </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">tid</span><span class="p">,</span> <span class="n">offer</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Launching task </span><span class="si">%d</span><span class="s2"> using offer </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">tid</span><span class="p">,</span> <span class="n">offer</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="n">task</span> <span class="o">=</span> <span class="n">mesos_pb2</span><span class="o">.</span><span class="n">TaskInfo</span><span class="p">()</span>
<span class="n">task</span><span class="o">.</span><span class="n">task_id</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">tid</span><span class="p">)</span>
@@ -323,15 +330,17 @@
<span class="n">driver</span><span class="o">.</span><span class="n">launchTasks</span><span class="p">(</span><span class="n">offer</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="n">tasks</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">statusUpdate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">driver</span><span class="p">,</span> <span class="n">update</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Task </span><span class="si">%s</span><span class="s2"> is in state </span><span class="si">%s</span><span class="s2">, data </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
- <span class="n">update</span><span class="o">.</span><span class="n">task_id</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">mesos_pb2</span><span class="o">.</span><span class="n">TaskState</span><span class="o">.</span><span class="n">Name</span><span class="p">(</span><span class="n">update</span><span class="o">.</span><span class="n">state</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">update</span><span class="o">.</span><span class="n">data</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Task </span><span class="si">%s</span><span class="s2"> is in state </span><span class="si">%s</span><span class="s2">, data </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
+ <span class="n">update</span><span class="o">.</span><span class="n">task_id</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="n">mesos_pb2</span><span class="o">.</span><span class="n">TaskState</span><span class="o">.</span><span class="n">Name</span><span class="p">(</span><span class="n">update</span><span class="o">.</span><span class="n">state</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">update</span><span class="o">.</span><span class="n">data</span><span class="p">)</span>
+ <span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_key_map</span><span class="p">[</span><span class="n">update</span><span class="o">.</span><span class="n">task_id</span><span class="o">.</span><span class="n">value</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">KeyError</span><span class="p">:</span>
<span class="c1"># The map may not contain an item if the framework re-registered after a failover.</span>
<span class="c1"># Discard these tasks.</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span><span class="s2">"Unrecognised task key </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">update</span><span class="o">.</span><span class="n">task_id</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Unrecognised task key </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">update</span><span class="o">.</span><span class="n">task_id</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
<span class="k">return</span>
<span class="k">if</span> <span class="n">update</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="n">mesos_pb2</span><span class="o">.</span><span class="n">TASK_FINISHED</span><span class="p">:</span>
@@ -345,7 +354,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
-<div class="viewcode-block" id="MesosExecutor"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.executors.mesos_executor.MesosExecutor">[docs]</a><span class="k">class</span> <span class="nc">MesosExecutor</span><span class="p">(</span><span class="n">BaseExecutor</span><span class="p">):</span>
+<div class="viewcode-block" id="MesosExecutor"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.executors.mesos_executor.MesosExecutor">[docs]</a><span class="k">class</span> <span class="nc">MesosExecutor</span><span class="p">(</span><span class="n">BaseExecutor</span><span class="p">,</span> <span class="n">LoginMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> MesosExecutor allows distributing the execution of task</span>
<span class="sd"> instances to multiple mesos workers.</span>
@@ -356,7 +365,6 @@
<span class="sd"> elastic distributed systems to easily be built and run effectively.</span>
<span class="sd"> See http://mesos.apache.org/</span>
<span class="sd"> """</span>
-
<span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>
@@ -364,7 +372,7 @@
<span class="n">framework</span><span class="o">.</span><span class="n">user</span> <span class="o">=</span> <span class="s1">''</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'MASTER'</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Expecting mesos master URL for mesos executor"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Expecting mesos master URL for mesos executor"</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"mesos.master not provided for mesos executor"</span><span class="p">)</span>
<span class="n">master</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'MASTER'</span><span class="p">)</span>
@@ -400,17 +408,19 @@
<span class="k">else</span><span class="p">:</span>
<span class="n">framework</span><span class="o">.</span><span class="n">checkpoint</span> <span class="o">=</span> <span class="kc">False</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'MesosFramework master : </span><span class="si">%s</span><span class="s1">, name : </span><span class="si">%s</span><span class="s1">, cpu : </span><span class="si">%s</span><span class="s1">, mem : </span><span class="si">%s</span><span class="s1">, checkpoint : </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
- <span class="n">master</span><span class="p">,</span> <span class="n">framework</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">task_cpu</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">task_memory</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">framework</span><span class="o">.</span><span class="n">checkpoint</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'MesosFramework master : </span><span class="si">%s</span><span class="s1">, name : </span><span class="si">%s</span><span class="s1">, cpu : </span><span class="si">%s</span><span class="s1">, mem : </span><span class="si">%s</span><span class="s1">, checkpoint : </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">master</span><span class="p">,</span> <span class="n">framework</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">task_cpu</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">task_memory</span><span class="p">),</span> <span class="nb">str</span><span class="p">(</span><span class="n">framework</span><span class="o">.</span><span class="n">checkpoint</span><span class="p">)</span>
+ <span class="p">)</span>
<span class="n">implicit_acknowledgements</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'AUTHENTICATE'</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'DEFAULT_PRINCIPAL'</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Expecting authentication principal in the environment"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Expecting authentication principal in the environment"</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"mesos.default_principal not provided in authenticated mode"</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'mesos'</span><span class="p">,</span> <span class="s1">'DEFAULT_SECRET'</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Expecting authentication secret in the environment"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Expecting authentication secret in the environment"</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"mesos.default_secret not provided in authenticated mode"</span><span class="p">)</span>
<span class="n">credential</span> <span class="o">=</span> <span class="n">mesos_pb2</span><span class="o">.</span><span class="n">Credential</span><span class="p">()</span>
[24/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/executors/local_executor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/executors/local_executor.html b/_modules/airflow/executors/local_executor.html
index 2e35470..cfbf241 100644
--- a/_modules/airflow/executors/local_executor.html
+++ b/_modules/airflow/executors/local_executor.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
@@ -184,16 +189,15 @@
<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">configuration</span>
<span class="kn">from</span> <span class="nn">airflow.executors.base_executor</span> <span class="k">import</span> <span class="n">BaseExecutor</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="k">import</span> <span class="n">State</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.logging</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
<span class="n">PARALLELISM</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'PARALLELISM'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">LocalWorker</span><span class="p">(</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
-
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_queue</span><span class="p">,</span> <span class="n">result_queue</span><span class="p">):</span>
- <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
+ <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span> <span class="o">=</span> <span class="n">task_queue</span>
<span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span> <span class="o">=</span> <span class="n">result_queue</span>
<span class="bp">self</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="kc">True</span>
@@ -205,15 +209,15 @@
<span class="c1"># Received poison pill, no more tasks to run</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
<span class="k">break</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"</span><span class="si">{}</span><span class="s2"> running </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="n">command</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"</span><span class="si">%s</span><span class="s2"> running </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span> <span class="n">command</span><span class="p">)</span>
<span class="n">command</span> <span class="o">=</span> <span class="s2">"exec bash -c '</span><span class="si">{0}</span><span class="s2">'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span>
<span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">FAILED</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"failed to execute task </span><span class="si">{}</span><span class="s2">:"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Failed to execute task </span><span class="si">%s</span><span class="s2">."</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="c1"># TODO: Why is this commented out?</span>
<span class="c1"># raise e</span>
<span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">state</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
@@ -232,7 +236,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">workers</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">LocalWorker</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">parallelism</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">parallelism</span><span class="p">)</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">workers</span><span class="p">:</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/executors/sequential_executor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/executors/sequential_executor.html b/_modules/airflow/executors/sequential_executor.html
index d5644c2..2c7dbac 100644
--- a/_modules/airflow/executors/sequential_executor.html
+++ b/_modules/airflow/executors/sequential_executor.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
@@ -193,7 +198,7 @@
<span class="sd"> SequentialExecutor alongside sqlite as you first install it.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SequentialExecutor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SequentialExecutor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">def</span> <span class="nf">execute_async</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">command</span><span class="p">,</span> <span class="n">queue</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
@@ -201,14 +206,14 @@
<span class="k">def</span> <span class="nf">sync</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">command</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing command: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">command</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing command: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">command</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">change_state</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">)</span>
<span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">change_state</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">State</span><span class="o">.</span><span class="n">FAILED</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Failed to execute task </span><span class="si">{}</span><span class="s2">:"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Failed to execute task </span><span class="si">%s</span><span class="s2">."</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span> <span class="o">=</span> <span class="p">[]</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/hooks/S3_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/hooks/S3_hook.html b/_modules/airflow/hooks/S3_hook.html
new file mode 100644
index 0000000..f1c0617
--- /dev/null
+++ b/_modules/airflow/hooks/S3_hook.html
@@ -0,0 +1,489 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.hooks.S3_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.hooks.S3_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.hooks.S3_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.aws_hook</span> <span class="k">import</span> <span class="n">AwsHook</span>
+
+<span class="kn">from</span> <span class="nn">six</span> <span class="k">import</span> <span class="n">BytesIO</span>
+<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
+<span class="kn">import</span> <span class="nn">re</span>
+<span class="kn">import</span> <span class="nn">fnmatch</span>
+
+<div class="viewcode-block" id="S3Hook"><a class="viewcode-back" href="../../../integration.html#airflow.hooks.S3_hook.S3Hook">[docs]</a><span class="k">class</span> <span class="nc">S3Hook</span><span class="p">(</span><span class="n">AwsHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with AWS S3, using the boto3 library.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_client_type</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
+
+ <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">parse_s3_url</span><span class="p">(</span><span class="n">s3url</span><span class="p">):</span>
+ <span class="n">parsed_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">s3url</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Please provide a bucket_name'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">bucket_name</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span>
+ <span class="n">key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">check_for_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Check if bucket_name exists.</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">head_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+
+ <span class="k">def</span> <span class="nf">get_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a boto3.S3.Bucket object</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="n">s3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_resource_type</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">s3</span><span class="o">.</span><span class="n">Bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">check_for_prefix</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that a prefix exists in a bucket</span>
+<span class="sd"> """</span>
+ <span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span> <span class="o">+</span> <span class="n">delimiter</span> <span class="k">if</span> <span class="n">prefix</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">!=</span> <span class="n">delimiter</span> <span class="k">else</span> <span class="n">prefix</span>
+ <span class="n">prefix_split</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="sa">r</span><span class="s1">'(\w+[</span><span class="si">{d}</span><span class="s1">])$'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">d</span><span class="o">=</span><span class="n">delimiter</span><span class="p">),</span> <span class="n">prefix</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="n">previous_level</span> <span class="o">=</span> <span class="n">prefix_split</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">plist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_prefixes</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">previous_level</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">False</span> <span class="k">if</span> <span class="n">plist</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">prefix</span> <span class="ow">in</span> <span class="n">plist</span>
+
+ <span class="k">def</span> <span class="nf">list_prefixes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lists prefixes in a bucket under prefix</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param prefix: a key prefix</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param delimiter: the delimiter marks key hierarchy.</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">list_objects_v2</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">Prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="n">Delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="p">[</span><span class="n">p</span><span class="p">[</span><span class="s1">'Prefix'</span><span class="p">]</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'CommonPrefixes'</span><span class="p">]]</span> <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'CommonPrefixes'</span><span class="p">)</span> <span class="k">else</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">list_keys</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lists keys in a bucket under prefix and not containing delimiter</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param prefix: a key prefix</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param delimiter: the delimiter marks key hierarchy.</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">list_objects_v2</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">Prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="n">Delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="p">[</span><span class="n">k</span><span class="p">[</span><span class="s1">'Key'</span><span class="p">]</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'Contents'</span><span class="p">]]</span> <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Contents'</span><span class="p">)</span> <span class="k">else</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks if a key exists in a bucket</span>
+
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which the file is stored</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">head_object</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">Key</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+
+ <span class="k">def</span> <span class="nf">get_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a boto3.s3.Object</span>
+
+<span class="sd"> :param key: the path to the key</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+
+ <span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_resource_type</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
+ <span class="n">obj</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">obj</span>
+
+ <span class="k">def</span> <span class="nf">read_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Reads a key from S3</span>
+
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which the file is stored</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">obj</span><span class="o">.</span><span class="n">get</span><span class="p">()[</span><span class="s1">'Body'</span><span class="p">]</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that a key matching a wildcard expression exists in a bucket</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_wildcard_key</span><span class="p">(</span><span class="n">wildcard_key</span><span class="o">=</span><span class="n">wildcard_key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a boto3.s3.Object object matching the regular expression</span>
+
+<span class="sd"> :param regex_key: the path to the key</span>
+<span class="sd"> :type regex_key: str</span>
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">wildcard_key</span><span class="p">)</span>
+
+ <span class="n">prefix</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="sa">r</span><span class="s1">'[*]'</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">klist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_keys</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">klist</span><span class="p">:</span>
+ <span class="n">key_matches</span> <span class="o">=</span> <span class="p">[</span><span class="n">k</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">klist</span> <span class="k">if</span> <span class="n">fnmatch</span><span class="o">.</span><span class="n">fnmatch</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)]</span>
+ <span class="k">if</span> <span class="n">key_matches</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key_matches</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">bucket_name</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">filename</span><span class="p">,</span>
+ <span class="n">key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">encrypt</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a local file to S3</span>
+
+<span class="sd"> :param filename: name of the file to load.</span>
+<span class="sd"> :type filename: str</span>
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which to store the file</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param replace: A flag to decide whether or not to overwrite the key</span>
+<span class="sd"> if it already exists. If replace is False and the key exists, an</span>
+<span class="sd"> error will be raised.</span>
+<span class="sd"> :type replace: bool</span>
+<span class="sd"> :param encrypt: If True, the file will be encrypted on the server-side</span>
+<span class="sd"> by S3 and will be stored in an encrypted form while at rest in S3.</span>
+<span class="sd"> :type encrypt: bool</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key </span><span class="si">{key}</span><span class="s2"> already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">key</span><span class="p">))</span>
+
+ <span class="n">extra_args</span><span class="o">=</span><span class="p">{}</span>
+ <span class="k">if</span> <span class="n">encrypt</span><span class="p">:</span>
+ <span class="n">extra_args</span><span class="p">[</span><span class="s1">'ServerSideEncryption'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"AES256"</span>
+
+ <span class="n">client</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">client</span><span class="o">.</span><span class="n">upload_file</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">ExtraArgs</span><span class="o">=</span><span class="n">extra_args</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">load_string</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">string_data</span><span class="p">,</span>
+ <span class="n">key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">encrypt</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a string to S3</span>
+
+<span class="sd"> This is provided as a convenience to drop a string in S3. It uses the</span>
+<span class="sd"> boto infrastructure to ship a file to s3. </span>
+
+<span class="sd"> :param string_data: string to set as content for the key.</span>
+<span class="sd"> :type string_data: str</span>
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which to store the file</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param replace: A flag to decide whether or not to overwrite the key</span>
+<span class="sd"> if it already exists</span>
+<span class="sd"> :type replace: bool</span>
+<span class="sd"> :param encrypt: If True, the file will be encrypted on the server-side</span>
+<span class="sd"> by S3 and will be stored in an encrypted form while at rest in S3.</span>
+<span class="sd"> :type encrypt: bool</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key </span><span class="si">{key}</span><span class="s2"> already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">key</span><span class="p">))</span>
+
+ <span class="n">extra_args</span><span class="o">=</span><span class="p">{}</span>
+ <span class="k">if</span> <span class="n">encrypt</span><span class="p">:</span>
+ <span class="n">extra_args</span><span class="p">[</span><span class="s1">'ServerSideEncryption'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"AES256"</span>
+
+ <span class="n">filelike_buffer</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="n">string_data</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">encoding</span><span class="p">))</span>
+
+ <span class="n">client</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">client</span><span class="o">.</span><span class="n">upload_fileobj</span><span class="p">(</span><span class="n">filelike_buffer</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">ExtraArgs</span><span class="o">=</span><span class="n">extra_args</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/macros.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/macros.html b/_modules/airflow/macros.html
index 34826d7..48eb480 100644
--- a/_modules/airflow/macros.html
+++ b/_modules/airflow/macros.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../start.html">Quick Start</a></li>
@@ -177,7 +182,6 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">absolute_import</span>
-<span class="kn">from</span> <span class="nn">random</span> <span class="k">import</span> <span class="n">random</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
<span class="kn">import</span> <span class="nn">dateutil</span>
<span class="kn">import</span> <span class="nn">time</span>
@@ -231,7 +235,7 @@
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">airflow.plugins_manager</span> <span class="k">import</span> <span class="n">macros_modules</span>
<span class="k">for</span> <span class="n">macros_module</span> <span class="ow">in</span> <span class="n">macros_modules</span><span class="p">:</span>
- <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">macros_module</span><span class="o">.</span><span class="n">__name__</span><span class="p">]</span> <span class="o">=</span> <span class="n">macros_module</span>
+ <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">macros_module</span><span class="o">.</span><span class="vm">__name__</span><span class="p">]</span> <span class="o">=</span> <span class="n">macros_module</span>
<span class="nb">globals</span><span class="p">()[</span><span class="n">macros_module</span><span class="o">.</span><span class="n">_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">macros_module</span>
<span class="c1">##########################################################</span>
@@ -241,7 +245,7 @@
<span class="k">if</span> <span class="ow">not</span> <span class="n">_os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'AIRFLOW_USE_NEW_IMPORTS'</span><span class="p">,</span> <span class="kc">False</span><span class="p">):</span>
<span class="kn">from</span> <span class="nn">zope.deprecation</span> <span class="k">import</span> <span class="n">deprecated</span> <span class="k">as</span> <span class="n">_deprecated</span>
<span class="k">for</span> <span class="n">_macro</span> <span class="ow">in</span> <span class="n">macros_module</span><span class="o">.</span><span class="n">_objects</span><span class="p">:</span>
- <span class="n">macro_name</span> <span class="o">=</span> <span class="n">_macro</span><span class="o">.</span><span class="n">__name__</span>
+ <span class="n">macro_name</span> <span class="o">=</span> <span class="n">_macro</span><span class="o">.</span><span class="vm">__name__</span>
<span class="nb">globals</span><span class="p">()[</span><span class="n">macro_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">_macro</span>
<span class="n">_deprecated</span><span class="p">(</span>
<span class="n">macro_name</span><span class="p">,</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/macros/hive.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/macros/hive.html b/_modules/airflow/macros/hive.html
index 8d13632..ba316f6 100644
--- a/_modules/airflow/macros/hive.html
+++ b/_modules/airflow/macros/hive.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
@@ -227,8 +232,8 @@
<span class="sd"> :returns: The closest date</span>
<span class="sd"> :rtype: datetime.date or None</span>
<span class="sd"> '''</span>
- <span class="n">fb</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span> <span class="k">if</span> <span class="n">d</span> <span class="o">>=</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="o">.</span><span class="n">max</span>
- <span class="n">fa</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span> <span class="k">if</span> <span class="n">d</span> <span class="o"><=</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="o">.</span><span class="n">min</span>
+ <span class="n">fb</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">target_dt</span> <span class="o">-</span> <span class="n">d</span> <span class="k">if</span> <span class="n">d</span> <span class="o"><=</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="o">.</span><span class="n">max</span>
+ <span class="n">fa</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span> <span class="k">if</span> <span class="n">d</span> <span class="o">>=</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="o">.</span><span class="n">max</span>
<span class="n">fnone</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">target_dt</span> <span class="o">-</span> <span class="n">d</span> <span class="k">if</span> <span class="n">d</span> <span class="o"><</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span>
<span class="k">if</span> <span class="n">before_target</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">min</span><span class="p">(</span><span class="n">date_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">fnone</span><span class="p">)</span><span class="o">.</span><span class="n">date</span><span class="p">()</span>
[22/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/operators/docker_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/docker_operator.html b/_modules/airflow/operators/docker_operator.html
index 06aa65e..10ff19a 100644
--- a/_modules/airflow/operators/docker_operator.html
+++ b/_modules/airflow/operators/docker_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../search.html"/>
<link rel="top" title="Airflow Documentation" href="../../../index.html"/>
<link rel="up" title="Module code" href="../../index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../../../index.html">Docs</a> »</li>
-
+
+ <li><a href="../../../index.html">Docs</a> »</li>
+
<li><a href="../../index.html">Module code</a> »</li>
-
- <li>airflow.operators.docker_operator</li>
+
+ <li>airflow.operators.docker_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -152,12 +182,13 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">import</span> <span class="nn">json</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="kn">import</span> <span class="n">TemporaryDirectory</span>
-<span class="kn">from</span> <span class="nn">docker</span> <span class="kn">import</span> <span class="n">Client</span><span class="p">,</span> <span class="n">tls</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.docker_hook</span> <span class="k">import</span> <span class="n">DockerHook</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="k">import</span> <span class="n">TemporaryDirectory</span>
+<span class="kn">from</span> <span class="nn">docker</span> <span class="k">import</span> <span class="n">Client</span><span class="p">,</span> <span class="n">tls</span>
<span class="kn">import</span> <span class="nn">ast</span>
@@ -169,9 +200,14 @@
<span class="sd"> that together exceed the default disk size of 10GB in a container. The path to the mounted</span>
<span class="sd"> directory can be accessed via the environment variable ``AIRFLOW_TMP_DIR``.</span>
+<span class="sd"> If a login to a private registry is required prior to pulling the image, a</span>
+<span class="sd"> Docker connection needs to be configured in Airflow and the connection ID</span>
+<span class="sd"> be provided with the parameter ``docker_conn_id``.</span>
+
<span class="sd"> :param image: Docker image from which to create the container.</span>
<span class="sd"> :type image: str</span>
-<span class="sd"> :param api_version: Remote API version.</span>
+<span class="sd"> :param api_version: Remote API version. Set to ``auto`` to automatically</span>
+<span class="sd"> detect the server's version.</span>
<span class="sd"> :type api_version: str</span>
<span class="sd"> :param command: Command to be run in the container.</span>
<span class="sd"> :type command: str or list</span>
@@ -180,10 +216,11 @@
<span class="sd"> https://docs.docker.com/engine/reference/run/#cpu-share-constraint</span>
<span class="sd"> :type cpus: float</span>
<span class="sd"> :param docker_url: URL of the host running the docker daemon.</span>
+<span class="sd"> Default is unix://var/run/docker.sock</span>
<span class="sd"> :type docker_url: str</span>
<span class="sd"> :param environment: Environment variables to set in the container.</span>
<span class="sd"> :type environment: dict</span>
-<span class="sd"> :param force_pull: Pull the docker image on every run.</span>
+<span class="sd"> :param force_pull: Pull the docker image on every run. Default is false.</span>
<span class="sd"> :type force_pull: bool</span>
<span class="sd"> :param mem_limit: Maximum amount of memory the container can use. Either a float value, which</span>
<span class="sd"> represents the limit in bytes, or a string like ``128m`` or ``1g``.</span>
@@ -209,11 +246,16 @@
<span class="sd"> :type user: int or str</span>
<span class="sd"> :param volumes: List of volumes to mount into the container, e.g.</span>
<span class="sd"> ``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.</span>
+<span class="sd"> :param working_dir: Working directory to set on the container (equivalent to the -w switch</span>
+<span class="sd"> the docker client)</span>
+<span class="sd"> :type working_dir: str</span>
<span class="sd"> :param xcom_push: Does the stdout will be pushed to the next step using XCom.</span>
<span class="sd"> The default is False.</span>
<span class="sd"> :type xcom_push: bool</span>
<span class="sd"> :param xcom_all: Push all the stdout or just the last line. The default is False (last line).</span>
<span class="sd"> :type xcom_all: bool</span>
+<span class="sd"> :param docker_conn_id: ID of the Airflow connection to use</span>
+<span class="sd"> :type docker_conn_id: str</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'command'</span><span class="p">,)</span>
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sh'</span><span class="p">,</span> <span class="s1">'.bash'</span><span class="p">,)</span>
@@ -222,28 +264,30 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">image</span><span class="p">,</span>
- <span class="n">api_version</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">command</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">api_version</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">command</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">cpus</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span>
<span class="n">docker_url</span><span class="o">=</span><span class="s1">'unix://var/run/docker.sock'</span><span class="p">,</span>
- <span class="n">environment</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">force_pull</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">mem_limit</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">network_mode</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">tls_ca_cert</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">tls_client_cert</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">tls_client_key</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">tls_hostname</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">tls_ssl_version</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">environment</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">force_pull</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">mem_limit</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">network_mode</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">tls_ca_cert</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">tls_client_cert</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">tls_client_key</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">tls_hostname</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">tls_ssl_version</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">tmp_dir</span><span class="o">=</span><span class="s1">'/tmp/airflow'</span><span class="p">,</span>
- <span class="n">user</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">volumes</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">xcom_push</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">xcom_all</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">user</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">volumes</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">working_dir</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">xcom_push</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">xcom_all</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">docker_conn_id</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">DockerOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DockerOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">api_version</span> <span class="o">=</span> <span class="n">api_version</span>
<span class="bp">self</span><span class="o">.</span><span class="n">command</span> <span class="o">=</span> <span class="n">command</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cpus</span> <span class="o">=</span> <span class="n">cpus</span>
@@ -261,27 +305,35 @@
<span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span> <span class="o">=</span> <span class="n">tmp_dir</span>
<span class="bp">self</span><span class="o">.</span><span class="n">user</span> <span class="o">=</span> <span class="n">user</span>
<span class="bp">self</span><span class="o">.</span><span class="n">volumes</span> <span class="o">=</span> <span class="n">volumes</span> <span class="ow">or</span> <span class="p">[]</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push</span> <span class="o">=</span> <span class="n">xcom_push</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">working_dir</span> <span class="o">=</span> <span class="n">working_dir</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push_flag</span> <span class="o">=</span> <span class="n">xcom_push</span>
<span class="bp">self</span><span class="o">.</span><span class="n">xcom_all</span> <span class="o">=</span> <span class="n">xcom_all</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">docker_conn_id</span> <span class="o">=</span> <span class="n">docker_conn_id</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="bp">None</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">DockerHook</span><span class="p">(</span>
+ <span class="n">docker_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">docker_conn_id</span><span class="p">,</span>
+ <span class="n">base_url</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="p">,</span>
+ <span class="n">version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">api_version</span><span class="p">,</span>
+ <span class="n">tls</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">__get_tls_config</span><span class="p">()</span>
+ <span class="p">)</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Starting docker container from image '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Starting docker container from image </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="p">)</span>
- <span class="n">tls_config</span> <span class="o">=</span> <span class="bp">None</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span><span class="p">:</span>
- <span class="n">tls_config</span> <span class="o">=</span> <span class="n">tls</span><span class="o">.</span><span class="n">TLSConfig</span><span class="p">(</span>
- <span class="n">ca_cert</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span><span class="p">,</span>
- <span class="n">client_cert</span><span class="o">=</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span><span class="p">),</span>
- <span class="n">verify</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
- <span class="n">ssl_version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_ssl_version</span><span class="p">,</span>
- <span class="n">assert_hostname</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_hostname</span>
- <span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'tcp://'</span><span class="p">,</span> <span class="s1">'https://'</span><span class="p">)</span>
+ <span class="n">tls_config</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">__get_tls_config</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="n">Client</span><span class="p">(</span><span class="n">base_url</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">api_version</span><span class="p">,</span> <span class="n">tls</span><span class="o">=</span><span class="n">tls_config</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">docker_conn_id</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="n">Client</span><span class="p">(</span>
+ <span class="n">base_url</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span><span class="p">,</span>
+ <span class="n">version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">api_version</span><span class="p">,</span>
+ <span class="n">tls</span><span class="o">=</span><span class="n">tls_config</span>
+ <span class="p">)</span>
<span class="k">if</span> <span class="s1">':'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="p">:</span>
<span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">+</span> <span class="s1">':latest'</span>
@@ -289,16 +341,16 @@
<span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">force_pull</span> <span class="ow">or</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">images</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">image</span><span class="p">))</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Pulling docker image '</span> <span class="o">+</span> <span class="n">image</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
- <span class="n">output</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">output</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Pulling docker image </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">image</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
+ <span class="n">output</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">l</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"</span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">output</span><span class="p">[</span><span class="s1">'status'</span><span class="p">])</span>
<span class="n">cpu_shares</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">round</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cpus</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">))</span>
<span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflowtmp'</span><span class="p">)</span> <span class="k">as</span> <span class="n">host_tmp_dir</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">environment</span><span class="p">[</span><span class="s1">'AIRFLOW_TMP_DIR'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'{0}:{1}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">host_tmp_dir</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'</span><span class="si">{0}</span><span class="s1">:</span><span class="si">{1}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">host_tmp_dir</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">create_container</span><span class="p">(</span>
<span class="n">command</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">get_command</span><span class="p">(),</span>
@@ -308,35 +360,55 @@
<span class="n">network_mode</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">network_mode</span><span class="p">),</span>
<span class="n">image</span><span class="o">=</span><span class="n">image</span><span class="p">,</span>
<span class="n">mem_limit</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mem_limit</span><span class="p">,</span>
- <span class="n">user</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">user</span>
+ <span class="n">user</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">user</span><span class="p">,</span>
+ <span class="n">working_dir</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">working_dir</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span>
<span class="n">line</span> <span class="o">=</span> <span class="s1">''</span>
- <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">logs</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">],</span> <span class="n">stream</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()))</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">logs</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">],</span> <span class="n">stream</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
+ <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">line</span><span class="p">,</span> <span class="s1">'decode'</span><span class="p">):</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="n">exit_code</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span>
<span class="k">if</span> <span class="n">exit_code</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'docker container failed'</span><span class="p">)</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">logs</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_all</span> <span class="k">else</span> <span class="nb">str</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push_flag</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">logs</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_all</span> <span class="k">else</span> <span class="nb">str</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_command</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'['</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'['</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">commands</span> <span class="o">=</span> <span class="n">ast</span><span class="o">.</span><span class="n">literal_eval</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">commands</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span>
<span class="k">return</span> <span class="n">commands</span>
<span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Stopping docker container'</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span></div>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Stopping docker container'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span>
+
+ <span class="k">def</span> <span class="nf">__get_tls_config</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">tls_config</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span><span class="p">:</span>
+ <span class="n">tls_config</span> <span class="o">=</span> <span class="n">tls</span><span class="o">.</span><span class="n">TLSConfig</span><span class="p">(</span>
+ <span class="n">ca_cert</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span><span class="p">,</span>
+ <span class="n">client_cert</span><span class="o">=</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span><span class="p">),</span>
+ <span class="n">verify</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">ssl_version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_ssl_version</span><span class="p">,</span>
+ <span class="n">assert_hostname</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_hostname</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'tcp://'</span><span class="p">,</span> <span class="s1">'https://'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">tls_config</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -369,7 +441,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../../../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/operators/redshift_to_s3_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/redshift_to_s3_operator.html b/_modules/airflow/operators/redshift_to_s3_operator.html
new file mode 100644
index 0000000..6b8e248
--- /dev/null
+++ b/_modules/airflow/operators/redshift_to_s3_operator.html
@@ -0,0 +1,337 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.operators.redshift_to_s3_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.operators.redshift_to_s3_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.operators.redshift_to_s3_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.postgres_hook</span> <span class="k">import</span> <span class="n">PostgresHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="RedshiftToS3Transfer"><a class="viewcode-back" href="../../../integration.html#airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer">[docs]</a><span class="k">class</span> <span class="nc">RedshiftToS3Transfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes an UNLOAD command to s3 as a CSV with headers</span>
+<span class="sd"> :param schema: reference to a specific schema in redshift database</span>
+<span class="sd"> :type schema: string</span>
+<span class="sd"> :param table: reference to a specific table in redshift database</span>
+<span class="sd"> :type table: string</span>
+<span class="sd"> :param s3_bucket: reference to a specific S3 bucket</span>
+<span class="sd"> :type s3_bucket: string</span>
+<span class="sd"> :param s3_key: reference to a specific S3 key</span>
+<span class="sd"> :type s3_key: string</span>
+<span class="sd"> :param redshift_conn_id: reference to a specific redshift database</span>
+<span class="sd"> :type redshift_conn_id: string</span>
+<span class="sd"> :param aws_conn_id: reference to a specific S3 connection</span>
+<span class="sd"> :type aws_conn_id: string</span>
+<span class="sd"> :param options: reference to a list of UNLOAD options</span>
+<span class="sd"> :type options: list</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#ededed'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">schema</span><span class="p">,</span>
+ <span class="n">table</span><span class="p">,</span>
+ <span class="n">s3_bucket</span><span class="p">,</span>
+ <span class="n">s3_key</span><span class="p">,</span>
+ <span class="n">redshift_conn_id</span><span class="o">=</span><span class="s1">'redshift_default'</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
+ <span class="n">unload_options</span><span class="o">=</span><span class="nb">tuple</span><span class="p">(),</span>
+ <span class="n">autocommit</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">RedshiftToS3Transfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="n">table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_bucket</span> <span class="o">=</span> <span class="n">s3_bucket</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span> <span class="o">=</span> <span class="n">s3_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">redshift_conn_id</span> <span class="o">=</span> <span class="n">redshift_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">unload_options</span> <span class="o">=</span> <span class="n">unload_options</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">parameters</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">PostgresHook</span><span class="p">(</span><span class="n">postgres_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">redshift_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
+ <span class="n">a_key</span><span class="p">,</span> <span class="n">s_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_credentials</span><span class="p">()</span>
+ <span class="n">unload_options</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\n\t\t\t</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">unload_options</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Retrieving headers from </span><span class="si">%s</span><span class="s2">.</span><span class="si">%s</span><span class="s2">..."</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">)</span>
+
+ <span class="n">columns_query</span> <span class="o">=</span> <span class="s2">"""SELECT column_name</span>
+<span class="s2"> FROM information_schema.columns</span>
+<span class="s2"> WHERE table_schema = '</span><span class="si">{0}</span><span class="s2">'</span>
+<span class="s2"> AND table_name = '</span><span class="si">{1}</span><span class="s2">'</span>
+<span class="s2"> ORDER BY ordinal_position</span>
+<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">)</span>
+
+ <span class="n">cursor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">columns_query</span><span class="p">)</span>
+ <span class="n">rows</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
+ <span class="n">columns</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">row</span><span class="p">:</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">rows</span><span class="p">)</span>
+ <span class="n">column_names</span> <span class="o">=</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">c</span><span class="p">:</span> <span class="s2">"</span><span class="se">\\</span><span class="s2">'</span><span class="si">{0}</span><span class="se">\\</span><span class="s2">'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">c</span><span class="p">),</span> <span class="n">columns</span><span class="p">))</span>
+ <span class="n">column_castings</span> <span class="o">=</span> <span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">c</span><span class="p">:</span> <span class="s2">"CAST(</span><span class="si">{0}</span><span class="s2"> AS text) AS </span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">c</span><span class="p">),</span>
+ <span class="n">columns</span><span class="p">))</span>
+
+ <span class="n">unload_query</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2"> UNLOAD ('SELECT </span><span class="si">{0}</span><span class="s2"></span>
+<span class="s2"> UNION ALL</span>
+<span class="s2"> SELECT </span><span class="si">{1}</span><span class="s2"> FROM </span><span class="si">{2}</span><span class="s2">.</span><span class="si">{3}</span><span class="s2"></span>
+<span class="s2"> ORDER BY 1 DESC')</span>
+<span class="s2"> TO 's3://</span><span class="si">{4}</span><span class="s2">/</span><span class="si">{5}</span><span class="s2">/</span><span class="si">{3}</span><span class="s2">_'</span>
+<span class="s2"> with</span>
+<span class="s2"> credentials 'aws_access_key_id=</span><span class="si">{6}</span><span class="s2">;aws_secret_access_key=</span><span class="si">{7}</span><span class="s2">'</span>
+<span class="s2"> </span><span class="si">{8}</span><span class="s2">;</span>
+<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">column_names</span><span class="p">,</span> <span class="n">column_castings</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_bucket</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">,</span> <span class="n">a_key</span><span class="p">,</span> <span class="n">s_key</span><span class="p">,</span> <span class="n">unload_options</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing UNLOAD command...'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">unload_query</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"UNLOAD command complete..."</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/operators/s3_file_transform_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/s3_file_transform_operator.html b/_modules/airflow/operators/s3_file_transform_operator.html
new file mode 100644
index 0000000..8db7bc2
--- /dev/null
+++ b/_modules/airflow/operators/s3_file_transform_operator.html
@@ -0,0 +1,342 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.operators.s3_file_transform_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.operators.s3_file_transform_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.operators.s3_file_transform_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="S3FileTransformOperator"><a class="viewcode-back" href="../../../integration.html#airflow.operators.s3_file_transform_operator.S3FileTransformOperator">[docs]</a><span class="k">class</span> <span class="nc">S3FileTransformOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Copies data from a source S3 location to a temporary location on the</span>
+<span class="sd"> local filesystem. Runs a transformation on this file as specified by</span>
+<span class="sd"> the transformation script and uploads the output to a destination S3</span>
+<span class="sd"> location.</span>
+
+<span class="sd"> The locations of the source and the destination files in the local</span>
+<span class="sd"> filesystem is provided as an first and second arguments to the</span>
+<span class="sd"> transformation script. The transformation script is expected to read the</span>
+<span class="sd"> data from source , transform it and write the output to the local</span>
+<span class="sd"> destination file. The operator then takes over control and uploads the</span>
+<span class="sd"> local destination file to S3.</span>
+
+<span class="sd"> :param source_s3_key: The key to be retrieved from S3</span>
+<span class="sd"> :type source_s3_key: str</span>
+<span class="sd"> :param source_aws_conn_id: source s3 connection</span>
+<span class="sd"> :type source_aws_conn_id: str</span>
+<span class="sd"> :param dest_s3_key: The key to be written from S3</span>
+<span class="sd"> :type dest_s3_key: str</span>
+<span class="sd"> :param dest_aws_conn_id: destination s3 connection</span>
+<span class="sd"> :type dest_aws_conn_id: str</span>
+<span class="sd"> :param replace: Replace dest S3 key if it already exists</span>
+<span class="sd"> :type replace: bool</span>
+<span class="sd"> :param transform_script: location of the executable transformation script</span>
+<span class="sd"> :type transform_script: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'source_s3_key'</span><span class="p">,</span> <span class="s1">'dest_s3_key'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f9c915'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">source_s3_key</span><span class="p">,</span>
+ <span class="n">dest_s3_key</span><span class="p">,</span>
+ <span class="n">transform_script</span><span class="p">,</span>
+ <span class="n">source_aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
+ <span class="n">dest_aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3FileTransformOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_s3_key</span> <span class="o">=</span> <span class="n">source_s3_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_aws_conn_id</span> <span class="o">=</span> <span class="n">source_aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dest_s3_key</span> <span class="o">=</span> <span class="n">dest_s3_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dest_aws_conn_id</span> <span class="o">=</span> <span class="n">dest_aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">replace</span> <span class="o">=</span> <span class="n">replace</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">transform_script</span> <span class="o">=</span> <span class="n">transform_script</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">source_s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">source_aws_conn_id</span><span class="p">)</span>
+ <span class="n">dest_s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">dest_aws_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Downloading source S3 file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">source_s3_key</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">source_s3</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_s3_key</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"The source key </span><span class="si">{0}</span><span class="s2"> does not exist"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_s3_key</span><span class="p">))</span>
+ <span class="n">source_s3_key_object</span> <span class="o">=</span> <span class="n">source_s3</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_s3_key</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_source</span><span class="p">,</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_dest</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Dumping S3 file </span><span class="si">%s</span><span class="s2"> contents to local file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_s3_key</span><span class="p">,</span> <span class="n">f_source</span><span class="o">.</span><span class="n">name</span>
+ <span class="p">)</span>
+ <span class="n">source_s3_key_object</span><span class="o">.</span><span class="n">get_contents_to_file</span><span class="p">(</span><span class="n">f_source</span><span class="p">)</span>
+ <span class="n">f_source</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">source_s3</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">transform_script_process</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
+ <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">transform_script</span><span class="p">,</span> <span class="n">f_source</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">f_dest</span><span class="o">.</span><span class="n">name</span><span class="p">],</span>
+ <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
+ <span class="p">(</span><span class="n">transform_script_stdoutdata</span><span class="p">,</span> <span class="n">transform_script_stderrdata</span><span class="p">)</span> <span class="o">=</span> <span class="n">transform_script_process</span><span class="o">.</span><span class="n">communicate</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Transform script stdout </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">transform_script_stdoutdata</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">transform_script_process</span><span class="o">.</span><span class="n">returncode</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Transform script failed </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">transform_script_stderrdata</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Transform script successful. Output temporarily located at </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
+ <span class="n">f_dest</span><span class="o">.</span><span class="n">name</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Uploading transformed file to S3"</span><span class="p">)</span>
+ <span class="n">f_dest</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">dest_s3</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">filename</span><span class="o">=</span><span class="n">f_dest</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="n">key</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">dest_s3_key</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">replace</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Upload successful"</span><span class="p">)</span>
+ <span class="n">dest_s3</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[23/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/models.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/models.html b/_modules/airflow/models.html
index 0b043ea..485f9ae 100644
--- a/_modules/airflow/models.html
+++ b/_modules/airflow/models.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../start.html">Quick Start</a></li>
@@ -175,6 +180,7 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
+
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">absolute_import</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">division</span>
<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">print_function</span>
@@ -193,7 +199,7 @@
<span class="kn">import</span> <span class="nn">getpass</span>
<span class="kn">import</span> <span class="nn">imp</span>
<span class="kn">import</span> <span class="nn">importlib</span>
-<span class="kn">import</span> <span class="nn">inspect</span>
+<span class="kn">import</span> <span class="nn">itertools</span>
<span class="kn">import</span> <span class="nn">zipfile</span>
<span class="kn">import</span> <span class="nn">jinja2</span>
<span class="kn">import</span> <span class="nn">json</span>
@@ -208,12 +214,11 @@
<span class="kn">import</span> <span class="nn">traceback</span>
<span class="kn">import</span> <span class="nn">warnings</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
-
<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="k">import</span> <span class="p">(</span>
<span class="n">Column</span><span class="p">,</span> <span class="n">Integer</span><span class="p">,</span> <span class="n">String</span><span class="p">,</span> <span class="n">DateTime</span><span class="p">,</span> <span class="n">Text</span><span class="p">,</span> <span class="n">Boolean</span><span class="p">,</span> <span class="n">ForeignKey</span><span class="p">,</span> <span class="n">PickleType</span><span class="p">,</span>
- <span class="n">Index</span><span class="p">,</span> <span class="n">Float</span><span class="p">)</span>
+ <span class="n">Index</span><span class="p">,</span> <span class="n">Float</span><span class="p">,</span> <span class="n">LargeBinary</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="k">import</span> <span class="n">func</span><span class="p">,</span> <span class="n">or_</span><span class="p">,</span> <span class="n">and_</span>
<span class="kn">from</span> <span class="nn">sqlalchemy.ext.declarative</span> <span class="k">import</span> <span class="n">declarative_base</span><span class="p">,</span> <span class="n">declared_attr</span>
<span class="kn">from</span> <span class="nn">sqlalchemy.dialects.mysql</span> <span class="k">import</span> <span class="n">LONGTEXT</span>
@@ -223,13 +228,15 @@
<span class="kn">import</span> <span class="nn">six</span>
<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">settings</span><span class="p">,</span> <span class="n">utils</span>
-<span class="kn">from</span> <span class="nn">airflow.executors</span> <span class="k">import</span> <span class="n">DEFAULT_EXECUTOR</span><span class="p">,</span> <span class="n">LocalExecutor</span>
+<span class="kn">from</span> <span class="nn">airflow.executors</span> <span class="k">import</span> <span class="n">GetDefaultExecutor</span><span class="p">,</span> <span class="n">LocalExecutor</span>
<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">configuration</span>
<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span><span class="p">,</span> <span class="n">AirflowSkipException</span><span class="p">,</span> <span class="n">AirflowTaskTimeout</span>
<span class="kn">from</span> <span class="nn">airflow.dag.base_dag</span> <span class="k">import</span> <span class="n">BaseDag</span><span class="p">,</span> <span class="n">BaseDagBag</span>
<span class="kn">from</span> <span class="nn">airflow.ti_deps.deps.not_in_retry_period_dep</span> <span class="k">import</span> <span class="n">NotInRetryPeriodDep</span>
<span class="kn">from</span> <span class="nn">airflow.ti_deps.deps.prev_dagrun_dep</span> <span class="k">import</span> <span class="n">PrevDagrunDep</span>
<span class="kn">from</span> <span class="nn">airflow.ti_deps.deps.trigger_rule_dep</span> <span class="k">import</span> <span class="n">TriggerRuleDep</span>
+<span class="kn">from</span> <span class="nn">airflow.ti_deps.deps.task_concurrency_dep</span> <span class="k">import</span> <span class="n">TaskConcurrencyDep</span>
+
<span class="kn">from</span> <span class="nn">airflow.ti_deps.dep_context</span> <span class="k">import</span> <span class="n">DepContext</span><span class="p">,</span> <span class="n">QUEUE_DEPS</span><span class="p">,</span> <span class="n">RUN_DEPS</span>
<span class="kn">from</span> <span class="nn">airflow.utils.dates</span> <span class="k">import</span> <span class="n">cron_presets</span><span class="p">,</span> <span class="n">date_range</span> <span class="k">as</span> <span class="n">utils_date_range</span>
<span class="kn">from</span> <span class="nn">airflow.utils.db</span> <span class="k">import</span> <span class="n">provide_session</span>
@@ -237,11 +244,11 @@
<span class="kn">from</span> <span class="nn">airflow.utils.email</span> <span class="k">import</span> <span class="n">send_email</span>
<span class="kn">from</span> <span class="nn">airflow.utils.helpers</span> <span class="k">import</span> <span class="p">(</span>
<span class="n">as_tuple</span><span class="p">,</span> <span class="n">is_container</span><span class="p">,</span> <span class="n">is_in</span><span class="p">,</span> <span class="n">validate_key</span><span class="p">,</span> <span class="n">pprinttable</span><span class="p">)</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.logging</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
<span class="kn">from</span> <span class="nn">airflow.utils.operator_resources</span> <span class="k">import</span> <span class="n">Resources</span>
<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="k">import</span> <span class="n">State</span>
<span class="kn">from</span> <span class="nn">airflow.utils.timeout</span> <span class="k">import</span> <span class="n">timeout</span>
<span class="kn">from</span> <span class="nn">airflow.utils.trigger_rule</span> <span class="k">import</span> <span class="n">TriggerRule</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
<span class="n">Base</span> <span class="o">=</span> <span class="n">declarative_base</span><span class="p">()</span>
<span class="n">ID_LEN</span> <span class="o">=</span> <span class="mi">250</span>
@@ -249,24 +256,36 @@
<span class="n">Stats</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Stats</span>
-<span class="n">ENCRYPTION_ON</span> <span class="o">=</span> <span class="kc">False</span>
-<span class="k">try</span><span class="p">:</span>
- <span class="kn">from</span> <span class="nn">cryptography.fernet</span> <span class="k">import</span> <span class="n">Fernet</span>
- <span class="n">FERNET</span> <span class="o">=</span> <span class="n">Fernet</span><span class="p">(</span><span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'FERNET_KEY'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
- <span class="n">ENCRYPTION_ON</span> <span class="o">=</span> <span class="kc">True</span>
-<span class="k">except</span><span class="p">:</span>
- <span class="k">pass</span>
+<span class="k">def</span> <span class="nf">get_fernet</span><span class="p">():</span>
+ <span class="sd">"""</span>
+<span class="sd"> Deferred load of Fernet key.</span>
+
+<span class="sd"> This function could fail either because Cryptography is not installed</span>
+<span class="sd"> or because the Fernet key is invalid.</span>
+
+<span class="sd"> :return: Fernet object</span>
+<span class="sd"> :raises: AirflowException if there's a problem trying to load Fernet</span>
+<span class="sd"> """</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">cryptography.fernet</span> <span class="k">import</span> <span class="n">Fernet</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Failed to import Fernet, it may not be installed'</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">Fernet</span><span class="p">(</span><span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'FERNET_KEY'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
+ <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">ve</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Could not create Fernet object: </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">ve</span><span class="p">))</span>
+
<span class="k">if</span> <span class="s1">'mysql'</span> <span class="ow">in</span> <span class="n">settings</span><span class="o">.</span><span class="n">SQL_ALCHEMY_CONN</span><span class="p">:</span>
<span class="n">LongText</span> <span class="o">=</span> <span class="n">LONGTEXT</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">LongText</span> <span class="o">=</span> <span class="n">Text</span>
-<span class="c1"># used by DAG context_managers</span>
+<span class="c1"># Used by DAG context_managers</span>
<span class="n">_CONTEXT_MANAGER_DAG</span> <span class="o">=</span> <span class="kc">None</span>
-<span class="k">def</span> <span class="nf">clear_task_instances</span><span class="p">(</span><span class="n">tis</span><span class="p">,</span> <span class="n">session</span><span class="p">,</span> <span class="n">activate_dag_runs</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
+<span class="k">def</span> <span class="nf">clear_task_instances</span><span class="p">(</span><span class="n">tis</span><span class="p">,</span> <span class="n">session</span><span class="p">,</span> <span class="n">activate_dag_runs</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Clears a set of task instances, but makes sure the running ones</span>
<span class="sd"> get killed.</span>
@@ -277,26 +296,34 @@
<span class="k">if</span> <span class="n">ti</span><span class="o">.</span><span class="n">job_id</span><span class="p">:</span>
<span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SHUTDOWN</span>
<span class="n">job_ids</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ti</span><span class="o">.</span><span class="n">job_id</span><span class="p">)</span>
- <span class="c1"># todo: this creates an issue with the webui tests</span>
- <span class="c1"># elif ti.state != State.REMOVED:</span>
- <span class="c1"># ti.state = State.NONE</span>
- <span class="c1"># session.merge(ti)</span>
<span class="k">else</span><span class="p">:</span>
- <span class="n">session</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="n">task_id</span> <span class="o">=</span> <span class="n">ti</span><span class="o">.</span><span class="n">task_id</span>
+ <span class="k">if</span> <span class="n">dag</span> <span class="ow">and</span> <span class="n">dag</span><span class="o">.</span><span class="n">has_task</span><span class="p">(</span><span class="n">task_id</span><span class="p">):</span>
+ <span class="n">task</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">get_task</span><span class="p">(</span><span class="n">task_id</span><span class="p">)</span>
+ <span class="n">task_retries</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">retries</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">max_tries</span> <span class="o">=</span> <span class="n">ti</span><span class="o">.</span><span class="n">try_number</span> <span class="o">+</span> <span class="n">task_retries</span> <span class="o">-</span> <span class="mi">1</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="c1"># Ignore errors when updating max_tries if dag is None or</span>
+ <span class="c1"># task not found in dag since database records could be</span>
+ <span class="c1"># outdated. We make max_tries the maximum value of its</span>
+ <span class="c1"># original max_tries or the current task try number.</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">max_tries</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">ti</span><span class="o">.</span><span class="n">max_tries</span><span class="p">,</span> <span class="n">ti</span><span class="o">.</span><span class="n">try_number</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">NONE</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+
<span class="k">if</span> <span class="n">job_ids</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">airflow.jobs</span> <span class="k">import</span> <span class="n">BaseJob</span> <span class="k">as</span> <span class="n">BJ</span>
<span class="k">for</span> <span class="n">job</span> <span class="ow">in</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">BJ</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">BJ</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">job_ids</span><span class="p">))</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
<span class="n">job</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SHUTDOWN</span>
- <span class="k">if</span> <span class="n">activate_dag_runs</span><span class="p">:</span>
- <span class="n">execution_dates</span> <span class="o">=</span> <span class="p">{</span><span class="n">ti</span><span class="o">.</span><span class="n">execution_date</span> <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">}</span>
- <span class="n">dag_ids</span> <span class="o">=</span> <span class="p">{</span><span class="n">ti</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">}</span>
+
+ <span class="k">if</span> <span class="n">activate_dag_runs</span> <span class="ow">and</span> <span class="n">tis</span><span class="p">:</span>
<span class="n">drs</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DagRun</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
- <span class="n">DagRun</span><span class="o">.</span><span class="n">dag_id</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">dag_ids</span><span class="p">),</span>
- <span class="n">DagRun</span><span class="o">.</span><span class="n">execution_date</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">execution_dates</span><span class="p">),</span>
+ <span class="n">DagRun</span><span class="o">.</span><span class="n">dag_id</span><span class="o">.</span><span class="n">in_</span><span class="p">({</span><span class="n">ti</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">}),</span>
+ <span class="n">DagRun</span><span class="o">.</span><span class="n">execution_date</span><span class="o">.</span><span class="n">in_</span><span class="p">({</span><span class="n">ti</span><span class="o">.</span><span class="n">execution_date</span> <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">}),</span>
<span class="p">)</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
<span class="k">for</span> <span class="n">dr</span> <span class="ow">in</span> <span class="n">drs</span><span class="p">:</span>
<span class="n">dr</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span>
- <span class="n">dr</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">dr</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<div class="viewcode-block" id="DagBag"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag">[docs]</a><span class="k">class</span> <span class="nc">DagBag</span><span class="p">(</span><span class="n">BaseDagBag</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
@@ -316,19 +343,19 @@
<span class="sd"> :param include_examples: whether to include the examples that ship</span>
<span class="sd"> with airflow or not</span>
<span class="sd"> :type include_examples: bool</span>
-<span class="sd"> :param sync_to_db: whether to sync the properties of the DAGs to</span>
-<span class="sd"> the metadata DB while finding them, typically should be done</span>
-<span class="sd"> by the scheduler job only</span>
-<span class="sd"> :type sync_to_db: bool</span>
<span class="sd"> """</span>
+
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">dag_folder</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
- <span class="n">executor</span><span class="o">=</span><span class="n">DEFAULT_EXECUTOR</span><span class="p">,</span>
+ <span class="n">executor</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">include_examples</span><span class="o">=</span><span class="n">configuration</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'LOAD_EXAMPLES'</span><span class="p">)):</span>
+ <span class="c1"># do not use default arg in signature, to fix import cycle on plugin load</span>
+ <span class="k">if</span> <span class="n">executor</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">executor</span> <span class="o">=</span> <span class="n">GetDefaultExecutor</span><span class="p">()</span>
<span class="n">dag_folder</span> <span class="o">=</span> <span class="n">dag_folder</span> <span class="ow">or</span> <span class="n">settings</span><span class="o">.</span><span class="n">DAGS_FOLDER</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Filling up the DagBag from </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Filling up the DagBag from </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">dag_folder</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dag_folder</span> <span class="o">=</span> <span class="n">dag_folder</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dags</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># the file's last modified timestamp when we last read it</span>
@@ -338,7 +365,7 @@
<span class="k">if</span> <span class="n">include_examples</span><span class="p">:</span>
<span class="n">example_dag_folder</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
- <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">),</span>
+ <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="vm">__file__</span><span class="p">),</span>
<span class="s1">'example_dags'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">collect_dags</span><span class="p">(</span><span class="n">example_dag_folder</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">collect_dags</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">)</span>
@@ -401,7 +428,7 @@
<span class="k">return</span> <span class="n">found_dags</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="k">return</span> <span class="n">found_dags</span>
<span class="n">mods</span> <span class="o">=</span> <span class="p">[]</span>
@@ -409,11 +436,11 @@
<span class="k">if</span> <span class="n">safe_mode</span> <span class="ow">and</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">([</span><span class="n">s</span> <span class="ow">in</span> <span class="n">content</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="p">(</span><span class="n">b</span><span class="s1">'DAG'</span><span class="p">,</span> <span class="n">b</span><span class="s1">'airflow'</span><span class="p">)]):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">([</span><span class="n">s</span> <span class="ow">in</span> <span class="n">content</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="p">(</span><span class="sa">b</span><span class="s1">'DAG'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'airflow'</span><span class="p">)]):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">file_last_changed_on_disk</span>
<span class="k">return</span> <span class="n">found_dags</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Importing </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filepath</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Importing </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
<span class="n">org_mod_name</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">filepath</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="n">mod_name</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'unusual_prefix_'</span> <span class="o">+</span>
<span class="n">hashlib</span><span class="o">.</span><span class="n">sha1</span><span class="p">(</span><span class="n">filepath</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span> <span class="o">+</span>
@@ -427,7 +454,7 @@
<span class="n">m</span> <span class="o">=</span> <span class="n">imp</span><span class="o">.</span><span class="n">load_source</span><span class="p">(</span><span class="n">mod_name</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
<span class="n">mods</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to import: "</span> <span class="o">+</span> <span class="n">filepath</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to import: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">import_errors</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">file_last_changed_on_disk</span>
@@ -438,15 +465,12 @@
<span class="n">mod_name</span><span class="p">,</span> <span class="n">ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">head</span> <span class="ow">and</span> <span class="p">(</span><span class="n">ext</span> <span class="o">==</span> <span class="s1">'.py'</span> <span class="ow">or</span> <span class="n">ext</span> <span class="o">==</span> <span class="s1">'.pyc'</span><span class="p">):</span>
<span class="k">if</span> <span class="n">mod_name</span> <span class="o">==</span> <span class="s1">'__init__'</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Found __init__.</span><span class="si">{0}</span><span class="s2"> at root of </span><span class="si">{1}</span><span class="s2">"</span><span class="o">.</span>
- <span class="nb">format</span><span class="p">(</span><span class="n">ext</span><span class="p">,</span> <span class="n">filepath</span><span class="p">))</span>
-
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Found __init__.</span><span class="si">%s</span><span class="s2"> at root of </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">ext</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
<span class="k">if</span> <span class="n">safe_mode</span><span class="p">:</span>
<span class="k">with</span> <span class="n">zip_file</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">zf</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Reading </span><span class="si">{}</span><span class="s2"> from </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span>
- <span class="nb">format</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="n">filepath</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Reading </span><span class="si">%s</span><span class="s2"> from </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">zf</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">([</span><span class="n">s</span> <span class="ow">in</span> <span class="n">content</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="p">(</span><span class="n">b</span><span class="s1">'DAG'</span><span class="p">,</span> <span class="n">b</span><span class="s1">'airflow'</span><span class="p">)]):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">([</span><span class="n">s</span> <span class="ow">in</span> <span class="n">content</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="p">(</span><span class="sa">b</span><span class="s1">'DAG'</span><span class="p">,</span> <span class="sa">b</span><span class="s1">'airflow'</span><span class="p">)]):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">file_last_changed_on_disk</span><span class="p">)</span>
<span class="c1"># todo: create ignore list</span>
@@ -460,12 +484,12 @@
<span class="n">m</span> <span class="o">=</span> <span class="n">importlib</span><span class="o">.</span><span class="n">import_module</span><span class="p">(</span><span class="n">mod_name</span><span class="p">)</span>
<span class="n">mods</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to import: "</span> <span class="o">+</span> <span class="n">filepath</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to import: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">import_errors</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">file_last_changed_on_disk</span>
<span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">mods</span><span class="p">:</span>
- <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">m</span><span class="o">.</span><span class="n">__dict__</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
+ <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">m</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">dag</span><span class="p">,</span> <span class="n">DAG</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">dag</span><span class="o">.</span><span class="n">full_filepath</span><span class="p">:</span>
<span class="n">dag</span><span class="o">.</span><span class="n">full_filepath</span> <span class="o">=</span> <span class="n">filepath</span>
@@ -477,19 +501,17 @@
<span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">file_last_changed_on_disk</span>
<span class="k">return</span> <span class="n">found_dags</span></div>
- <span class="nd">@provide_session</span>
-<div class="viewcode-block" id="DagBag.kill_zombies"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.kill_zombies">[docs]</a> <span class="k">def</span> <span class="nf">kill_zombies</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">session</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+<div class="viewcode-block" id="DagBag.kill_zombies"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.kill_zombies">[docs]</a> <span class="nd">@provide_session</span>
+ <span class="k">def</span> <span class="nf">kill_zombies</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">session</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Fails tasks that haven't had a heartbeat in too long</span>
<span class="sd"> """</span>
<span class="kn">from</span> <span class="nn">airflow.jobs</span> <span class="k">import</span> <span class="n">LocalTaskJob</span> <span class="k">as</span> <span class="n">LJ</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Finding 'running' jobs without a recent heartbeat"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Finding 'running' jobs without a recent heartbeat"</span><span class="p">)</span>
<span class="n">TI</span> <span class="o">=</span> <span class="n">TaskInstance</span>
- <span class="n">secs</span> <span class="o">=</span> <span class="p">(</span>
- <span class="n">configuration</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s1">'scheduler'</span><span class="p">,</span> <span class="s1">'scheduler_zombie_task_threshold'</span><span class="p">))</span>
- <span class="n">limit_dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="n">secs</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s2">"Failing jobs without heartbeat after </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">limit_dttm</span><span class="p">))</span>
+ <span class="n">secs</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s1">'scheduler'</span><span class="p">,</span> <span class="s1">'scheduler_zombie_task_threshold'</span><span class="p">)</span>
+ <span class="n">limit_dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="n">secs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Failing jobs without heartbeat after </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">limit_dttm</span><span class="p">)</span>
<span class="n">tis</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">TI</span><span class="p">)</span>
@@ -509,9 +531,8 @@
<span class="k">if</span> <span class="n">ti</span><span class="o">.</span><span class="n">task_id</span> <span class="ow">in</span> <span class="n">dag</span><span class="o">.</span><span class="n">task_ids</span><span class="p">:</span>
<span class="n">task</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">get_task</span><span class="p">(</span><span class="n">ti</span><span class="o">.</span><span class="n">task_id</span><span class="p">)</span>
<span class="n">ti</span><span class="o">.</span><span class="n">task</span> <span class="o">=</span> <span class="n">task</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">handle_failure</span><span class="p">(</span><span class="s2">"</span><span class="si">{}</span><span class="s2"> killed as zombie"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">ti</span><span class="p">))</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s1">'Marked zombie job </span><span class="si">{}</span><span class="s1"> as failed'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">ti</span><span class="p">))</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">handle_failure</span><span class="p">(</span><span class="s2">"</span><span class="si">{}</span><span class="s2"> killed as zombie"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">ti</span><span class="p">)))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Marked zombie job </span><span class="si">%s</span><span class="s1"> as failed'</span><span class="p">,</span> <span class="n">ti</span><span class="p">)</span>
<span class="n">Stats</span><span class="o">.</span><span class="n">incr</span><span class="p">(</span><span class="s1">'zombies_killed'</span><span class="p">)</span>
<span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span></div>
@@ -521,7 +542,7 @@
<span class="sd"> """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">[</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">dag</span>
<span class="n">dag</span><span class="o">.</span><span class="n">resolve_template_files</span><span class="p">()</span>
- <span class="n">dag</span><span class="o">.</span><span class="n">last_loaded</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">last_loaded</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">dag</span><span class="o">.</span><span class="n">tasks</span><span class="p">:</span>
<span class="n">settings</span><span class="o">.</span><span class="n">policy</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
@@ -531,7 +552,7 @@
<span class="n">subdag</span><span class="o">.</span><span class="n">parent_dag</span> <span class="o">=</span> <span class="n">dag</span>
<span class="n">subdag</span><span class="o">.</span><span class="n">is_subdag</span> <span class="o">=</span> <span class="kc">True</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bag_dag</span><span class="p">(</span><span class="n">subdag</span><span class="p">,</span> <span class="n">parent_dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span> <span class="n">root_dag</span><span class="o">=</span><span class="n">root_dag</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Loaded DAG </span><span class="si">{dag}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Loaded DAG </span><span class="si">{dag}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
<div class="viewcode-block" id="DagBag.collect_dags"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.collect_dags">[docs]</a> <span class="k">def</span> <span class="nf">collect_dags</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
@@ -546,7 +567,7 @@
<span class="sd"> ignoring files that match any of the regex patterns specified</span>
<span class="sd"> in the file.</span>
<span class="sd"> """</span>
- <span class="n">start_dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">start_dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="n">dag_folder</span> <span class="o">=</span> <span class="n">dag_folder</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dag_folder</span>
<span class="c1"># Used to store stats around DagBag processing</span>
@@ -574,11 +595,11 @@
<span class="k">continue</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">any</span><span class="p">(</span>
<span class="p">[</span><span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">patterns</span><span class="p">]):</span>
- <span class="n">ts</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">ts</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span>
<span class="n">found_dags</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">process_file</span><span class="p">(</span>
<span class="n">filepath</span><span class="p">,</span> <span class="n">only_if_updated</span><span class="o">=</span><span class="n">only_if_updated</span><span class="p">)</span>
- <span class="n">td</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">ts</span>
+ <span class="n">td</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">ts</span>
<span class="n">td</span> <span class="o">=</span> <span class="n">td</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">+</span> <span class="p">(</span>
<span class="nb">float</span><span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">microseconds</span><span class="p">)</span> <span class="o">/</span> <span class="mi">1000000</span><span class="p">)</span>
<span class="n">stats</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">FileLoadStat</span><span class="p">(</span>
@@ -589,9 +610,9 @@
<span class="nb">str</span><span class="p">([</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="n">found_dags</span><span class="p">]),</span>
<span class="p">))</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="n">Stats</span><span class="o">.</span><span class="n">gauge</span><span class="p">(</span>
- <span class="s1">'collect_dags'</span><span class="p">,</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_dttm</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">(),</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="s1">'collect_dags'</span><span class="p">,</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_dttm</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">(),</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">Stats</span><span class="o">.</span><span class="n">gauge</span><span class="p">(</span>
<span class="s1">'dagbag_size'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">Stats</span><span class="o">.</span><span class="n">gauge</span><span class="p">(</span>
@@ -632,7 +653,7 @@
<span class="k">def</span> <span class="nf">paused_dags</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
<span class="n">dag_ids</span> <span class="o">=</span> <span class="p">[</span><span class="n">dp</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">dp</span> <span class="ow">in</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DagModel</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
- <span class="n">DagModel</span><span class="o">.</span><span class="n">is_paused</span><span class="o">.</span><span class="n">is_</span><span class="p">(</span><span class="kc">True</span><span class="p">))]</span>
+ <span class="n">DagModel</span><span class="o">.</span><span class="n">is_paused</span><span class="o">.</span><span class="fm">__eq__</span><span class="p">(</span><span class="kc">True</span><span class="p">))]</span>
<span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">return</span> <span class="n">dag_ids</span></div>
@@ -656,7 +677,7 @@
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">superuser</span>
-<div class="viewcode-block" id="Connection"><a class="viewcode-back" href="../../code.html#airflow.models.Connection">[docs]</a><span class="k">class</span> <span class="nc">Connection</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
+<div class="viewcode-block" id="Connection"><a class="viewcode-back" href="../../code.html#airflow.models.Connection">[docs]</a><span class="k">class</span> <span class="nc">Connection</span><span class="p">(</span><span class="n">Base</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Placeholder to store information about different database instances</span>
<span class="sd"> connection information. The idea here is that scripts use references to</span>
@@ -678,6 +699,7 @@
<span class="n">_extra</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="s1">'extra'</span><span class="p">,</span> <span class="n">String</span><span class="p">(</span><span class="mi">5000</span><span class="p">))</span>
<span class="n">_types</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="p">(</span><span class="s1">'docker'</span><span class="p">,</span> <span class="s1">'Docker Registry'</span><span class="p">,),</span>
<span class="p">(</span><span class="s1">'fs'</span><span class="p">,</span> <span class="s1">'File (path)'</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'ftp'</span><span class="p">,</span> <span class="s1">'FTP'</span><span class="p">,),</span>
<span class="p">(</span><span class="s1">'google_cloud_platform'</span><span class="p">,</span> <span class="s1">'Google Cloud Platform'</span><span class="p">),</span>
@@ -700,6 +722,11 @@
<span class="p">(</span><span class="s1">'mssql'</span><span class="p">,</span> <span class="s1">'Microsoft SQL Server'</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'mesos_framework-id'</span><span class="p">,</span> <span class="s1">'Mesos Framework ID'</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'jira'</span><span class="p">,</span> <span class="s1">'JIRA'</span><span class="p">,),</span>
+ <span class="p">(</span><span class="s1">'redis'</span><span class="p">,</span> <span class="s1">'Redis'</span><span class="p">,),</span>
+ <span class="p">(</span><span class="s1">'wasb'</span><span class="p">,</span> <span class="s1">'Azure Blob Storage'</span><span class="p">),</span>
+ <span class="p">(</span><span class="s1">'databricks'</span><span class="p">,</span> <span class="s1">'Databricks'</span><span class="p">,),</span>
+ <span class="p">(</span><span class="s1">'aws'</span><span class="p">,</span> <span class="s1">'Amazon Web Services'</span><span class="p">,),</span>
+ <span class="p">(</span><span class="s1">'emr'</span><span class="p">,</span> <span class="s1">'Elastic MapReduce'</span><span class="p">,),</span>
<span class="p">]</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
@@ -736,51 +763,61 @@
<span class="k">def</span> <span class="nf">get_password</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_password</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">is_encrypted</span><span class="p">:</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">ENCRYPTION_ON</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">fernet</span> <span class="o">=</span> <span class="n">get_fernet</span><span class="p">()</span>
+ <span class="k">except</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
<span class="s2">"Can't decrypt encrypted password for login=</span><span class="si">{}</span><span class="s2">, </span><span class="se">\</span>
<span class="s2"> FERNET_KEY configuration is missing"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">login</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">FERNET</span><span class="o">.</span><span class="n">decrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_password</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">fernet</span><span class="o">.</span><span class="n">decrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_password</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_password</span>
<span class="k">def</span> <span class="nf">set_password</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">if</span> <span class="n">value</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_password</span> <span class="o">=</span> <span class="n">FERNET</span><span class="o">.</span><span class="n">encrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
+ <span class="n">fernet</span> <span class="o">=</span> <span class="n">get_fernet</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_password</span> <span class="o">=</span> <span class="n">fernet</span><span class="o">.</span><span class="n">encrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_encrypted</span> <span class="o">=</span> <span class="kc">True</span>
- <span class="k">except</span> <span class="ne">NameError</span><span class="p">:</span>
+ <span class="k">except</span> <span class="n">AirflowException</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to load fernet while encrypting value, "</span>
+ <span class="s2">"using non-encrypted value."</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_password</span> <span class="o">=</span> <span class="n">value</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_encrypted</span> <span class="o">=</span> <span class="kc">False</span>
<span class="nd">@declared_attr</span>
- <span class="k">def</span> <span class="nf">password</span><span class="p">(</span><span class="n">cls</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">password</span><span class="p">(</span><span class="bp">cls</span><span class="p">):</span>
<span class="k">return</span> <span class="n">synonym</span><span class="p">(</span><span class="s1">'_password'</span><span class="p">,</span>
- <span class="n">descriptor</span><span class="o">=</span><span class="nb">property</span><span class="p">(</span><span class="n">cls</span><span class="o">.</span><span class="n">get_password</span><span class="p">,</span> <span class="n">cls</span><span class="o">.</span><span class="n">set_password</span><span class="p">))</span>
+ <span class="n">descriptor</span><span class="o">=</span><span class="nb">property</span><span class="p">(</span><span class="bp">cls</span><span class="o">.</span><span class="n">get_password</span><span class="p">,</span> <span class="bp">cls</span><span class="o">.</span><span class="n">set_password</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">get_extra</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_extra</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">is_extra_encrypted</span><span class="p">:</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">ENCRYPTION_ON</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">fernet</span> <span class="o">=</span> <span class="n">get_fernet</span><span class="p">()</span>
+ <span class="k">except</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
<span class="s2">"Can't decrypt `extra` params for login=</span><span class="si">{}</span><span class="s2">,</span><span class="se">\</span>
<span class="s2"> FERNET_KEY configuration is missing"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">login</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">FERNET</span><span class="o">.</span><span class="n">decrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_extra</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">fernet</span><span class="o">.</span><span class="n">decrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_extra</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_extra</span>
<span class="k">def</span> <span class="nf">set_extra</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">if</span> <span class="n">value</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_extra</span> <span class="o">=</span> <span class="n">FERNET</span><span class="o">.</span><span class="n">encrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
+ <span class="n">fernet</span> <span class="o">=</span> <span class="n">get_fernet</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_extra</span> <span class="o">=</span> <span class="n">fernet</span><span class="o">.</span><span class="n">encrypt</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="s1">'utf-8'</span><span class="p">))</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_extra_encrypted</span> <span class="o">=</span> <span class="kc">True</span>
- <span class="k">except</span> <span class="ne">NameError</span><span class="p">:</span>
+ <span class="k">except</span> <span class="n">AirflowException</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to load fernet while encrypting value, "</span>
+ <span class="s2">"using non-encrypted value."</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_extra</span> <span class="o">=</span> <span class="n">value</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_extra_encrypted</span> <span class="o">=</span> <span class="kc">False</span>
<span class="nd">@declared_attr</span>
- <span class="k">def</span> <span class="nf">extra</span><span class="p">(</span><span class="n">cls</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">extra</span><span class="p">(</span><span class="bp">cls</span><span class="p">):</span>
<span class="k">return</span> <span class="n">synonym</span><span class="p">(</span><span class="s1">'_extra'</span><span class="p">,</span>
- <span class="n">descriptor</span><span class="o">=</span><span class="nb">property</span><span class="p">(</span><span class="n">cls</span><span class="o">.</span><span class="n">get_extra</span><span class="p">,</span> <span class="n">cls</span><span class="o">.</span><span class="n">set_extra</span><span class="p">))</span>
+ <span class="n">descriptor</span><span class="o">=</span><span class="nb">property</span><span class="p">(</span><span class="bp">cls</span><span class="o">.</span><span class="n">get_extra</span><span class="p">,</span> <span class="bp">cls</span><span class="o">.</span><span class="n">set_extra</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
@@ -823,6 +860,15 @@
<span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_type</span> <span class="o">==</span> <span class="s1">'jira'</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.jira_hook</span> <span class="k">import</span> <span class="n">JiraHook</span>
<span class="k">return</span> <span class="n">JiraHook</span><span class="p">(</span><span class="n">jira_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_type</span> <span class="o">==</span> <span class="s1">'redis'</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">airflow.contrib.hooks.redis_hook</span> <span class="k">import</span> <span class="n">RedisHook</span>
+ <span class="k">return</span> <span class="n">RedisHook</span><span class="p">(</span><span class="n">redis_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_type</span> <span class="o">==</span> <span class="s1">'wasb'</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">airflow.contrib.hooks.wasb_hook</span> <span class="k">import</span> <span class="n">WasbHook</span>
+ <span class="k">return</span> <span class="n">WasbHook</span><span class="p">(</span><span class="n">wasb_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_type</span> <span class="o">==</span> <span class="s1">'docker'</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks.docker_hook</span> <span class="k">import</span> <span class="n">DockerHook</span>
+ <span class="k">return</span> <span class="n">DockerHook</span><span class="p">(</span><span class="n">docker_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
<span class="k">pass</span>
@@ -837,8 +883,8 @@
<span class="k">try</span><span class="p">:</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">extra</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Failed parsing the json for conn_id </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Failed parsing the json for conn_id </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span>
<span class="k">return</span> <span class="n">obj</span></div>
@@ -869,13 +915,13 @@
<span class="bp">self</span><span class="o">.</span><span class="n">pickle</span> <span class="o">=</span> <span class="n">dag</span>
-<div class="viewcode-block" id="TaskInstance"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance">[docs]</a><span class="k">class</span> <span class="nc">TaskInstance</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
+<div class="viewcode-block" id="TaskInstance"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance">[docs]</a><span class="k">class</span> <span class="nc">TaskInstance</span><span class="p">(</span><span class="n">Base</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Task instances store the state of a task instance. This table is the</span>
<span class="sd"> authority and single source of truth around what tasks have run and the</span>
<span class="sd"> state they are in.</span>
-<span class="sd"> The SqlAchemy model doesn't have a SqlAlchemy foreign key to the task or</span>
+<span class="sd"> The SqlAlchemy model doesn't have a SqlAlchemy foreign key to the task or</span>
<span class="sd"> dag model deliberately to have more control over transactions.</span>
<span class="sd"> Database transactions on this table should insure double triggers and</span>
@@ -892,7 +938,8 @@
<span class="n">end_date</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">DateTime</span><span class="p">)</span>
<span class="n">duration</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Float</span><span class="p">)</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">20</span><span class="p">))</span>
- <span class="n">try_number</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
+ <span class="n">_try_number</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="s1">'try_number'</span><span class="p">,</span> <span class="n">Integer</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
+ <span class="n">max_tries</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">)</span>
<span class="n">hostname</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">1000</span><span class="p">))</span>
<span class="n">unixname</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">1000</span><span class="p">))</span>
<span class="n">job_id</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">)</span>
@@ -919,18 +966,42 @@
<span class="bp">self</span><span class="o">.</span><span class="n">pool</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">pool</span>
<span class="bp">self</span><span class="o">.</span><span class="n">priority_weight</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">priority_weight_total</span>
<span class="bp">self</span><span class="o">.</span><span class="n">try_number</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">max_tries</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">task</span><span class="o">.</span><span class="n">retries</span>
<span class="bp">self</span><span class="o">.</span><span class="n">unixname</span> <span class="o">=</span> <span class="n">getpass</span><span class="o">.</span><span class="n">getuser</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_as_user</span> <span class="o">=</span> <span class="n">task</span><span class="o">.</span><span class="n">run_as_user</span>
<span class="k">if</span> <span class="n">state</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">state</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hostname</span> <span class="o">=</span> <span class="s1">''</span>
<span class="bp">self</span><span class="o">.</span><span class="n">init_on_load</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_log</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"airflow.task"</span><span class="p">)</span>
- <span class="nd">@reconstructor</span>
-<div class="viewcode-block" id="TaskInstance.init_on_load"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance.init_on_load">[docs]</a> <span class="k">def</span> <span class="nf">init_on_load</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+<div class="viewcode-block" id="TaskInstance.init_on_load"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance.init_on_load">[docs]</a> <span class="nd">@reconstructor</span>
+ <span class="k">def</span> <span class="nf">init_on_load</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">""" Initialize the attributes that aren't stored in the DB. """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">test_mode</span> <span class="o">=</span> <span class="kc">False</span> <span class="c1"># can be changed when calling 'run'</span></div>
+ <span class="nd">@property</span>
+ <span class="k">def</span> <span class="nf">try_number</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Return the try number that this task number will be when it is acutally</span>
+<span class="sd"> run.</span>
+
+<span class="sd"> If the TI is currently running, this will match the column in the</span>
+<span class="sd"> databse, in all othercases this will be incremenetd</span>
+<span class="sd"> """</span>
+ <span class="c1"># This is designed so that task logs end up in the right file.</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_try_number</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_try_number</span> <span class="o">+</span> <span class="mi">1</span>
+
+ <span class="nd">@try_number</span><span class="o">.</span><span class="n">setter</span>
+ <span class="k">def</span> <span class="nf">try_number</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_try_number</span> <span class="o">=</span> <span class="n">value</span>
+
+ <span class="nd">@property</span>
+ <span class="k">def</span> <span class="nf">next_try_number</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_try_number</span> <span class="o">+</span> <span class="mi">1</span>
+
<div class="viewcode-block" id="TaskInstance.command"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance.command">[docs]</a> <span class="k">def</span> <span class="nf">command</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">mark_success</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
@@ -1007,8 +1078,8 @@
<span class="n">pool</span><span class="o">=</span><span class="n">pool</span><span class="p">,</span>
<span class="n">cfg_path</span><span class="o">=</span><span class="n">cfg_path</span><span class="p">)</span></div>
- <span class="nd">@staticmethod</span>
-<div class="viewcode-block" id="TaskInstance.generate_command"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance.generate_command">[docs]</a> <span class="k">def</span> <span class="nf">generate_command</span><span class="p">(</span><span class="n">dag_id</span><span class="p">,</span>
+<div class="viewcode-block" id="TaskInstance.generate_command"><a class="viewcode-back" href="../../code.html#airflow.models.TaskInstance.generate_command">[docs]</a> <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">generate_command</span><span class="p">(</span><span class="n">
<TRUNCATED>
[13/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/mysql_hook.html
----------------------------------------------------------------------
diff --git a/_modules/mysql_hook.html b/_modules/mysql_hook.html
index a8f6aea..1045e28 100644
--- a/_modules/mysql_hook.html
+++ b/_modules/mysql_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -183,18 +188,22 @@
<div class="viewcode-block" id="MySqlHook"><a class="viewcode-back" href="../code.html#airflow.hooks.MySqlHook">[docs]</a><span class="k">class</span> <span class="nc">MySqlHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
- <span class="sd">'''</span>
+ <span class="sd">"""</span>
<span class="sd"> Interact with MySQL.</span>
<span class="sd"> You can specify charset in the extra field of your connection</span>
<span class="sd"> as ``{"charset": "utf8"}``. Also you can choose cursor as</span>
<span class="sd"> ``{"cursor": "SSCursor"}``. Refer to the MySQLdb.cursors for more details.</span>
-<span class="sd"> '''</span>
+<span class="sd"> """</span>
<span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'mysql_conn_id'</span>
<span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'mysql_default'</span>
<span class="n">supports_autocommit</span> <span class="o">=</span> <span class="kc">True</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MySqlHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s2">"schema"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
+
<div class="viewcode-block" id="MySqlHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.MySqlHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a mysql connection object</span>
@@ -202,17 +211,16 @@
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
<span class="n">conn_config</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">"user"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
- <span class="s2">"passwd"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span> <span class="ow">or</span> <span class="s1">''</span>
+ <span class="s2">"passwd"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span> <span class="ow">or</span> <span class="s1">''</span><span class="p">,</span>
+ <span class="s2">"host"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span> <span class="ow">or</span> <span class="s1">'localhost'</span><span class="p">,</span>
+ <span class="s2">"db"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="s1">''</span>
<span class="p">}</span>
- <span class="n">conn_config</span><span class="p">[</span><span class="s2">"host"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span> <span class="ow">or</span> <span class="s1">'localhost'</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
<span class="n">conn_config</span><span class="p">[</span><span class="s2">"port"</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3306</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">conn_config</span><span class="p">[</span><span class="s2">"port"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
- <span class="n">conn_config</span><span class="p">[</span><span class="s2">"db"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="s1">''</span>
-
<span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'charset'</span><span class="p">,</span> <span class="kc">False</span><span class="p">):</span>
<span class="n">conn_config</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">]</span>
<span class="k">if</span> <span class="p">(</span><span class="n">conn_config</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'utf8'</span> <span class="ow">or</span>\
@@ -248,17 +256,18 @@
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">conn</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Returns the MySQL literal of the cell as a string.</span>
+<span class="sd"> MySQLdb converts an argument to a literal when passing those seperately to execute.</span>
+<span class="sd"> Hence, this method does nothing.</span>
<span class="sd"> :param cell: The cell to insert into the table</span>
<span class="sd"> :type cell: object</span>
<span class="sd"> :param conn: The database connection</span>
<span class="sd"> :type conn: connection object</span>
-<span class="sd"> :return: The serialized cell</span>
-<span class="sd"> :rtype: str</span>
+<span class="sd"> :return: The same cell</span>
+<span class="sd"> :rtype: object</span>
<span class="sd"> """</span>
- <span class="k">return</span> <span class="n">conn</span><span class="o">.</span><span class="n">literal</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span></div>
+ <span class="k">return</span> <span class="n">cell</span></div>
</pre></div>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/mysql_operator.html
----------------------------------------------------------------------
diff --git a/_modules/mysql_operator.html b/_modules/mysql_operator.html
index a60fab2..6acf146 100644
--- a/_modules/mysql_operator.html
+++ b/_modules/mysql_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -175,9 +180,6 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-
-<span class="kn">import</span> <span class="nn">logging</span>
-
<span class="kn">from</span> <span class="nn">airflow.hooks.mysql_hook</span> <span class="k">import</span> <span class="n">MySqlHook</span>
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
@@ -193,6 +195,8 @@
<span class="sd"> :type sql: Can receive a str representing a sql statement,</span>
<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
+<span class="sd"> :param database: name of database which overwrite defined one in connection</span>
+<span class="sd"> :type database: string</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
@@ -202,16 +206,18 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">mysql_conn_id</span><span class="o">=</span><span class="s1">'mysql_default'</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
- <span class="n">autocommit</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">MySqlOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">autocommit</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">database</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MySqlOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
<span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">parameters</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">database</span> <span class="o">=</span> <span class="n">database</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">))</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">database</span><span class="p">)</span>
<span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span>
<span class="n">autocommit</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span><span class="p">,</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/mysql_to_hive.html
----------------------------------------------------------------------
diff --git a/_modules/mysql_to_hive.html b/_modules/mysql_to_hive.html
index cc8496b..007b81c 100644
--- a/_modules/mysql_to_hive.html
+++ b/_modules/mysql_to_hive.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>mysql_to_hive</li>
+
+ <li>mysql_to_hive</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,17 +181,16 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">chr</span>
-<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
-<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="kn">as</span> <span class="nn">csv</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">chr</span>
+<span class="kn">from</span> <span class="nn">collections</span> <span class="k">import</span> <span class="n">OrderedDict</span>
+<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="k">as</span> <span class="nn">csv</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">NamedTemporaryFile</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.mysql_hook</span> <span class="kn">import</span> <span class="n">MySqlHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveCliHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.mysql_hook</span> <span class="k">import</span> <span class="n">MySqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
<div class="viewcode-block" id="MySqlToHiveTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.MySqlToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">MySqlToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -197,6 +226,8 @@
<span class="sd"> :type mysql_conn_id: str</span>
<span class="sd"> :param hive_conn_id: destination hive connection</span>
<span class="sd"> :type hive_conn_id: str</span>
+<span class="sd"> :param tblproperties: TBLPROPERTIES of the hive table being created</span>
+<span class="sd"> :type tblproperties: dict</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
@@ -208,14 +239,15 @@
<span class="bp">self</span><span class="p">,</span>
<span class="n">sql</span><span class="p">,</span>
<span class="n">hive_table</span><span class="p">,</span>
- <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
- <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="nb">chr</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span>
<span class="n">mysql_conn_id</span><span class="o">=</span><span class="s1">'mysql_default'</span><span class="p">,</span>
<span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">MySqlToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MySqlToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
<span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
@@ -225,9 +257,10 @@
<span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span> <span class="o">=</span> <span class="n">tblproperties</span>
<span class="nd">@classmethod</span>
- <span class="k">def</span> <span class="nf">type_map</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">mysql_type</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">type_map</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">mysql_type</span><span class="p">):</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">constants</span><span class="o">.</span><span class="n">FIELD_TYPE</span>
<span class="n">d</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">t</span><span class="o">.</span><span class="n">BIT</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
@@ -246,7 +279,7 @@
<span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
<span class="n">mysql</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping MySQL query results to local file"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping MySQL query results to local file"</span><span class="p">)</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">mysql</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
@@ -259,7 +292,7 @@
<span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
<span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
<span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
@@ -267,10 +300,14 @@
<span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
<span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
- <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span></div>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -303,7 +340,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/postgres_hook.html
----------------------------------------------------------------------
diff --git a/_modules/postgres_hook.html b/_modules/postgres_hook.html
index 3ecfa69..b1e8237 100644
--- a/_modules/postgres_hook.html
+++ b/_modules/postgres_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>postgres_hook</li>
+
+ <li>postgres_hook</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -153,19 +183,27 @@
<span class="kn">import</span> <span class="nn">psycopg2</span>
<span class="kn">import</span> <span class="nn">psycopg2.extensions</span>
+<span class="kn">from</span> <span class="nn">contextlib</span> <span class="k">import</span> <span class="n">closing</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="k">import</span> <span class="n">DbApiHook</span>
<div class="viewcode-block" id="PostgresHook"><a class="viewcode-back" href="../code.html#airflow.hooks.PostgresHook">[docs]</a><span class="k">class</span> <span class="nc">PostgresHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
- <span class="sd">'''</span>
+ <span class="sd">"""</span>
<span class="sd"> Interact with Postgres.</span>
<span class="sd"> You can specify ssl parameters in the extra field of your connection</span>
<span class="sd"> as ``{"sslmode": "require", "sslcert": "/path/to/cert.pem", etc}``.</span>
-<span class="sd"> '''</span>
+
+<span class="sd"> Note: For Redshift, use keepalives_idle in the extra connection parameters</span>
+<span class="sd"> and set it to less than 300 seconds.</span>
+<span class="sd"> """</span>
<span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'postgres_conn_id'</span>
<span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'postgres_default'</span>
- <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="kc">True</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PostgresHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s2">"schema"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span><span class="p">)</span>
@@ -173,23 +211,51 @@
<span class="n">host</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
<span class="n">user</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
<span class="n">password</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">,</span>
- <span class="n">dbname</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
+ <span class="n">dbname</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
<span class="n">port</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
<span class="c1"># check for ssl parameters in conn.extra</span>
<span class="k">for</span> <span class="n">arg_name</span><span class="p">,</span> <span class="n">arg_val</span> <span class="ow">in</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
- <span class="k">if</span> <span class="n">arg_name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'sslmode'</span><span class="p">,</span> <span class="s1">'sslcert'</span><span class="p">,</span> <span class="s1">'sslkey'</span><span class="p">,</span> <span class="s1">'sslrootcert'</span><span class="p">,</span> <span class="s1">'sslcrl'</span><span class="p">]:</span>
+ <span class="k">if</span> <span class="n">arg_name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'sslmode'</span><span class="p">,</span> <span class="s1">'sslcert'</span><span class="p">,</span> <span class="s1">'sslkey'</span><span class="p">,</span>
+ <span class="s1">'sslrootcert'</span><span class="p">,</span> <span class="s1">'sslcrl'</span><span class="p">,</span> <span class="s1">'application_name'</span><span class="p">,</span>
+ <span class="s1">'keepalives_idle'</span><span class="p">]:</span>
<span class="n">conn_args</span><span class="p">[</span><span class="n">arg_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">arg_val</span>
+
<span class="n">psycopg2_conn</span> <span class="o">=</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="o">**</span><span class="n">conn_args</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">psycopg2_conn</span><span class="o">.</span><span class="n">server_version</span> <span class="o"><</span> <span class="mi">70400</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">True</span>
<span class="k">return</span> <span class="n">psycopg2_conn</span>
+<div class="viewcode-block" id="PostgresHook.copy_expert"><a class="viewcode-back" href="../code.html#airflow.hooks.PostgresHook.copy_expert">[docs]</a> <span class="k">def</span> <span class="nf">copy_expert</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="nb">open</span><span class="o">=</span><span class="nb">open</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Executes SQL using psycopg2 copy_expert method</span>
+<span class="sd"> Necessary to execute COPY command without access to a superuser</span>
+<span class="sd"> '''</span>
+ <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">())</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">())</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">copy_expert</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span></div>
+
<span class="nd">@staticmethod</span>
- <span class="k">def</span> <span class="nf">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">):</span>
- <span class="k">return</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">extensions</span><span class="o">.</span><span class="n">adapt</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span><span class="o">.</span><span class="n">getquoted</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span></div>
+ <span class="k">def</span> <span class="nf">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">conn</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Postgresql will adapt all arguments to the execute() method internally,</span>
+<span class="sd"> hence we return cell without any conversion.</span>
+
+<span class="sd"> See http://initd.org/psycopg/docs/advanced.html#adapting-new-types for</span>
+<span class="sd"> more information.</span>
+
+<span class="sd"> :param cell: The cell to insert into the table</span>
+<span class="sd"> :type cell: object</span>
+<span class="sd"> :param conn: The database connection</span>
+<span class="sd"> :type conn: connection object</span>
+<span class="sd"> :return: The cell</span>
+<span class="sd"> :rtype: object</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="n">cell</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -222,7 +288,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/postgres_operator.html
----------------------------------------------------------------------
diff --git a/_modules/postgres_operator.html b/_modules/postgres_operator.html
index 5ade37d..61cf4a2 100644
--- a/_modules/postgres_operator.html
+++ b/_modules/postgres_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>postgres_operator</li>
+
+ <li>postgres_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -150,12 +180,9 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.hooks.postgres_hook</span> <span class="kn">import</span> <span class="n">PostgresHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.postgres_hook</span> <span class="k">import</span> <span class="n">PostgresHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
<div class="viewcode-block" id="PostgresOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PostgresOperator">[docs]</a><span class="k">class</span> <span class="nc">PostgresOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -168,6 +195,8 @@
<span class="sd"> :type sql: Can receive a str representing a sql statement,</span>
<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
+<span class="sd"> :param database: name of database which overwrite defined one in connection</span>
+<span class="sd"> :type database: string</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
@@ -177,22 +206,28 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span>
- <span class="n">postgres_conn_id</span><span class="o">=</span><span class="s1">'postgres_default'</span><span class="p">,</span> <span class="n">autocommit</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">postgres_conn_id</span><span class="o">=</span><span class="s1">'postgres_default'</span><span class="p">,</span> <span class="n">autocommit</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">database</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">PostgresOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PostgresOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span> <span class="o">=</span> <span class="n">postgres_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
<span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">parameters</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">database</span> <span class="o">=</span> <span class="n">database</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">))</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">PostgresHook</span><span class="p">(</span><span class="n">postgres_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">PostgresHook</span><span class="p">(</span><span class="n">postgres_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">database</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -225,7 +260,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/presto_check_operator.html
----------------------------------------------------------------------
diff --git a/_modules/presto_check_operator.html b/_modules/presto_check_operator.html
index 325c884..58be6bb 100644
--- a/_modules/presto_check_operator.html
+++ b/_modules/presto_check_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -220,7 +225,7 @@
<span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span>
<span class="n">presto_conn_id</span><span class="o">=</span><span class="s1">'presto_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">PrestoCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PrestoCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span> <span class="o">=</span> <span class="n">presto_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
@@ -244,7 +249,7 @@
<span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">pass_value</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">presto_conn_id</span><span class="o">=</span><span class="s1">'presto_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">PrestoValueCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PrestoValueCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">,</span> <span class="n">pass_value</span><span class="o">=</span><span class="n">pass_value</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="n">tolerance</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span> <span class="o">=</span> <span class="n">presto_conn_id</span>
@@ -275,7 +280,7 @@
<span class="n">date_filter_column</span><span class="o">=</span><span class="s1">'ds'</span><span class="p">,</span> <span class="n">days_back</span><span class="o">=-</span><span class="mi">7</span><span class="p">,</span>
<span class="n">presto_conn_id</span><span class="o">=</span><span class="s1">'presto_default'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">PrestoIntervalCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PrestoIntervalCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">table</span><span class="o">=</span><span class="n">table</span><span class="p">,</span> <span class="n">metrics_thresholds</span><span class="o">=</span><span class="n">metrics_thresholds</span><span class="p">,</span>
<span class="n">date_filter_column</span><span class="o">=</span><span class="n">date_filter_column</span><span class="p">,</span> <span class="n">days_back</span><span class="o">=</span><span class="n">days_back</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/presto_hook.html
----------------------------------------------------------------------
diff --git a/_modules/presto_hook.html b/_modules/presto_hook.html
index 573ddc8..717b871 100644
--- a/_modules/presto_hook.html
+++ b/_modules/presto_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -177,16 +182,12 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">pyhive</span> <span class="k">import</span> <span class="n">presto</span>
<span class="kn">from</span> <span class="nn">pyhive.exc</span> <span class="k">import</span> <span class="n">DatabaseError</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="k">import</span> <span class="n">DbApiHook</span>
-<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"pyhive"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
-
-
<span class="k">class</span> <span class="nc">PrestoException</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
<span class="k">pass</span>
@@ -218,6 +219,19 @@
<span class="k">def</span> <span class="nf">_strip_sql</span><span class="p">(</span><span class="n">sql</span><span class="p">):</span>
<span class="k">return</span> <span class="n">sql</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s1">';'</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">_get_pretty_exception_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">e</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Parses some DatabaseError to provide a better error message</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="p">(</span><span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">'message'</span><span class="p">)</span>
+ <span class="ow">and</span> <span class="s1">'errorName'</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">message</span>
+ <span class="ow">and</span> <span class="s1">'message'</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">):</span>
+ <span class="k">return</span> <span class="p">(</span><span class="s1">'</span><span class="si">{name}</span><span class="s1">: </span><span class="si">{message}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">[</span><span class="s1">'errorName'</span><span class="p">],</span>
+ <span class="n">message</span><span class="o">=</span><span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">[</span><span class="s1">'message'</span><span class="p">]))</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+
<div class="viewcode-block" id="PrestoHook.get_records"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_records">[docs]</a> <span class="k">def</span> <span class="nf">get_records</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get a set of records from Presto</span>
@@ -226,14 +240,7 @@
<span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">PrestoHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span>
<span class="k">except</span> <span class="n">DatabaseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="k">if</span> <span class="p">(</span><span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">'message'</span><span class="p">)</span> <span class="ow">and</span>
- <span class="s1">'errorName'</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">message</span> <span class="ow">and</span>
- <span class="s1">'message'</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">):</span>
- <span class="c1"># Use the structured error data in the raised exception</span>
- <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="s1">'</span><span class="si">{name}</span><span class="s1">: </span><span class="si">{message}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="n">name</span><span class="o">=</span><span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">[</span><span class="s1">'errorName'</span><span class="p">],</span> <span class="n">message</span><span class="o">=</span><span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">[</span><span class="s1">'message'</span><span class="p">]))</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span></div>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_parse_exception_message</span><span class="p">(</span><span class="n">e</span><span class="p">))</span></div>
<div class="viewcode-block" id="PrestoHook.get_first"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_first">[docs]</a> <span class="k">def</span> <span class="nf">get_first</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -244,7 +251,7 @@
<span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">PrestoHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_first</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span>
<span class="k">except</span> <span class="n">DatabaseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="n">e</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s1">'message'</span><span class="p">])</span></div>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_parse_exception_message</span><span class="p">(</span><span class="n">e</span><span class="p">))</span></div>
<div class="viewcode-block" id="PrestoHook.get_pandas_df"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -256,7 +263,7 @@
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
<span class="k">except</span> <span class="n">DatabaseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="n">e</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="s1">'message'</span><span class="p">])</span>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_parse_exception_message</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
<span class="n">column_descriptions</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">description</span>
<span class="k">if</span> <span class="n">data</span><span class="p">:</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
[16/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/ftp_hook.html
----------------------------------------------------------------------
diff --git a/_modules/ftp_hook.html b/_modules/ftp_hook.html
index 894f72f..b8225ed 100644
--- a/_modules/ftp_hook.html
+++ b/_modules/ftp_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -179,11 +184,12 @@
<span class="kn">import</span> <span class="nn">datetime</span>
<span class="kn">import</span> <span class="nn">ftplib</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">os.path</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
<span class="k">def</span> <span class="nf">mlsd</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">""</span><span class="p">,</span> <span class="n">facts</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -218,7 +224,7 @@
<span class="k">yield</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">entry</span><span class="p">)</span>
-<div class="viewcode-block" id="FTPHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook">[docs]</a><span class="k">class</span> <span class="nc">FTPHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+<div class="viewcode-block" id="FTPHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook">[docs]</a><span class="k">class</span> <span class="nc">FTPHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Interact with FTP.</span>
@@ -330,10 +336,9 @@
<span class="n">remote_path</span><span class="p">,</span> <span class="n">remote_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">remote_full_path</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">remote_path</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Retrieving file from FTP: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">remote_full_path</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Retrieving file from FTP: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">remote_full_path</span><span class="p">)</span>
<span class="n">conn</span><span class="o">.</span><span class="n">retrbinary</span><span class="p">(</span><span class="s1">'RETR </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">remote_file_name</span><span class="p">,</span> <span class="n">output_handle</span><span class="o">.</span><span class="n">write</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Finished retrieving file from FTP: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="n">remote_full_path</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Finished retrieving file from FTP: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">remote_full_path</span><span class="p">)</span>
<span class="k">if</span> <span class="n">is_path</span><span class="p">:</span>
<span class="n">output_handle</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
@@ -390,7 +395,12 @@
<span class="k">def</span> <span class="nf">get_mod_time</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
<span class="n">ftp_mdtm</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">sendcmd</span><span class="p">(</span><span class="s1">'MDTM '</span> <span class="o">+</span> <span class="n">path</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">ftp_mdtm</span><span class="p">[</span><span class="mi">4</span><span class="p">:],</span> <span class="s1">'%Y%m</span><span class="si">%d</span><span class="s1">%H%M%S'</span><span class="p">)</span></div>
+ <span class="n">time_val</span> <span class="o">=</span> <span class="n">ftp_mdtm</span><span class="p">[</span><span class="mi">4</span><span class="p">:]</span>
+ <span class="c1"># time_val optionally has microseconds</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">time_val</span><span class="p">,</span> <span class="s2">"%Y%m</span><span class="si">%d</span><span class="s2">%H%M%S.</span><span class="si">%f</span><span class="s2">"</span><span class="p">)</span>
+ <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">time_val</span><span class="p">,</span> <span class="s1">'%Y%m</span><span class="si">%d</span><span class="s1">%H%M%S'</span><span class="p">)</span></div>
<span class="k">class</span> <span class="nc">FTPSHook</span><span class="p">(</span><span class="n">FTPHook</span><span class="p">):</span>
@@ -401,9 +411,14 @@
<span class="sd"> """</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">params</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ftp_conn_id</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">params</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
+ <span class="n">ftplib</span><span class="o">.</span><span class="n">FTP_TLS</span><span class="o">.</span><span class="n">port</span><span class="o">=</span><span class="n">params</span><span class="o">.</span><span class="n">port</span>
+
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">ftplib</span><span class="o">.</span><span class="n">FTP_TLS</span><span class="p">(</span>
<span class="n">params</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">params</span><span class="o">.</span><span class="n">login</span><span class="p">,</span> <span class="n">params</span><span class="o">.</span><span class="n">password</span>
<span class="p">)</span>
+
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>
</pre></div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/gcs_hook.html
----------------------------------------------------------------------
diff --git a/_modules/gcs_hook.html b/_modules/gcs_hook.html
index c118b86..46ee89d 100644
--- a/_modules/gcs_hook.html
+++ b/_modules/gcs_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>gcs_hook</li>
+
+ <li>gcs_hook</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,14 +181,11 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="c1">#</span>
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span>
+<span class="kn">from</span> <span class="nn">apiclient.http</span> <span class="k">import</span> <span class="n">MediaFileUpload</span>
+<span class="kn">from</span> <span class="nn">googleapiclient</span> <span class="k">import</span> <span class="n">errors</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="kn">import</span> <span class="n">GoogleCloudBaseHook</span>
-<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="kn">import</span> <span class="n">build</span>
-<span class="kn">from</span> <span class="nn">apiclient.http</span> <span class="kn">import</span> <span class="n">MediaFileUpload</span>
-
-<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"google_cloud_storage"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
<div class="viewcode-block" id="GoogleCloudStorageHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
@@ -169,8 +196,9 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_storage_default'</span><span class="p">,</span>
- <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="p">)</span>
<div class="viewcode-block" id="GoogleCloudStorageHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -179,7 +207,54 @@
<span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
<span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'storage'</span><span class="p">,</span> <span class="s1">'v1'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
-<div class="viewcode-block" id="GoogleCloudStorageHook.download"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.download">[docs]</a> <span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.copy"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.copy">[docs]</a> <span class="k">def</span> <span class="nf">copy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source_bucket</span><span class="p">,</span> <span class="n">source_object</span><span class="p">,</span> <span class="n">destination_bucket</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">destination_object</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Copies an object from a bucket to another, with renaming if requested.</span>
+
+<span class="sd"> destination_bucket or destination_object can be omitted, in which case</span>
+<span class="sd"> source bucket/object is used, but not both.</span>
+
+<span class="sd"> :param bucket: The bucket of the object to copy from.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The object to copy.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param destination_bucket: The destination of the object to copied to.</span>
+<span class="sd"> Can be omitted; then the same bucket is used.</span>
+<span class="sd"> :type destination_bucket: string</span>
+<span class="sd"> :param destination_object: The (renamed) path of the object if given.</span>
+<span class="sd"> Can be omitted; then the same name is used.</span>
+<span class="sd"> """</span>
+ <span class="n">destination_bucket</span> <span class="o">=</span> <span class="n">destination_bucket</span> <span class="ow">or</span> <span class="n">source_bucket</span>
+ <span class="n">destination_object</span> <span class="o">=</span> <span class="n">destination_object</span> <span class="ow">or</span> <span class="n">source_object</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">source_bucket</span> <span class="o">==</span> <span class="n">destination_bucket</span> <span class="ow">and</span>
+ <span class="n">source_object</span> <span class="o">==</span> <span class="n">destination_object</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s1">'Either source/destination bucket or source/destination object '</span>
+ <span class="s1">'must be different, not both the same: bucket=</span><span class="si">%s</span><span class="s1">, object=</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span>
+ <span class="p">(</span><span class="n">source_bucket</span><span class="p">,</span> <span class="n">source_object</span><span class="p">))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">source_bucket</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">source_object</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'source_bucket and source_object cannot be empty.'</span><span class="p">)</span>
+
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">copy</span><span class="p">(</span><span class="n">sourceBucket</span><span class="o">=</span><span class="n">source_bucket</span><span class="p">,</span> <span class="n">sourceObject</span><span class="o">=</span><span class="n">source_object</span><span class="p">,</span>
+ <span class="n">destinationBucket</span><span class="o">=</span><span class="n">destination_bucket</span><span class="p">,</span>
+ <span class="n">destinationObject</span><span class="o">=</span><span class="n">destination_object</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div>
+
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.download"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.download">[docs]</a> <span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get a file from Google Cloud Storage.</span>
@@ -198,11 +273,13 @@
<span class="c1"># Write the file to local file path, if requested.</span>
<span class="k">if</span> <span class="n">filename</span><span class="p">:</span>
- <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_fd</span><span class="p">:</span>
+ <span class="n">write_argument</span> <span class="o">=</span> <span class="s1">'wb'</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">downloaded_file_bytes</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">)</span> <span class="k">else</span> <span class="s1">'w'</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">write_argument</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_fd</span><span class="p">:</span>
<span class="n">file_fd</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">downloaded_file_bytes</span><span class="p">)</span>
<span class="k">return</span> <span class="n">downloaded_file_bytes</span></div>
+ <span class="c1"># pylint:disable=redefined-builtin</span>
<div class="viewcode-block" id="GoogleCloudStorageHook.upload"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.upload">[docs]</a> <span class="k">def</span> <span class="nf">upload</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mime_type</span><span class="o">=</span><span class="s1">'application/octet-stream'</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Uploads a local file to Google Cloud Storage.</span>
@@ -221,10 +298,146 @@
<span class="n">response</span> <span class="o">=</span> <span class="n">service</span> \
<span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
<span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="nb">object</span><span class="p">,</span> <span class="n">media_body</span><span class="o">=</span><span class="n">media</span><span class="p">)</span> \
- <span class="o">.</span><span class="n">execute</span><span class="p">()</span></div></div>
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span></div>
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.exists"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.exists">[docs]</a> <span class="k">def</span> <span class="nf">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks for the existence of a file in Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The Google cloud storage bucket where the object is.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The name of the object to check in the Google cloud</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div>
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.is_updated_after"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.is_updated_after">[docs]</a> <span class="k">def</span> <span class="nf">is_updated_after</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">ts</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks if an object is updated in Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The Google cloud storage bucket where the object is.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The name of the object to check in the Google cloud</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param ts: The timestamp to check against.</span>
+<span class="sd"> :type ts: datetime</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="p">(</span><span class="n">service</span>
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">)</span>
+ <span class="o">.</span><span class="n">execute</span><span class="p">())</span>
+
+ <span class="k">if</span> <span class="s1">'updated'</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="kn">import</span> <span class="nn">dateutil.parser</span>
+ <span class="kn">import</span> <span class="nn">dateutil.tz</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">ts</span><span class="o">.</span><span class="n">tzinfo</span><span class="p">:</span>
+ <span class="n">ts</span> <span class="o">=</span> <span class="n">ts</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">tzinfo</span><span class="o">=</span><span class="n">dateutil</span><span class="o">.</span><span class="n">tz</span><span class="o">.</span><span class="n">tzutc</span><span class="p">())</span>
+
+ <span class="n">updated</span> <span class="o">=</span> <span class="n">dateutil</span><span class="o">.</span><span class="n">parser</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">response</span><span class="p">[</span><span class="s1">'updated'</span><span class="p">])</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Verify object date: </span><span class="si">%s</span><span class="s2"> > </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">updated</span><span class="p">,</span> <span class="n">ts</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">updated</span> <span class="o">></span> <span class="n">ts</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">True</span>
+
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">raise</span>
+
+ <span class="k">return</span> <span class="kc">False</span></div>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.delete"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.delete">[docs]</a> <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">generation</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Delete an object if versioning is not enabled for the bucket, or if generation</span>
+<span class="sd"> parameter is used.</span>
+
+<span class="sd"> :param bucket: name of the bucket, where the object resides</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: name of the object to delete</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param generation: if present, permanently delete the object of this generation</span>
+<span class="sd"> :type generation: string</span>
+<span class="sd"> :return: True if succeeded</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">,</span> <span class="n">generation</span><span class="o">=</span><span class="n">generation</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.list"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.list">[docs]</a> <span class="k">def</span> <span class="nf">list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="n">versions</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">maxResults</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> List all objects from the bucket with the give string prefix in name</span>
+
+<span class="sd"> :param bucket: bucket name</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param versions: if true, list all versions of the objects</span>
+<span class="sd"> :type versions: boolean</span>
+<span class="sd"> :param maxResults: max count of items to return in a single page of responses</span>
+<span class="sd"> :type maxResults: integer</span>
+<span class="sd"> :param prefix: prefix string which filters objects whose name begin with this prefix</span>
+<span class="sd"> :type prefix: string</span>
+<span class="sd"> :return: a stream of object names matching the filtering criteria</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="n">ids</span> <span class="o">=</span> <span class="nb">list</span><span class="p">()</span>
+ <span class="n">pageToken</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="k">while</span><span class="p">(</span><span class="kc">True</span><span class="p">):</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">objects</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span>
+ <span class="n">versions</span><span class="o">=</span><span class="n">versions</span><span class="p">,</span>
+ <span class="n">maxResults</span><span class="o">=</span><span class="n">maxResults</span><span class="p">,</span>
+ <span class="n">pageToken</span><span class="o">=</span><span class="n">pageToken</span><span class="p">,</span>
+ <span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="s1">'items'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No items found for prefix: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">prefix</span><span class="p">)</span>
+ <span class="k">break</span>
+
+ <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'items'</span><span class="p">]:</span>
+ <span class="k">if</span> <span class="n">item</span> <span class="ow">and</span> <span class="s1">'name'</span> <span class="ow">in</span> <span class="n">item</span><span class="p">:</span>
+ <span class="n">ids</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+
+ <span class="k">if</span> <span class="s1">'nextPageToken'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="c1"># no further pages of results, so stop the loop</span>
+ <span class="k">break</span>
+
+ <span class="n">pageToken</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="s1">'nextPageToken'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">pageToken</span><span class="p">:</span>
+ <span class="c1"># empty next page token</span>
+ <span class="k">break</span>
+ <span class="k">return</span> <span class="n">ids</span></div></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -257,7 +470,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/generic_transfer.html
----------------------------------------------------------------------
diff --git a/_modules/generic_transfer.html b/_modules/generic_transfer.html
index 71c5825..77f3de1 100644
--- a/_modules/generic_transfer.html
+++ b/_modules/generic_transfer.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -175,9 +180,6 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-
-<span class="kn">import</span> <span class="nn">logging</span>
-
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
@@ -190,7 +192,7 @@
<span class="sd"> needs to expose a `get_records` method, and the destination a</span>
<span class="sd"> `insert_rows` method.</span>
-<span class="sd"> This is mean to be used on small-ish datasets that fit in memory.</span>
+<span class="sd"> This is meant to be used on small-ish datasets that fit in memory.</span>
<span class="sd"> :param sql: SQL query to execute against the source database</span>
<span class="sd"> :type sql: str</span>
@@ -218,7 +220,7 @@
<span class="n">destination_conn_id</span><span class="p">,</span>
<span class="n">preoperator</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">GenericTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GenericTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">destination_table</span> <span class="o">=</span> <span class="n">destination_table</span>
<span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span> <span class="o">=</span> <span class="n">source_conn_id</span>
@@ -228,17 +230,17 @@
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">source_hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span><span class="p">))</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing: </span><span class="se">\n</span><span class="s2">"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing: </span><span class="se">\n</span><span class="s2"> </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">source_hook</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="n">destination_hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_conn_id</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running preoperator"</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running preoperator"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">)</span>
<span class="n">destination_hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_conn_id</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">destination_conn_id</span><span class="p">)</span>
<span class="n">destination_hook</span><span class="o">.</span><span class="n">insert_rows</span><span class="p">(</span><span class="n">table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_table</span><span class="p">,</span> <span class="n">rows</span><span class="o">=</span><span class="n">results</span><span class="p">)</span></div>
</pre></div>
[21/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/operators/s3_to_hive_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/s3_to_hive_operator.html b/_modules/airflow/operators/s3_to_hive_operator.html
new file mode 100644
index 0000000..be79923
--- /dev/null
+++ b/_modules/airflow/operators/s3_to_hive_operator.html
@@ -0,0 +1,482 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.operators.s3_to_hive_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.operators.s3_to_hive_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.operators.s3_to_hive_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">next</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">zip</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="k">import</span> <span class="n">TemporaryDirectory</span>
+<span class="kn">import</span> <span class="nn">gzip</span>
+<span class="kn">import</span> <span class="nn">bz2</span>
+<span class="kn">import</span> <span class="nn">tempfile</span>
+<span class="kn">import</span> <span class="nn">os</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveCliHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.compression</span> <span class="k">import</span> <span class="n">uncompress_file</span>
+
+
+<div class="viewcode-block" id="S3ToHiveTransfer"><a class="viewcode-back" href="../../../integration.html#airflow.operators.s3_to_hive_operator.S3ToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">S3ToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from S3 to Hive. The operator downloads a file from S3,</span>
+<span class="sd"> stores the file locally before loading it into a Hive table.</span>
+<span class="sd"> If the ``create`` or ``recreate`` arguments are set to ``True``,</span>
+<span class="sd"> a ``CREATE TABLE`` and ``DROP TABLE`` statements are generated.</span>
+<span class="sd"> Hive data types are inferred from the cursor's metadata from.</span>
+
+<span class="sd"> Note that the table generated in Hive uses ``STORED AS textfile``</span>
+<span class="sd"> which isn't the most efficient serialization format. If a</span>
+<span class="sd"> large amount of data is loaded and/or if the tables gets</span>
+<span class="sd"> queried considerably, you may want to use this operator only to</span>
+<span class="sd"> stage the data into a temporary table before loading it into its</span>
+<span class="sd"> final destination using a ``HiveOperator``.</span>
+
+<span class="sd"> :param s3_key: The key to be retrieved from S3</span>
+<span class="sd"> :type s3_key: str</span>
+<span class="sd"> :param field_dict: A dictionary of the fields name in the file</span>
+<span class="sd"> as keys and their Hive types as values</span>
+<span class="sd"> :type field_dict: dict</span>
+<span class="sd"> :param hive_table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type hive_table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
+<span class="sd"> execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param partition: target partition as a dict of partition columns</span>
+<span class="sd"> and values</span>
+<span class="sd"> :type partition: dict</span>
+<span class="sd"> :param headers: whether the file contains column names on the first</span>
+<span class="sd"> line</span>
+<span class="sd"> :type headers: bool</span>
+<span class="sd"> :param check_headers: whether the column names on the first line should be</span>
+<span class="sd"> checked against the keys of field_dict</span>
+<span class="sd"> :type check_headers: bool</span>
+<span class="sd"> :param wildcard_match: whether the s3_key should be interpreted as a Unix</span>
+<span class="sd"> wildcard pattern</span>
+<span class="sd"> :type wildcard_match: bool</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param aws_conn_id: source s3 connection</span>
+<span class="sd"> :type aws_conn_id: str</span>
+<span class="sd"> :param hive_cli_conn_id: destination hive connection</span>
+<span class="sd"> :type hive_cli_conn_id: str</span>
+<span class="sd"> :param input_compressed: Boolean to determine if file decompression is</span>
+<span class="sd"> required to process headers</span>
+<span class="sd"> :type input_compressed: bool</span>
+<span class="sd"> :param tblproperties: TBLPROPERTIES of the hive table being created</span>
+<span class="sd"> :type tblproperties: dict</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'s3_key'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#a0e08c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">s3_key</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">check_headers</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">wildcard_match</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="n">input_compressed</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3ToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span> <span class="o">=</span> <span class="n">s3_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span> <span class="o">=</span> <span class="n">field_dict</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create</span> <span class="o">=</span> <span class="n">create</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">recreate</span> <span class="o">=</span> <span class="n">recreate</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span> <span class="o">=</span> <span class="n">check_headers</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">input_compressed</span> <span class="o">=</span> <span class="n">input_compressed</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span> <span class="o">=</span> <span class="n">tblproperties</span>
+
+ <span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span> <span class="ow">and</span>
+ <span class="ow">not</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">)):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"To check_headers provide "</span> <span class="o">+</span>
+ <span class="s2">"field_dict and headers"</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="c1"># Downloading file from S3</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Downloading S3 file"</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"No key matches </span><span class="si">{0}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
+ <span class="n">s3_key_object</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s2">"The key </span><span class="si">{0}</span><span class="s2"> does not exists"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
+ <span class="n">s3_key_object</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">)</span>
+ <span class="n">root</span><span class="p">,</span> <span class="n">file_ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">s3_key_object</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'tmps32hive_'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">,</span>\
+ <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="n">mode</span><span class="o">=</span><span class="s2">"wb"</span><span class="p">,</span>
+ <span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">,</span>
+ <span class="n">suffix</span><span class="o">=</span><span class="n">file_ext</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping S3 key </span><span class="si">{0}</span><span class="s2"> contents to local file </span><span class="si">{1}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">s3_key_object</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
+ <span class="n">s3_key_object</span><span class="o">.</span><span class="n">download_fileobj</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file </span><span class="si">%s</span><span class="s2"> into Hive"</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="c1"># Decompressing file</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">input_compressed</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Uncompressing file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
+ <span class="n">fn_uncompressed</span> <span class="o">=</span> <span class="n">uncompress_file</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="n">file_ext</span><span class="p">,</span>
+ <span class="n">tmp_dir</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Uncompressed to </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">fn_uncompressed</span><span class="p">)</span>
+ <span class="c1"># uncompressed file available now so deleting</span>
+ <span class="c1"># compressed file to save disk space</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">fn_uncompressed</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span>
+
+ <span class="c1"># Testing if header matches field_dict</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Matching file header against field_dict"</span><span class="p">)</span>
+ <span class="n">header_list</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_top_row_as_list</span><span class="p">(</span><span class="n">fn_uncompressed</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_match_headers</span><span class="p">(</span><span class="n">header_list</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Header check failed"</span><span class="p">)</span>
+
+ <span class="c1"># Deleting top header row</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Removing header from file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">fn_uncompressed</span><span class="p">)</span>
+ <span class="n">headless_file</span> <span class="o">=</span> <span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_delete_top_row_and_compress</span><span class="p">(</span><span class="n">fn_uncompressed</span><span class="p">,</span>
+ <span class="n">file_ext</span><span class="p">,</span>
+ <span class="n">tmp_dir</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Headless file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">headless_file</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file </span><span class="si">%s</span><span class="s2"> into Hive"</span><span class="p">,</span> <span class="n">headless_file</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span><span class="n">headless_file</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_get_top_row_as_list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">):</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_name</span><span class="p">,</span> <span class="s1">'rt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">header_line</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
+ <span class="n">header_list</span> <span class="o">=</span> <span class="n">header_line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">header_list</span>
+
+ <span class="k">def</span> <span class="nf">_match_headers</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">header_list</span><span class="p">):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">header_list</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Unable to retrieve header row from file"</span><span class="p">)</span>
+ <span class="n">field_names</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">field_names</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="n">header_list</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Headers count mismatch"</span>
+ <span class="s2">"File headers:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{header_list}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">"Field names: </span><span class="se">\n</span><span class="s2"> </span><span class="si">{field_names}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="n">test_field_match</span> <span class="o">=</span> <span class="p">[</span><span class="n">h1</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">h2</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">h1</span><span class="p">,</span> <span class="n">h2</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">header_list</span><span class="p">,</span> <span class="n">field_names</span><span class="p">)]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="n">test_field_match</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Headers do not match field names"</span>
+ <span class="s2">"File headers:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{header_list}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">"Field names: </span><span class="se">\n</span><span class="s2"> </span><span class="si">{field_names}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">True</span>
+
+ <span class="k">def</span> <span class="nf">_delete_top_row_and_compress</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">input_file_name</span><span class="p">,</span>
+ <span class="n">output_file_ext</span><span class="p">,</span>
+ <span class="n">dest_dir</span><span class="p">):</span>
+ <span class="c1"># When output_file_ext is not defined, file is not compressed</span>
+ <span class="n">open_fn</span> <span class="o">=</span> <span class="nb">open</span>
+ <span class="k">if</span> <span class="n">output_file_ext</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'.gz'</span><span class="p">:</span>
+ <span class="n">open_fn</span> <span class="o">=</span> <span class="n">gzip</span><span class="o">.</span><span class="n">GzipFile</span>
+ <span class="k">elif</span> <span class="n">output_file_ext</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'.bz2'</span><span class="p">:</span>
+ <span class="n">open_fn</span> <span class="o">=</span> <span class="n">bz2</span><span class="o">.</span><span class="n">BZ2File</span>
+
+ <span class="n">os_fh_output</span><span class="p">,</span> <span class="n">fn_output</span> <span class="o">=</span> \
+ <span class="n">tempfile</span><span class="o">.</span><span class="n">mkstemp</span><span class="p">(</span><span class="n">suffix</span><span class="o">=</span><span class="n">output_file_ext</span><span class="p">,</span> <span class="nb">dir</span><span class="o">=</span><span class="n">dest_dir</span><span class="p">)</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">input_file_name</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_in</span><span class="p">,</span>\
+ <span class="n">open_fn</span><span class="p">(</span><span class="n">fn_output</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_out</span><span class="p">:</span>
+ <span class="n">f_in</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
+ <span class="nb">next</span><span class="p">(</span><span class="n">f_in</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f_in</span><span class="p">:</span>
+ <span class="n">f_out</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">fn_output</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[35/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
1.9.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/28a3eb60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/28a3eb60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/28a3eb60
Branch: refs/heads/asf-site
Commit: 28a3eb6000f0d877c8c8088facb82bf2ab169c8d
Parents: 69cff49
Author: Chris Riccomini <cr...@apache.org>
Authored: Wed Jan 3 09:47:55 2018 -0800
Committer: Chris Riccomini <cr...@apache.org>
Committed: Wed Jan 3 09:47:55 2018 -0800
----------------------------------------------------------------------
_modules/S3_hook.html | 455 ++-
.../contrib/executors/mesos_executor.html | 58 +-
.../airflow/contrib/hooks/bigquery_hook.html | 1279 ++++++++
.../airflow/contrib/hooks/datastore_hook.html | 410 +++
_modules/airflow/contrib/hooks/emr_hook.html | 295 ++
.../contrib/hooks/gcp_dataflow_hook.html | 395 +++
.../contrib/hooks/gcp_mlengine_hook.html | 500 ++++
_modules/airflow/contrib/hooks/gcs_hook.html | 264 +-
_modules/airflow/contrib/hooks/wasb_hook.html | 345 +++
.../operators/bigquery_check_operator.html | 358 +++
.../contrib/operators/bigquery_operator.html | 144 +-
.../contrib/operators/bigquery_to_bigquery.html | 316 ++
.../contrib/operators/bigquery_to_gcs.html | 157 +-
.../contrib/operators/databricks_operator.html | 487 +++
.../contrib/operators/dataflow_operator.html | 470 +++
.../contrib/operators/dataproc_operator.html | 1139 +++++++
.../airflow/contrib/operators/ecs_operator.html | 360 +++
.../operators/emr_add_steps_operator.html | 291 ++
.../operators/emr_create_job_flow_operator.html | 296 ++
.../emr_terminate_job_flow_operator.html | 285 ++
.../airflow/contrib/operators/file_to_wasb.html | 290 ++
.../operators/gcs_download_operator.html | 134 +-
.../airflow/contrib/operators/gcs_to_bq.html | 429 +++
.../contrib/operators/hipchat_operator.html | 18 +-
.../contrib/operators/mlengine_operator.html | 795 +++++
.../airflow/contrib/sensors/wasb_sensor.html | 327 ++
_modules/airflow/executors/local_executor.html | 20 +-
.../airflow/executors/sequential_executor.html | 13 +-
_modules/airflow/hooks/S3_hook.html | 489 +++
_modules/airflow/macros.html | 12 +-
_modules/airflow/macros/hive.html | 11 +-
_modules/airflow/models.html | 1466 ++++++---
_modules/airflow/operators/docker_operator.html | 197 +-
.../operators/redshift_to_s3_operator.html | 337 +++
.../operators/s3_file_transform_operator.html | 342 +++
.../airflow/operators/s3_to_hive_operator.html | 482 +++
_modules/airflow/operators/sensors.html | 168 +-
_modules/bash_operator.html | 42 +-
_modules/bigquery_hook.html | 544 +++-
_modules/dagrun_operator.html | 17 +-
_modules/dbapi_hook.html | 153 +-
_modules/druid_hook.html | 271 +-
_modules/dummy_operator.html | 9 +-
_modules/email_operator.html | 11 +-
_modules/ftp_hook.html | 29 +-
_modules/gcs_hook.html | 256 +-
_modules/generic_transfer.html | 24 +-
_modules/hive_hooks.html | 329 +-
_modules/hive_operator.html | 80 +-
_modules/hive_to_druid.html | 207 +-
_modules/hive_to_mysql.html | 86 +-
_modules/http_hook.html | 31 +-
_modules/http_operator.html | 17 +-
_modules/index.html | 53 +-
_modules/mysql_hook.html | 31 +-
_modules/mysql_operator.html | 22 +-
_modules/mysql_to_hive.html | 90 +-
_modules/postgres_hook.html | 105 +-
_modules/postgres_operator.html | 76 +-
_modules/presto_check_operator.html | 13 +-
_modules/presto_hook.html | 37 +-
_modules/python_operator.html | 297 +-
_modules/qubole_operator.html | 144 +-
_modules/s3_to_hive_operator.html | 259 +-
_modules/sensors.html | 178 +-
_modules/slack_operator.html | 77 +-
_modules/sqlite_hook.html | 7 +-
_sources/api.rst.txt | 20 +-
_sources/code.rst.txt | 14 +-
_sources/concepts.rst.txt | 10 +-
_sources/configuration.rst.txt | 63 +-
_sources/faq.rst.txt | 12 +-
_sources/installation.rst.txt | 124 +-
_sources/integration.rst.txt | 351 ++-
_sources/scheduler.rst.txt | 22 +-
_sources/security.rst.txt | 82 +-
_sources/start.rst.txt | 2 +-
_sources/tutorial.rst.txt | 19 +-
_static/basic.css | 15 +-
_static/doctools.js | 54 +-
_static/pygments.css | 4 +
_static/searchtools.js | 5 +-
_static/websupport.js | 2 +-
api.html | 28 +-
cli.html | 1413 +++++----
code.html | 2806 ++++++++++++++----
concepts.html | 105 +-
configuration.html | 83 +-
faq.html | 45 +-
genindex.html | 380 ++-
index.html | 192 +-
installation.html | 79 +-
integration.html | 1738 ++++++++++-
license.html | 7 +-
objects.inv | Bin 2159 -> 3480 bytes
plugins.html | 21 +-
profiling.html | 11 +-
project.html | 19 +-
py-modindex.html | 7 +-
scheduler.html | 68 +-
search.html | 7 +-
searchindex.js | 2 +-
security.html | 140 +-
start.html | 23 +-
tutorial.html | 85 +-
ui.html | 15 +-
106 files changed, 21632 insertions(+), 3770 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/S3_hook.html
----------------------------------------------------------------------
diff --git a/_modules/S3_hook.html b/_modules/S3_hook.html
index 40a14c3..ae1091a 100644
--- a/_modules/S3_hook.html
+++ b/_modules/S3_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>S3_hook</li>
+
+ <li>S3_hook</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,135 +181,21 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">division</span>
-<span class="kn">from</span> <span class="nn">future</span> <span class="kn">import</span> <span class="n">standard_library</span>
-<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
-<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.aws_hook</span> <span class="k">import</span> <span class="n">AwsHook</span>
+
+<span class="kn">from</span> <span class="nn">six</span> <span class="k">import</span> <span class="n">BytesIO</span>
+<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="k">import</span> <span class="n">urlparse</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">fnmatch</span>
-<span class="kn">import</span> <span class="nn">configparser</span>
-<span class="kn">import</span> <span class="nn">math</span>
-<span class="kn">import</span> <span class="nn">os</span>
-<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>
-<span class="kn">import</span> <span class="nn">warnings</span>
-
-<span class="kn">import</span> <span class="nn">boto</span>
-<span class="kn">from</span> <span class="nn">boto.s3.connection</span> <span class="kn">import</span> <span class="n">S3Connection</span>
-<span class="kn">from</span> <span class="nn">boto.sts</span> <span class="kn">import</span> <span class="n">STSConnection</span>
-<span class="n">boto</span><span class="o">.</span><span class="n">set_stream_logger</span><span class="p">(</span><span class="s1">'boto'</span><span class="p">)</span>
-<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"boto"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
-<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
-
-
-<span class="k">def</span> <span class="nf">_parse_s3_config</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">,</span> <span class="n">config_format</span><span class="o">=</span><span class="s1">'boto'</span><span class="p">,</span> <span class="n">profile</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+<div class="viewcode-block" id="S3Hook"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook">[docs]</a><span class="k">class</span> <span class="nc">S3Hook</span><span class="p">(</span><span class="n">AwsHook</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Parses a config file for s3 credentials. Can currently</span>
-<span class="sd"> parse boto, s3cmd.conf and AWS SDK config formats</span>
-
-<span class="sd"> :param config_file_name: path to the config file</span>
-<span class="sd"> :type config_file_name: str</span>
-<span class="sd"> :param config_format: config type. One of "boto", "s3cmd" or "aws".</span>
-<span class="sd"> Defaults to "boto"</span>
-<span class="sd"> :type config_format: str</span>
-<span class="sd"> :param profile: profile name in AWS type config file</span>
-<span class="sd"> :type profile: str</span>
+<span class="sd"> Interact with AWS S3, using the boto3 library.</span>
<span class="sd"> """</span>
- <span class="n">Config</span> <span class="o">=</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">()</span>
- <span class="k">if</span> <span class="n">Config</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">):</span> <span class="c1"># pragma: no cover</span>
- <span class="n">sections</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">sections</span><span class="p">()</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Couldn't read {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">))</span>
- <span class="c1"># Setting option names depending on file format</span>
- <span class="k">if</span> <span class="n">config_format</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
- <span class="n">config_format</span> <span class="o">=</span> <span class="s1">'boto'</span>
- <span class="n">conf_format</span> <span class="o">=</span> <span class="n">config_format</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
- <span class="k">if</span> <span class="n">conf_format</span> <span class="o">==</span> <span class="s1">'boto'</span><span class="p">:</span> <span class="c1"># pragma: no cover</span>
- <span class="k">if</span> <span class="n">profile</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="s1">'profile '</span> <span class="o">+</span> <span class="n">profile</span> <span class="ow">in</span> <span class="n">sections</span><span class="p">:</span>
- <span class="n">cred_section</span> <span class="o">=</span> <span class="s1">'profile '</span> <span class="o">+</span> <span class="n">profile</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">cred_section</span> <span class="o">=</span> <span class="s1">'Credentials'</span>
- <span class="k">elif</span> <span class="n">conf_format</span> <span class="o">==</span> <span class="s1">'aws'</span> <span class="ow">and</span> <span class="n">profile</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
- <span class="n">cred_section</span> <span class="o">=</span> <span class="n">profile</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">cred_section</span> <span class="o">=</span> <span class="s1">'default'</span>
- <span class="c1"># Option names</span>
- <span class="k">if</span> <span class="n">conf_format</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'boto'</span><span class="p">,</span> <span class="s1">'aws'</span><span class="p">):</span> <span class="c1"># pragma: no cover</span>
- <span class="n">key_id_option</span> <span class="o">=</span> <span class="s1">'aws_access_key_id'</span>
- <span class="n">secret_key_option</span> <span class="o">=</span> <span class="s1">'aws_secret_access_key'</span>
- <span class="c1"># security_token_option = 'aws_security_token'</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">key_id_option</span> <span class="o">=</span> <span class="s1">'access_key'</span>
- <span class="n">secret_key_option</span> <span class="o">=</span> <span class="s1">'secret_key'</span>
- <span class="c1"># Actual Parsing</span>
- <span class="k">if</span> <span class="n">cred_section</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">sections</span><span class="p">:</span>
- <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"This config file format is not recognized"</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="n">access_key</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="n">key_id_option</span><span class="p">)</span>
- <span class="n">secret_key</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="n">secret_key_option</span><span class="p">)</span>
- <span class="n">calling_format</span> <span class="o">=</span> <span class="bp">None</span>
- <span class="k">if</span> <span class="n">Config</span><span class="o">.</span><span class="n">has_option</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="s1">'calling_format'</span><span class="p">):</span>
- <span class="n">calling_format</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="s1">'calling_format'</span><span class="p">)</span>
- <span class="k">except</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Option Error in parsing s3 config file"</span><span class="p">)</span>
- <span class="k">raise</span>
- <span class="k">return</span> <span class="p">(</span><span class="n">access_key</span><span class="p">,</span> <span class="n">secret_key</span><span class="p">,</span> <span class="n">calling_format</span><span class="p">)</span>
-
-<div class="viewcode-block" id="S3Hook"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook">[docs]</a><span class="k">class</span> <span class="nc">S3Hook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Interact with S3. This class is a wrapper around the boto library.</span>
-<span class="sd"> """</span>
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span>
- <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn</span><span class="o">.</span><span class="n">extra_dejson</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">profile</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'profile'</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">calling_format</span> <span class="o">=</span> <span class="bp">None</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_conn</span> <span class="o">=</span> <span class="s1">'aws_secret_access_key'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_config_file</span> <span class="o">=</span> <span class="s1">'s3_config_file'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_default_to_boto</span> <span class="o">=</span> <span class="bp">False</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_conn</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_a_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_access_key_id'</span><span class="p">]</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_s_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_secret_access_key'</span><span class="p">]</span>
- <span class="k">if</span> <span class="s1">'calling_format'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">calling_format</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'calling_format'</span><span class="p">]</span>
- <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_config_file</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_config_file</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'s3_config_file'</span><span class="p">]</span>
- <span class="c1"># The format can be None and will default to boto in the parser</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_config_format</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'s3_config_format'</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_default_to_boto</span> <span class="o">=</span> <span class="bp">True</span>
- <span class="c1"># STS support for cross account resource access</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">_sts_conn_required</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'aws_account_id'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span> <span class="ow">or</span>
- <span class="s1">'role_arn'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">)</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sts_conn_required</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">role_arn</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'role_arn'</span><span class="p">)</span> <span class="ow">or</span>
- <span class="s2">"arn:aws:iam::"</span> <span class="o">+</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_account_id'</span><span class="p">]</span> <span class="o">+</span>
- <span class="s2">":role/"</span> <span class="o">+</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_iam_role'</span><span class="p">])</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">__getstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="n">pickled_dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">)</span>
- <span class="k">del</span> <span class="n">pickled_dict</span><span class="p">[</span><span class="s1">'connection'</span><span class="p">]</span>
- <span class="k">return</span> <span class="n">pickled_dict</span>
-
- <span class="k">def</span> <span class="nf">__setstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">d</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">[</span><span class="s1">'connection'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
-
- <span class="k">def</span> <span class="nf">_parse_s3_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s3url</span><span class="p">):</span>
- <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span>
- <span class="s1">'Please note: S3Hook._parse_s3_url() is now '</span>
- <span class="s1">'S3Hook.parse_s3_url() (no leading underscore).'</span><span class="p">,</span>
- <span class="ne">DeprecationWarning</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">s3url</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_client_type</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">parse_s3_url</span><span class="p">(</span><span class="n">s3url</span><span class="p">):</span>
@@ -291,47 +207,6 @@
<span class="n">key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
<span class="k">return</span> <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
-<div class="viewcode-block" id="S3Hook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Returns the boto S3Connection object.</span>
-<span class="sd"> """</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_default_to_boto</span><span class="p">:</span>
- <span class="k">return</span> <span class="n">S3Connection</span><span class="p">(</span><span class="n">profile_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
- <span class="n">a_key</span> <span class="o">=</span> <span class="n">s_key</span> <span class="o">=</span> <span class="bp">None</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_config_file</span><span class="p">:</span>
- <span class="n">a_key</span><span class="p">,</span> <span class="n">s_key</span><span class="p">,</span> <span class="n">calling_format</span> <span class="o">=</span> <span class="n">_parse_s3_config</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_config_file</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_config_format</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
- <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_conn</span><span class="p">:</span>
- <span class="n">a_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_a_key</span>
- <span class="n">s_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_s_key</span>
- <span class="n">calling_format</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">calling_format</span>
-
- <span class="k">if</span> <span class="n">calling_format</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
- <span class="n">calling_format</span> <span class="o">=</span> <span class="s1">'boto.s3.connection.SubdomainCallingFormat'</span>
-
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sts_conn_required</span><span class="p">:</span>
- <span class="n">sts_connection</span> <span class="o">=</span> <span class="n">STSConnection</span><span class="p">(</span><span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">a_key</span><span class="p">,</span>
- <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">s_key</span><span class="p">,</span>
- <span class="n">profile_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
- <span class="n">assumed_role_object</span> <span class="o">=</span> <span class="n">sts_connection</span><span class="o">.</span><span class="n">assume_role</span><span class="p">(</span>
- <span class="n">role_arn</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">role_arn</span><span class="p">,</span>
- <span class="n">role_session_name</span><span class="o">=</span><span class="s2">"Airflow_"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span>
- <span class="p">)</span>
- <span class="n">creds</span> <span class="o">=</span> <span class="n">assumed_role_object</span><span class="o">.</span><span class="n">credentials</span>
- <span class="n">connection</span> <span class="o">=</span> <span class="n">S3Connection</span><span class="p">(</span>
- <span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">creds</span><span class="o">.</span><span class="n">access_key</span><span class="p">,</span>
- <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">creds</span><span class="o">.</span><span class="n">secret_key</span><span class="p">,</span>
- <span class="n">calling_format</span><span class="o">=</span><span class="n">calling_format</span><span class="p">,</span>
- <span class="n">security_token</span><span class="o">=</span><span class="n">creds</span><span class="o">.</span><span class="n">session_token</span>
- <span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">connection</span> <span class="o">=</span> <span class="n">S3Connection</span><span class="p">(</span><span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">a_key</span><span class="p">,</span>
- <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">s_key</span><span class="p">,</span>
- <span class="n">calling_format</span><span class="o">=</span><span class="n">calling_format</span><span class="p">,</span>
- <span class="n">profile_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">connection</span></div>
-
<div class="viewcode-block" id="S3Hook.check_for_bucket"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_bucket">[docs]</a> <span class="k">def</span> <span class="nf">check_for_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Check if bucket_name exists.</span>
@@ -339,20 +214,35 @@
<span class="sd"> :param bucket_name: the name of the bucket</span>
<span class="sd"> :type bucket_name: str</span>
<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">lookup</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span></div>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">head_bucket</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span></div>
<div class="viewcode-block" id="S3Hook.get_bucket"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_bucket">[docs]</a> <span class="k">def</span> <span class="nf">get_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Returns a boto.s3.bucket.Bucket object</span>
+<span class="sd"> Returns a boto3.S3.Bucket object</span>
<span class="sd"> :param bucket_name: the name of the bucket</span>
<span class="sd"> :type bucket_name: str</span>
<span class="sd"> """</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span></div>
+ <span class="n">s3</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_resource_type</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">s3</span><span class="o">.</span><span class="n">Bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span></div>
-<div class="viewcode-block" id="S3Hook.list_keys"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.list_keys">[docs]</a> <span class="k">def</span> <span class="nf">list_keys</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+<div class="viewcode-block" id="S3Hook.check_for_prefix"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_prefix">[docs]</a> <span class="k">def</span> <span class="nf">check_for_prefix</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Lists keys in a bucket under prefix and not containing delimiter</span>
+<span class="sd"> Checks that a prefix exists in a bucket</span>
+<span class="sd"> """</span>
+ <span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span> <span class="o">+</span> <span class="n">delimiter</span> <span class="k">if</span> <span class="n">prefix</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">!=</span> <span class="n">delimiter</span> <span class="k">else</span> <span class="n">prefix</span>
+ <span class="n">prefix_split</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="sa">r</span><span class="s1">'(\w+[</span><span class="si">{d}</span><span class="s1">])$'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">d</span><span class="o">=</span><span class="n">delimiter</span><span class="p">),</span> <span class="n">prefix</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="n">previous_level</span> <span class="o">=</span> <span class="n">prefix_split</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">plist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_prefixes</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">previous_level</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">False</span> <span class="k">if</span> <span class="n">plist</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="n">prefix</span> <span class="ow">in</span> <span class="n">plist</span></div>
+
+<div class="viewcode-block" id="S3Hook.list_prefixes"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.list_prefixes">[docs]</a> <span class="k">def</span> <span class="nf">list_prefixes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lists prefixes in a bucket under prefix</span>
<span class="sd"> :param bucket_name: the name of the bucket</span>
<span class="sd"> :type bucket_name: str</span>
@@ -361,13 +251,14 @@
<span class="sd"> :param delimiter: the delimiter marks key hierarchy.</span>
<span class="sd"> :type delimiter: str</span>
<span class="sd"> """</span>
- <span class="n">b</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="n">keylist</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">))</span>
- <span class="k">return</span> <span class="p">[</span><span class="n">k</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">keylist</span><span class="p">]</span> <span class="k">if</span> <span class="n">keylist</span> <span class="o">!=</span> <span class="p">[]</span> <span class="k">else</span> <span class="bp">None</span></div>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">list_objects_v2</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">Prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="n">Delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="p">[</span><span class="n">p</span><span class="p">[</span><span class="s1">'Prefix'</span><span class="p">]</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'CommonPrefixes'</span><span class="p">]]</span> <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'CommonPrefixes'</span><span class="p">)</span> <span class="k">else</span> <span class="kc">None</span></div>
-<div class="viewcode-block" id="S3Hook.list_prefixes"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.list_prefixes">[docs]</a> <span class="k">def</span> <span class="nf">list_prefixes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+<div class="viewcode-block" id="S3Hook.list_keys"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.list_keys">[docs]</a> <span class="k">def</span> <span class="nf">list_keys</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Lists prefixes in a bucket under prefix</span>
+<span class="sd"> Lists keys in a bucket under prefix and not containing delimiter</span>
<span class="sd"> :param bucket_name: the name of the bucket</span>
<span class="sd"> :type bucket_name: str</span>
@@ -376,24 +267,32 @@
<span class="sd"> :param delimiter: the delimiter marks key hierarchy.</span>
<span class="sd"> :type delimiter: str</span>
<span class="sd"> """</span>
- <span class="n">b</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="n">plist</span> <span class="o">=</span> <span class="n">b</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
- <span class="n">prefix_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">plist</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">boto</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">prefix</span><span class="o">.</span><span class="n">Prefix</span><span class="p">)]</span>
- <span class="k">return</span> <span class="n">prefix_names</span> <span class="k">if</span> <span class="n">prefix_names</span> <span class="o">!=</span> <span class="p">[]</span> <span class="k">else</span> <span class="bp">None</span></div>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">list_objects_v2</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">Prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="n">Delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="p">[</span><span class="n">k</span><span class="p">[</span><span class="s1">'Key'</span><span class="p">]</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'Contents'</span><span class="p">]]</span> <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'Contents'</span><span class="p">)</span> <span class="k">else</span> <span class="kc">None</span></div>
-<div class="viewcode-block" id="S3Hook.check_for_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_key">[docs]</a> <span class="k">def</span> <span class="nf">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+<div class="viewcode-block" id="S3Hook.check_for_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_key">[docs]</a> <span class="k">def</span> <span class="nf">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Checks that a key exists in a bucket</span>
+<span class="sd"> Checks if a key exists in a bucket</span>
+
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which the file is stored</span>
+<span class="sd"> :type bucket_name: str</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
<span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span></div>
-<div class="viewcode-block" id="S3Hook.get_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_key">[docs]</a> <span class="k">def</span> <span class="nf">get_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">head_object</span><span class="p">(</span><span class="n">Bucket</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">Key</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span></div>
+
+<div class="viewcode-block" id="S3Hook.get_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_key">[docs]</a> <span class="k">def</span> <span class="nf">get_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Returns a boto.s3.key.Key object</span>
+<span class="sd"> Returns a boto3.s3.Object</span>
<span class="sd"> :param key: the path to the key</span>
<span class="sd"> :type key: str</span>
@@ -402,21 +301,36 @@
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
<span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span></div>
+
+ <span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_resource_type</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
+ <span class="n">obj</span><span class="o">.</span><span class="n">load</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">obj</span></div>
+
+<div class="viewcode-block" id="S3Hook.read_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.read_key">[docs]</a> <span class="k">def</span> <span class="nf">read_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Reads a key from S3</span>
+
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which the file is stored</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">obj</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">obj</span><span class="o">.</span><span class="n">get</span><span class="p">()[</span><span class="s1">'Body'</span><span class="p">]</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span></div>
<div class="viewcode-block" id="S3Hook.check_for_wildcard_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_wildcard_key">[docs]</a> <span class="k">def</span> <span class="nf">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
- <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Checks that a key matching a wildcard expression exists in a bucket</span>
<span class="sd"> """</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_wildcard_key</span><span class="p">(</span><span class="n">wildcard_key</span><span class="o">=</span><span class="n">wildcard_key</span><span class="p">,</span>
<span class="n">bucket_name</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
- <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span></div>
+ <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span></div>
-<div class="viewcode-block" id="S3Hook.get_wildcard_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_wildcard_key">[docs]</a> <span class="k">def</span> <span class="nf">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+<div class="viewcode-block" id="S3Hook.get_wildcard_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_wildcard_key">[docs]</a> <span class="k">def</span> <span class="nf">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Returns a boto.s3.key.Key object matching the regular expression</span>
+<span class="sd"> Returns a boto3.s3.Object object matching the regular expression</span>
<span class="sd"> :param regex_key: the path to the key</span>
<span class="sd"> :type regex_key: str</span>
@@ -425,31 +339,20 @@
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
<span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">wildcard_key</span><span class="p">)</span>
- <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="n">prefix</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">r'[*]'</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
- <span class="n">klist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_keys</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">klist</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">None</span>
- <span class="n">key_matches</span> <span class="o">=</span> <span class="p">[</span><span class="n">k</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">klist</span> <span class="k">if</span> <span class="n">fnmatch</span><span class="o">.</span><span class="n">fnmatch</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)]</span>
- <span class="k">return</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key_matches</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="k">if</span> <span class="n">key_matches</span> <span class="k">else</span> <span class="bp">None</span></div>
-<div class="viewcode-block" id="S3Hook.check_for_prefix"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_prefix">[docs]</a> <span class="k">def</span> <span class="nf">check_for_prefix</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Checks that a prefix exists in a bucket</span>
-<span class="sd"> """</span>
- <span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span> <span class="o">+</span> <span class="n">delimiter</span> <span class="k">if</span> <span class="n">prefix</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">!=</span> <span class="n">delimiter</span> <span class="k">else</span> <span class="n">prefix</span>
- <span class="n">prefix_split</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">r'(\w+[{d}])$'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">d</span><span class="o">=</span><span class="n">delimiter</span><span class="p">),</span> <span class="n">prefix</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
- <span class="n">previous_level</span> <span class="o">=</span> <span class="n">prefix_split</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
- <span class="n">plist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_prefixes</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">previous_level</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">False</span> <span class="k">if</span> <span class="n">plist</span> <span class="ow">is</span> <span class="bp">None</span> <span class="k">else</span> <span class="n">prefix</span> <span class="ow">in</span> <span class="n">plist</span></div>
-
-<div class="viewcode-block" id="S3Hook.load_file"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.load_file">[docs]</a> <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span>
- <span class="n">filename</span><span class="p">,</span>
- <span class="n">key</span><span class="p">,</span>
- <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">replace</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">multipart_bytes</span><span class="o">=</span><span class="mi">5</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1024</span> <span class="o">**</span> <span class="mi">3</span><span class="p">)):</span>
+ <span class="n">prefix</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="sa">r</span><span class="s1">'[*]'</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">klist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_keys</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">klist</span><span class="p">:</span>
+ <span class="n">key_matches</span> <span class="o">=</span> <span class="p">[</span><span class="n">k</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">klist</span> <span class="k">if</span> <span class="n">fnmatch</span><span class="o">.</span><span class="n">fnmatch</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)]</span>
+ <span class="k">if</span> <span class="n">key_matches</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key_matches</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">bucket_name</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="S3Hook.load_file"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.load_file">[docs]</a> <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">filename</span><span class="p">,</span>
+ <span class="n">key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">encrypt</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Loads a local file to S3</span>
@@ -463,60 +366,35 @@
<span class="sd"> if it already exists. If replace is False and the key exists, an</span>
<span class="sd"> error will be raised.</span>
<span class="sd"> :type replace: bool</span>
-<span class="sd"> :param multipart_bytes: If provided, the file is uploaded in parts of</span>
-<span class="sd"> this size (minimum 5242880). The default value is 5GB, since S3</span>
-<span class="sd"> cannot accept non-multipart uploads for files larger than 5GB. If</span>
-<span class="sd"> the file is smaller than the specified limit, the option will be</span>
-<span class="sd"> ignored.</span>
-<span class="sd"> :type multipart_bytes: int</span>
+<span class="sd"> :param encrypt: If True, the file will be encrypted on the server-side</span>
+<span class="sd"> by S3 and will be stored in an encrypted form while at rest in S3.</span>
+<span class="sd"> :type encrypt: bool</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
<span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="n">key_obj</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key {key} already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
-
- <span class="n">key_size</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">getsize</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">multipart_bytes</span> <span class="ow">and</span> <span class="n">key_size</span> <span class="o">>=</span> <span class="n">multipart_bytes</span><span class="p">:</span>
- <span class="c1"># multipart upload</span>
- <span class="kn">from</span> <span class="nn">filechunkio</span> <span class="kn">import</span> <span class="n">FileChunkIO</span>
- <span class="n">mp</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">initiate_multipart_upload</span><span class="p">(</span><span class="n">key_name</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
- <span class="n">total_chunks</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">ceil</span><span class="p">(</span><span class="n">key_size</span> <span class="o">/</span> <span class="n">multipart_bytes</span><span class="p">))</span>
- <span class="n">sent_bytes</span> <span class="o">=</span> <span class="mi">0</span>
- <span class="k">try</span><span class="p">:</span>
- <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">total_chunks</span><span class="p">):</span>
- <span class="n">offset</span> <span class="o">=</span> <span class="n">chunk</span> <span class="o">*</span> <span class="n">multipart_bytes</span>
- <span class="nb">bytes</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">multipart_bytes</span><span class="p">,</span> <span class="n">key_size</span> <span class="o">-</span> <span class="n">offset</span><span class="p">)</span>
- <span class="k">with</span> <span class="n">FileChunkIO</span><span class="p">(</span>
- <span class="n">filename</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">,</span> <span class="n">offset</span><span class="o">=</span><span class="n">offset</span><span class="p">,</span> <span class="nb">bytes</span><span class="o">=</span><span class="nb">bytes</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sending chunk {c} of {tc}...'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="n">c</span><span class="o">=</span><span class="n">chunk</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="n">tc</span><span class="o">=</span><span class="n">total_chunks</span><span class="p">))</span>
- <span class="n">mp</span><span class="o">.</span><span class="n">upload_part_from_file</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">part_num</span><span class="o">=</span><span class="n">chunk</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
- <span class="k">except</span><span class="p">:</span>
- <span class="n">mp</span><span class="o">.</span><span class="n">cancel_upload</span><span class="p">()</span>
- <span class="k">raise</span>
- <span class="n">mp</span><span class="o">.</span><span class="n">complete_upload</span><span class="p">()</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="c1"># regular upload</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">key_obj</span><span class="p">:</span>
- <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">new_key</span><span class="p">(</span><span class="n">key_name</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
- <span class="n">key_size</span> <span class="o">=</span> <span class="n">key_obj</span><span class="o">.</span><span class="n">set_contents_from_filename</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span>
- <span class="n">replace</span><span class="o">=</span><span class="n">replace</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"The key {key} now contains"</span>
- <span class="s2">" {key_size} bytes"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
-
-<div class="viewcode-block" id="S3Hook.load_string"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.load_string">[docs]</a> <span class="k">def</span> <span class="nf">load_string</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string_data</span><span class="p">,</span>
- <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">replace</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">encrypt</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key </span><span class="si">{key}</span><span class="s2"> already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">key</span><span class="p">))</span>
+
+ <span class="n">extra_args</span><span class="o">=</span><span class="p">{}</span>
+ <span class="k">if</span> <span class="n">encrypt</span><span class="p">:</span>
+ <span class="n">extra_args</span><span class="p">[</span><span class="s1">'ServerSideEncryption'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"AES256"</span>
+
+ <span class="n">client</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">client</span><span class="o">.</span><span class="n">upload_file</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">ExtraArgs</span><span class="o">=</span><span class="n">extra_args</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="S3Hook.load_string"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.load_string">[docs]</a> <span class="k">def</span> <span class="nf">load_string</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">string_data</span><span class="p">,</span>
+ <span class="n">key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">encrypt</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">):</span>
<span class="sd">"""</span>
-<span class="sd"> Loads a local file to S3</span>
+<span class="sd"> Loads a string to S3</span>
-<span class="sd"> This is provided as a convenience to drop a file in S3. It uses the</span>
-<span class="sd"> boto infrastructure to ship a file to s3. It is currently using only</span>
-<span class="sd"> a single part download, and should not be used to move large files.</span>
+<span class="sd"> This is provided as a convenience to drop a string in S3. It uses the</span>
+<span class="sd"> boto infrastructure to ship a file to s3. </span>
<span class="sd"> :param string_data: string to set as content for the key.</span>
<span class="sd"> :type string_data: str</span>
@@ -527,24 +405,30 @@
<span class="sd"> :param replace: A flag to decide whether or not to overwrite the key</span>
<span class="sd"> if it already exists</span>
<span class="sd"> :type replace: bool</span>
+<span class="sd"> :param encrypt: If True, the file will be encrypted on the server-side</span>
+<span class="sd"> by S3 and will be stored in an encrypted form while at rest in S3.</span>
+<span class="sd"> :type encrypt: bool</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
<span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
- <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="n">key_obj</span><span class="p">:</span>
- <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key {key} already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="n">key_obj</span><span class="p">:</span>
- <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">new_key</span><span class="p">(</span><span class="n">key_name</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
- <span class="n">key_size</span> <span class="o">=</span> <span class="n">key_obj</span><span class="o">.</span><span class="n">set_contents_from_string</span><span class="p">(</span><span class="n">string_data</span><span class="p">,</span>
- <span class="n">replace</span><span class="o">=</span><span class="n">replace</span><span class="p">,</span>
- <span class="n">encrypt_key</span><span class="o">=</span><span class="n">encrypt</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"The key {key} now contains"</span>
- <span class="s2">" {key_size} bytes"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div></div>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key </span><span class="si">{key}</span><span class="s2"> already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="n">key</span><span class="p">))</span>
+
+ <span class="n">extra_args</span><span class="o">=</span><span class="p">{}</span>
+ <span class="k">if</span> <span class="n">encrypt</span><span class="p">:</span>
+ <span class="n">extra_args</span><span class="p">[</span><span class="s1">'ServerSideEncryption'</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"AES256"</span>
+
+ <span class="n">filelike_buffer</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="n">string_data</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">encoding</span><span class="p">))</span>
+
+ <span class="n">client</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">client</span><span class="o">.</span><span class="n">upload_fileobj</span><span class="p">(</span><span class="n">filelike_buffer</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">ExtraArgs</span><span class="o">=</span><span class="n">extra_args</span><span class="p">)</span></div></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -577,7 +461,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
[29/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/bigquery_to_gcs.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/bigquery_to_gcs.html b/_modules/airflow/contrib/operators/bigquery_to_gcs.html
index 97cf815..31355ff 100644
--- a/_modules/airflow/contrib/operators/bigquery_to_gcs.html
+++ b/_modules/airflow/contrib/operators/bigquery_to_gcs.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
<link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
<link rel="up" title="Module code" href="../../../index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../../../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../../../../index.html">Docs</a> »</li>
-
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
<li><a href="../../../index.html">Module code</a> »</li>
-
- <li>airflow.contrib.operators.bigquery_to_gcs</li>
+
+ <li>airflow.contrib.operators.bigquery_to_gcs</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,65 +181,63 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="k">import</span> <span class="n">BigQueryHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="kn">import</span> <span class="n">BigQueryHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
-
-<div class="viewcode-block" id="BigQueryToCloudStorageOperator"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryToCloudStorageOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+<div class="viewcode-block" id="BigQueryToCloudStorageOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator">[docs]</a><span class="k">class</span> <span class="nc">BigQueryToCloudStorageOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Transfers a BigQuery table to a Google Cloud Storage bucket.</span>
+
+<span class="sd"> See here:</span>
+
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs</span>
+
+<span class="sd"> For more details about these parameters.</span>
+
+<span class="sd"> :param source_project_dataset_table: The dotted</span>
+<span class="sd"> (<project>.|<project>:)<dataset>.<table> BigQuery table to use as the source</span>
+<span class="sd"> data. If <project> is not included, project will be the project defined in</span>
+<span class="sd"> the connection json.</span>
+<span class="sd"> :type source_project_dataset_table: string</span>
+<span class="sd"> :param destination_cloud_storage_uris: The destination Google Cloud</span>
+<span class="sd"> Storage URI (e.g. gs://some-bucket/some-file.txt). Follows</span>
+<span class="sd"> convention defined here:</span>
+<span class="sd"> https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple</span>
+<span class="sd"> :type destination_cloud_storage_uris: list</span>
+<span class="sd"> :param compression: Type of compression to use.</span>
+<span class="sd"> :type compression: string</span>
+<span class="sd"> :param export_format: File format to export.</span>
+<span class="sd"> :type field_delimiter: string</span>
+<span class="sd"> :param field_delimiter: The delimiter to use when extracting to a CSV.</span>
+<span class="sd"> :type field_delimiter: string</span>
+<span class="sd"> :param print_header: Whether to print a header for a CSV file extract.</span>
+<span class="sd"> :type print_header: boolean</span>
+<span class="sd"> :param bigquery_conn_id: reference to a specific BigQuery hook.</span>
+<span class="sd"> :type bigquery_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'source_project_dataset_table'</span><span class="p">,</span><span class="s1">'destination_cloud_storage_uris'</span><span class="p">,)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'source_project_dataset_table'</span><span class="p">,</span> <span class="s1">'destination_cloud_storage_uris'</span><span class="p">)</span>
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
<span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e4e6f0'</span>
<span class="nd">@apply_defaults</span>
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span>
- <span class="n">source_project_dataset_table</span><span class="p">,</span>
- <span class="n">destination_cloud_storage_uris</span><span class="p">,</span>
- <span class="n">compression</span><span class="o">=</span><span class="s1">'NONE'</span><span class="p">,</span>
- <span class="n">export_format</span><span class="o">=</span><span class="s1">'CSV'</span><span class="p">,</span>
- <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
- <span class="n">print_header</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
- <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
- <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="o">*</span><span class="n">args</span><span class="p">,</span>
- <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Create a new BigQueryToCloudStorage to move data from BigQuery to</span>
-<span class="sd"> Google Cloud Storage. See here:</span>
-
-<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs</span>
-
-<span class="sd"> For more details about these parameters.</span>
-
-<span class="sd"> :param source_project_dataset_table: The dotted (<project>.)<dataset>.<table> BigQuery table to use as the</span>
-<span class="sd"> source data. If <project> is not included, project will be the project defined in the connection json.</span>
-<span class="sd"> :type source_project_dataset_table: string</span>
-<span class="sd"> :param destination_cloud_storage_uris: The destination Google Cloud</span>
-<span class="sd"> Storage URI (e.g. gs://some-bucket/some-file.txt). Follows</span>
-<span class="sd"> convention defined here:</span>
-<span class="sd"> https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple</span>
-<span class="sd"> :type destination_cloud_storage_uris: list</span>
-<span class="sd"> :param compression: Type of compression to use.</span>
-<span class="sd"> :type compression: string</span>
-<span class="sd"> :param export_format: File format to export.</span>
-<span class="sd"> :type field_delimiter: string</span>
-<span class="sd"> :param field_delimiter: The delimiter to use when extracting to a CSV.</span>
-<span class="sd"> :type field_delimiter: string</span>
-<span class="sd"> :param print_header: Whether to print a header for a CSV file extract.</span>
-<span class="sd"> :type print_header: boolean</span>
-<span class="sd"> :param bigquery_conn_id: reference to a specific BigQuery hook.</span>
-<span class="sd"> :type bigquery_conn_id: string</span>
-<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
-<span class="sd"> For this to work, the service account making the request must have domain-wide delegation enabled.</span>
-<span class="sd"> :type delegate_to: string</span>
-<span class="sd"> """</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryToCloudStorageOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">source_project_dataset_table</span><span class="p">,</span>
+ <span class="n">destination_cloud_storage_uris</span><span class="p">,</span>
+ <span class="n">compression</span><span class="o">=</span><span class="s1">'NONE'</span><span class="p">,</span>
+ <span class="n">export_format</span><span class="o">=</span><span class="s1">'CSV'</span><span class="p">,</span>
+ <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">print_header</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BigQueryToCloudStorageOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">source_project_dataset_table</span> <span class="o">=</span> <span class="n">source_project_dataset_table</span>
<span class="bp">self</span><span class="o">.</span><span class="n">destination_cloud_storage_uris</span> <span class="o">=</span> <span class="n">destination_cloud_storage_uris</span>
<span class="bp">self</span><span class="o">.</span><span class="n">compression</span> <span class="o">=</span> <span class="n">compression</span>
@@ -220,8 +248,11 @@
<span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing extract of </span><span class="si">%s</span><span class="s1"> into: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">source_project_dataset_table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">destination_cloud_storage_uris</span><span class="p">)</span>
- <span class="n">hook</span> <span class="o">=</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing extract of </span><span class="si">%s</span><span class="s1"> into: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_project_dataset_table</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_cloud_storage_uris</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">run_extract</span><span class="p">(</span>
@@ -234,6 +265,9 @@
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -266,7 +300,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../../../../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/databricks_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/databricks_operator.html b/_modules/airflow/contrib/operators/databricks_operator.html
new file mode 100644
index 0000000..cb38349
--- /dev/null
+++ b/_modules/airflow/contrib/operators/databricks_operator.html
@@ -0,0 +1,487 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.databricks_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.databricks_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.databricks_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">import</span> <span class="nn">six</span>
+<span class="kn">import</span> <span class="nn">time</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.databricks_hook</span> <span class="k">import</span> <span class="n">DatabricksHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+
+
+<div class="viewcode-block" id="DatabricksSubmitRunOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator">[docs]</a><span class="k">class</span> <span class="nc">DatabricksSubmitRunOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Submits an Spark job run to Databricks using the</span>
+<span class="sd"> `api/2.0/jobs/runs/submit</span>
+<span class="sd"> <https://docs.databricks.com/api/latest/jobs.html#runs-submit>`_</span>
+<span class="sd"> API endpoint.</span>
+
+<span class="sd"> There are two ways to instantiate this operator.</span>
+
+<span class="sd"> In the first way, you can take the JSON payload that you typically use</span>
+<span class="sd"> to call the ``api/2.0/jobs/runs/submit`` endpoint and pass it directly</span>
+<span class="sd"> to our ``DatabricksSubmitRunOperator`` through the ``json`` parameter.</span>
+<span class="sd"> For example ::</span>
+<span class="sd"> json = {</span>
+<span class="sd"> 'new_cluster': {</span>
+<span class="sd"> 'spark_version': '2.1.0-db3-scala2.11',</span>
+<span class="sd"> 'num_workers': 2</span>
+<span class="sd"> },</span>
+<span class="sd"> 'notebook_task': {</span>
+<span class="sd"> 'notebook_path': '/Users/airflow@example.com/PrepareData',</span>
+<span class="sd"> },</span>
+<span class="sd"> }</span>
+<span class="sd"> notebook_run = DatabricksSubmitRunOperator(task_id='notebook_run', json=json)</span>
+
+<span class="sd"> Another way to accomplish the same thing is to use the named parameters</span>
+<span class="sd"> of the ``DatabricksSubmitRunOperator`` directly. Note that there is exactly</span>
+<span class="sd"> one named parameter for each top level parameter in the ``runs/submit``</span>
+<span class="sd"> endpoint. In this method, your code would look like this: ::</span>
+<span class="sd"> new_cluster = {</span>
+<span class="sd"> 'spark_version': '2.1.0-db3-scala2.11',</span>
+<span class="sd"> 'num_workers': 2</span>
+<span class="sd"> }</span>
+<span class="sd"> notebook_task = {</span>
+<span class="sd"> 'notebook_path': '/Users/airflow@example.com/PrepareData',</span>
+<span class="sd"> }</span>
+<span class="sd"> notebook_run = DatabricksSubmitRunOperator(</span>
+<span class="sd"> task_id='notebook_run',</span>
+<span class="sd"> new_cluster=new_cluster,</span>
+<span class="sd"> notebook_task=notebook_task)</span>
+
+<span class="sd"> In the case where both the json parameter **AND** the named parameters</span>
+<span class="sd"> are provided, they will be merged together. If there are conflicts during the merge,</span>
+<span class="sd"> the named parameters will take precedence and override the top level ``json`` keys.</span>
+
+<span class="sd"> Currently the named parameters that ``DatabricksSubmitRunOperator`` supports are</span>
+<span class="sd"> - ``spark_jar_task``</span>
+<span class="sd"> - ``notebook_task``</span>
+<span class="sd"> - ``new_cluster``</span>
+<span class="sd"> - ``existing_cluster_id``</span>
+<span class="sd"> - ``libraries``</span>
+<span class="sd"> - ``run_name``</span>
+<span class="sd"> - ``timeout_seconds``</span>
+
+<span class="sd"> :param json: A JSON object containing API parameters which will be passed</span>
+<span class="sd"> directly to the ``api/2.0/jobs/runs/submit`` endpoint. The other named parameters</span>
+<span class="sd"> (i.e. ``spark_jar_task``, ``notebook_task``..) to this operator will</span>
+<span class="sd"> be merged with this json dictionary if they are provided.</span>
+<span class="sd"> If there are conflicts during the merge, the named parameters will</span>
+<span class="sd"> take precedence and override the top level json keys. This field will be</span>
+<span class="sd"> templated.</span>
+
+<span class="sd"> .. seealso::</span>
+<span class="sd"> For more information about templating see :ref:`jinja-templating`.</span>
+<span class="sd"> https://docs.databricks.com/api/latest/jobs.html#runs-submit</span>
+<span class="sd"> :type json: dict</span>
+<span class="sd"> :param spark_jar_task: The main class and parameters for the JAR task. Note that</span>
+<span class="sd"> the actual JAR is specified in the ``libraries``.</span>
+<span class="sd"> *EITHER* ``spark_jar_task`` *OR* ``notebook_task`` should be specified.</span>
+<span class="sd"> This field will be templated.</span>
+
+<span class="sd"> .. seealso::</span>
+<span class="sd"> https://docs.databricks.com/api/latest/jobs.html#jobssparkjartask</span>
+<span class="sd"> :type spark_jar_task: dict</span>
+<span class="sd"> :param notebook_task: The notebook path and parameters for the notebook task.</span>
+<span class="sd"> *EITHER* ``spark_jar_task`` *OR* ``notebook_task`` should be specified.</span>
+<span class="sd"> This field will be templated.</span>
+
+<span class="sd"> .. seealso::</span>
+<span class="sd"> https://docs.databricks.com/api/latest/jobs.html#jobsnotebooktask</span>
+<span class="sd"> :type notebook_task: dict</span>
+<span class="sd"> :param new_cluster: Specs for a new cluster on which this task will be run.</span>
+<span class="sd"> *EITHER* ``new_cluster`` *OR* ``existing_cluster_id`` should be specified.</span>
+<span class="sd"> This field will be templated.</span>
+
+<span class="sd"> .. seealso::</span>
+<span class="sd"> https://docs.databricks.com/api/latest/jobs.html#jobsclusterspecnewcluster</span>
+<span class="sd"> :type new_cluster: dict</span>
+<span class="sd"> :param existing_cluster_id: ID for existing cluster on which to run this task.</span>
+<span class="sd"> *EITHER* ``new_cluster`` *OR* ``existing_cluster_id`` should be specified.</span>
+<span class="sd"> This field will be templated.</span>
+<span class="sd"> :type existing_cluster_id: string</span>
+<span class="sd"> :param libraries: Libraries which this run will use.</span>
+<span class="sd"> This field will be templated.</span>
+
+<span class="sd"> .. seealso::</span>
+<span class="sd"> https://docs.databricks.com/api/latest/libraries.html#managedlibrarieslibrary</span>
+<span class="sd"> :type libraries: list of dicts</span>
+<span class="sd"> :param run_name: The run name used for this task.</span>
+<span class="sd"> By default this will be set to the Airflow ``task_id``. This ``task_id`` is a</span>
+<span class="sd"> required parameter of the superclass ``BaseOperator``.</span>
+<span class="sd"> This field will be templated.</span>
+<span class="sd"> :type run_name: string</span>
+<span class="sd"> :param timeout_seconds: The timeout for this run. By default a value of 0 is used</span>
+<span class="sd"> which means to have no timeout.</span>
+<span class="sd"> This field will be templated.</span>
+<span class="sd"> :type timeout_seconds: int32</span>
+<span class="sd"> :param databricks_conn_id: The name of the Airflow connection to use.</span>
+<span class="sd"> By default and in the common case this will be ``databricks_default``. To use</span>
+<span class="sd"> token based authentication, provide the key ``token`` in the extra field for the</span>
+<span class="sd"> connection.</span>
+<span class="sd"> :type databricks_conn_id: string</span>
+<span class="sd"> :param polling_period_seconds: Controls the rate which we poll for the result of</span>
+<span class="sd"> this run. By default the operator will poll every 30 seconds.</span>
+<span class="sd"> :type polling_period_seconds: int</span>
+<span class="sd"> :param databricks_retry_limit: Amount of times retry if the Databricks backend is</span>
+<span class="sd"> unreachable. Its value must be greater than or equal to 1.</span>
+<span class="sd"> :type databricks_retry_limit: int</span>
+<span class="sd"> """</span>
+ <span class="c1"># Used in airflow.models.BaseOperator</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'json'</span><span class="p">,)</span>
+ <span class="c1"># Databricks brand color (blue) under white text</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#1CB1C2'</span>
+ <span class="n">ui_fgcolor</span> <span class="o">=</span> <span class="s1">'#fff'</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">json</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">spark_jar_task</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">notebook_task</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">new_cluster</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">existing_cluster_id</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">libraries</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">run_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">timeout_seconds</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">databricks_conn_id</span><span class="o">=</span><span class="s1">'databricks_default'</span><span class="p">,</span>
+ <span class="n">polling_period_seconds</span><span class="o">=</span><span class="mi">30</span><span class="p">,</span>
+ <span class="n">databricks_retry_limit</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Creates a new ``DatabricksSubmitRunOperator``.</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DatabricksSubmitRunOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span> <span class="o">=</span> <span class="n">json</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">databricks_conn_id</span> <span class="o">=</span> <span class="n">databricks_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">polling_period_seconds</span> <span class="o">=</span> <span class="n">polling_period_seconds</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">databricks_retry_limit</span> <span class="o">=</span> <span class="n">databricks_retry_limit</span>
+ <span class="k">if</span> <span class="n">spark_jar_task</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'spark_jar_task'</span><span class="p">]</span> <span class="o">=</span> <span class="n">spark_jar_task</span>
+ <span class="k">if</span> <span class="n">notebook_task</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'notebook_task'</span><span class="p">]</span> <span class="o">=</span> <span class="n">notebook_task</span>
+ <span class="k">if</span> <span class="n">new_cluster</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'new_cluster'</span><span class="p">]</span> <span class="o">=</span> <span class="n">new_cluster</span>
+ <span class="k">if</span> <span class="n">existing_cluster_id</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'existing_cluster_id'</span><span class="p">]</span> <span class="o">=</span> <span class="n">existing_cluster_id</span>
+ <span class="k">if</span> <span class="n">libraries</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'libraries'</span><span class="p">]</span> <span class="o">=</span> <span class="n">libraries</span>
+ <span class="k">if</span> <span class="n">run_name</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'run_name'</span><span class="p">]</span> <span class="o">=</span> <span class="n">run_name</span>
+ <span class="k">if</span> <span class="n">timeout_seconds</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'timeout_seconds'</span><span class="p">]</span> <span class="o">=</span> <span class="n">timeout_seconds</span>
+ <span class="k">if</span> <span class="s1">'run_name'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">[</span><span class="s1">'run_name'</span><span class="p">]</span> <span class="o">=</span> <span class="n">run_name</span> <span class="ow">or</span> <span class="n">kwargs</span><span class="p">[</span><span class="s1">'task_id'</span><span class="p">]</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">json</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_deep_string_coerce</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">)</span>
+ <span class="c1"># This variable will be used in case our task gets killed.</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_id</span> <span class="o">=</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">_deep_string_coerce</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">content</span><span class="p">,</span> <span class="n">json_path</span><span class="o">=</span><span class="s1">'json'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Coerces content or all values of content if it is a dict to a string. The</span>
+<span class="sd"> function will throw if content contains non-string or non-numeric types.</span>
+
+<span class="sd"> The reason why we have this function is because the ``self.json`` field must be a dict</span>
+<span class="sd"> with only string values. This is because ``render_template`` will fail for numerical values.</span>
+<span class="sd"> """</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_deep_string_coerce</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">six</span><span class="o">.</span><span class="n">string_types</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">content</span>
+ <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">six</span><span class="o">.</span><span class="n">integer_types</span><span class="o">+</span><span class="p">(</span><span class="nb">float</span><span class="p">,)):</span>
+ <span class="c1"># Databricks can tolerate either numeric or string types in the API backend.</span>
+ <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="p">(</span><span class="nb">list</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">)):</span>
+ <span class="k">return</span> <span class="p">[</span><span class="n">c</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">'</span><span class="si">{0}</span><span class="s1">[</span><span class="si">{1}</span><span class="s1">]'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">json_path</span><span class="p">,</span> <span class="n">i</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">e</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">content</span><span class="p">)]</span>
+ <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
+ <span class="k">return</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">c</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="s1">'</span><span class="si">{0}</span><span class="s1">[</span><span class="si">{1}</span><span class="s1">]'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">json_path</span><span class="p">,</span> <span class="n">k</span><span class="p">))</span>
+ <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">content</span><span class="o">.</span><span class="n">items</span><span class="p">())}</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">param_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
+ <span class="n">msg</span> <span class="o">=</span> <span class="s1">'Type </span><span class="si">{0}</span><span class="s1"> used for parameter </span><span class="si">{1}</span><span class="s1"> is not a number or a string'</span> \
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">param_type</span><span class="p">,</span> <span class="n">json_path</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_log_run_page_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">url</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'View run status, Spark UI, and logs at </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">DatabricksHook</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">databricks_conn_id</span><span class="p">,</span>
+ <span class="n">retry_limit</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">databricks_retry_limit</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_id</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">submit_run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">json</span><span class="p">)</span>
+ <span class="n">run_page_url</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_run_page_url</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Run submitted with run_id: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_log_run_page_url</span><span class="p">(</span><span class="n">run_page_url</span><span class="p">)</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="n">run_state</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_run_state</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">run_state</span><span class="o">.</span><span class="n">is_terminal</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">run_state</span><span class="o">.</span><span class="n">is_successful</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1"> completed successfully.'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_log_run_page_url</span><span class="p">(</span><span class="n">run_page_url</span><span class="p">)</span>
+ <span class="k">return</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">error_message</span> <span class="o">=</span> <span class="s1">'</span><span class="si">{t}</span><span class="s1"> failed with terminal state: </span><span class="si">{s}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">t</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span>
+ <span class="n">s</span><span class="o">=</span><span class="n">run_state</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="n">error_message</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'</span><span class="si">%s</span><span class="s1"> in run state: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="n">run_state</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_log_run_page_url</span><span class="p">(</span><span class="n">run_page_url</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sleeping for </span><span class="si">%s</span><span class="s1"> seconds.'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">polling_period_seconds</span><span class="p">)</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">polling_period_seconds</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+ <span class="n">hook</span><span class="o">.</span><span class="n">cancel_run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Task: </span><span class="si">%s</span><span class="s1"> with run_id: </span><span class="si">%s</span><span class="s1"> was requested to be cancelled.'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_id</span>
+ <span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/dataflow_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/dataflow_operator.html b/_modules/airflow/contrib/operators/dataflow_operator.html
new file mode 100644
index 0000000..6807e24
--- /dev/null
+++ b/_modules/airflow/contrib/operators/dataflow_operator.html
@@ -0,0 +1,470 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.dataflow_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.dataflow_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.dataflow_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">import</span> <span class="nn">copy</span>
+<span class="kn">import</span> <span class="nn">re</span>
+<span class="kn">import</span> <span class="nn">uuid</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcs_hook</span> <span class="k">import</span> <span class="n">GoogleCloudStorageHook</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_dataflow_hook</span> <span class="k">import</span> <span class="n">DataFlowHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="DataFlowJavaOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator">[docs]</a><span class="k">class</span> <span class="nc">DataFlowJavaOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Start a Java Cloud DataFlow batch job. The parameters of the operation</span>
+<span class="sd"> will be passed to the job.</span>
+
+<span class="sd"> It's a good practice to define dataflow_* parameters in the default_args of the dag</span>
+<span class="sd"> like the project, zone and staging location.</span>
+
+<span class="sd"> ```</span>
+<span class="sd"> default_args = {</span>
+<span class="sd"> 'dataflow_default_options': {</span>
+<span class="sd"> 'project': 'my-gcp-project',</span>
+<span class="sd"> 'zone': 'europe-west1-d',</span>
+<span class="sd"> 'stagingLocation': 'gs://my-staging-bucket/staging/'</span>
+<span class="sd"> }</span>
+<span class="sd"> }</span>
+<span class="sd"> ```</span>
+
+<span class="sd"> You need to pass the path to your dataflow as a file reference with the ``jar``</span>
+<span class="sd"> parameter, the jar needs to be a self executing jar. Use ``options`` to pass on</span>
+<span class="sd"> options to your job.</span>
+
+<span class="sd"> ```</span>
+<span class="sd"> t1 = DataFlowOperation(</span>
+<span class="sd"> task_id='datapflow_example',</span>
+<span class="sd"> jar='{{var.value.gcp_dataflow_base}}pipeline/build/libs/pipeline-example-1.0.jar',</span>
+<span class="sd"> options={</span>
+<span class="sd"> 'autoscalingAlgorithm': 'BASIC',</span>
+<span class="sd"> 'maxNumWorkers': '50',</span>
+<span class="sd"> 'start': '{{ds}}',</span>
+<span class="sd"> 'partitionType': 'DAY'</span>
+<span class="sd"> },</span>
+<span class="sd"> dag=my-dag)</span>
+<span class="sd"> ```</span>
+
+<span class="sd"> Both ``jar`` and ``options`` are templated so you can use variables in them.</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'options'</span><span class="p">,</span> <span class="s1">'jar'</span><span class="p">]</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#0273d4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">jar</span><span class="p">,</span>
+ <span class="n">dataflow_default_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataFlowJavaOperator. Note that both</span>
+<span class="sd"> dataflow_default_options and options will be merged to specify pipeline</span>
+<span class="sd"> execution parameter, and dataflow_default_options is expected to save</span>
+<span class="sd"> high-level options, for instances, project and zone information, which</span>
+<span class="sd"> apply to all dataflow operators in the DAG.</span>
+
+<span class="sd"> For more detail on job submission have a look at the reference:</span>
+
+<span class="sd"> https://cloud.google.com/dataflow/pipelines/specifying-exec-params</span>
+
+<span class="sd"> :param jar: The reference to a self executing DataFlow jar.</span>
+<span class="sd"> :type jar: string</span>
+<span class="sd"> :param dataflow_default_options: Map of default job options.</span>
+<span class="sd"> :type dataflow_default_options: dict</span>
+<span class="sd"> :param options: Map of job specific options.</span>
+<span class="sd"> :type options: dict</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud</span>
+<span class="sd"> Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have</span>
+<span class="sd"> domain-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataFlowJavaOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="n">dataflow_default_options</span> <span class="o">=</span> <span class="n">dataflow_default_options</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="n">options</span> <span class="o">=</span> <span class="n">options</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">jar</span> <span class="o">=</span> <span class="n">jar</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataflow_default_options</span> <span class="o">=</span> <span class="n">dataflow_default_options</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="n">options</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">bucket_helper</span> <span class="o">=</span> <span class="n">GoogleCloudBucketHelper</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">jar</span> <span class="o">=</span> <span class="n">bucket_helper</span><span class="o">.</span><span class="n">google_cloud_to_local</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">jar</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataFlowHook</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+
+ <span class="n">dataflow_options</span> <span class="o">=</span> <span class="n">copy</span><span class="o">.</span><span class="n">copy</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dataflow_default_options</span><span class="p">)</span>
+ <span class="n">dataflow_options</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="p">)</span>
+
+ <span class="n">hook</span><span class="o">.</span><span class="n">start_java_dataflow</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="n">dataflow_options</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">jar</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="DataFlowPythonOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator">[docs]</a><span class="k">class</span> <span class="nc">DataFlowPythonOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'options'</span><span class="p">,</span> <span class="s1">'dataflow_default_options'</span><span class="p">]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">py_file</span><span class="p">,</span>
+ <span class="n">py_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">dataflow_default_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataFlowPythonOperator. Note that both</span>
+<span class="sd"> dataflow_default_options and options will be merged to specify pipeline</span>
+<span class="sd"> execution parameter, and dataflow_default_options is expected to save</span>
+<span class="sd"> high-level options, for instances, project and zone information, which</span>
+<span class="sd"> apply to all dataflow operators in the DAG.</span>
+
+<span class="sd"> For more detail on job submission have a look at the reference:</span>
+
+<span class="sd"> https://cloud.google.com/dataflow/pipelines/specifying-exec-params</span>
+
+<span class="sd"> :param py_file: Reference to the python dataflow pipleline file, e.g.,</span>
+<span class="sd"> /some/local/file/path/to/your/python/pipeline/file.py.</span>
+<span class="sd"> :type py_file: string</span>
+<span class="sd"> :param py_options: Additional python options.</span>
+<span class="sd"> :type pyt_options: list of strings, e.g., ["-m", "-v"].</span>
+<span class="sd"> :param dataflow_default_options: Map of default job options.</span>
+<span class="sd"> :type dataflow_default_options: dict</span>
+<span class="sd"> :param options: Map of job specific options.</span>
+<span class="sd"> :type options: dict</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud</span>
+<span class="sd"> Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have</span>
+<span class="sd"> domain-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataFlowPythonOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">py_file</span> <span class="o">=</span> <span class="n">py_file</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">py_options</span> <span class="o">=</span> <span class="n">py_options</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataflow_default_options</span> <span class="o">=</span> <span class="n">dataflow_default_options</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">options</span> <span class="o">=</span> <span class="n">options</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="sd">"""Execute the python dataflow job."""</span>
+ <span class="n">bucket_helper</span> <span class="o">=</span> <span class="n">GoogleCloudBucketHelper</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">py_file</span> <span class="o">=</span> <span class="n">bucket_helper</span><span class="o">.</span><span class="n">google_cloud_to_local</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">py_file</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataFlowHook</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="n">dataflow_options</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dataflow_default_options</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span>
+ <span class="n">dataflow_options</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">options</span><span class="p">)</span>
+ <span class="c1"># Convert argument names from lowerCamelCase to snake case.</span>
+ <span class="n">camel_to_snake</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">name</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span>
+ <span class="sa">r</span><span class="s1">'[A-Z]'</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="s1">'_'</span> <span class="o">+</span> <span class="n">x</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">(),</span> <span class="n">name</span><span class="p">)</span>
+ <span class="n">formatted_options</span> <span class="o">=</span> <span class="p">{</span><span class="n">camel_to_snake</span><span class="p">(</span><span class="n">key</span><span class="p">):</span> <span class="n">dataflow_options</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">dataflow_options</span><span class="p">}</span>
+ <span class="n">hook</span><span class="o">.</span><span class="n">start_python_dataflow</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="n">formatted_options</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">py_file</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">py_options</span><span class="p">)</span></div>
+
+
+<span class="k">class</span> <span class="nc">GoogleCloudBucketHelper</span><span class="p">():</span>
+ <span class="sd">"""GoogleCloudStorageHook helper class to download GCS object."""</span>
+ <span class="n">GCS_PREFIX_LENGTH</span> <span class="o">=</span> <span class="mi">5</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_gcs_hook</span> <span class="o">=</span> <span class="n">GoogleCloudStorageHook</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">google_cloud_to_local</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks whether the file specified by file_name is stored in Google Cloud</span>
+<span class="sd"> Storage (GCS), if so, downloads the file and saves it locally. The full</span>
+<span class="sd"> path of the saved file will be returned. Otherwise the local file_name</span>
+<span class="sd"> will be returned immediately.</span>
+
+<span class="sd"> :param file_name: The full path of input file.</span>
+<span class="sd"> :type file_name: string</span>
+<span class="sd"> :return: The full path of local file.</span>
+<span class="sd"> :type: string</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">file_name</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'gs://'</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">file_name</span>
+
+ <span class="c1"># Extracts bucket_id and object_id by first removing 'gs://' prefix and</span>
+ <span class="c1"># then split the remaining by path delimiter '/'.</span>
+ <span class="n">path_components</span> <span class="o">=</span> <span class="n">file_name</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">GCS_PREFIX_LENGTH</span><span class="p">:]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">path_components</span> <span class="o"><</span> <span class="mi">2</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'Invalid Google Cloud Storage (GCS) object path: </span><span class="si">{}</span><span class="s1">.'</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">file_name</span><span class="p">))</span>
+
+ <span class="n">bucket_id</span> <span class="o">=</span> <span class="n">path_components</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">object_id</span> <span class="o">=</span> <span class="s1">'/'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">path_components</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span>
+ <span class="n">local_file</span> <span class="o">=</span> <span class="s1">'/tmp/dataflow</span><span class="si">{}</span><span class="s1">-</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid1</span><span class="p">())[:</span><span class="mi">8</span><span class="p">],</span>
+ <span class="n">path_components</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
+ <span class="n">file_size</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_gcs_hook</span><span class="o">.</span><span class="n">download</span><span class="p">(</span><span class="n">bucket_id</span><span class="p">,</span> <span class="n">object_id</span><span class="p">,</span> <span class="n">local_file</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">file_size</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">local_file</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span>
+ <span class="s1">'Failed to download Google Cloud Storage GCS object: </span><span class="si">{}</span><span class="s1">'</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">file_name</span><span class="p">))</span>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[05/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/genindex.html
----------------------------------------------------------------------
diff --git a/genindex.html b/genindex.html
index bb1eacf..635cc29 100644
--- a/genindex.html
+++ b/genindex.html
@@ -14,6 +14,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -178,6 +183,7 @@
| <a href="#N"><strong>N</strong></a>
| <a href="#O"><strong>O</strong></a>
| <a href="#P"><strong>P</strong></a>
+ | <a href="#Q"><strong>Q</strong></a>
| <a href="#R"><strong>R</strong></a>
| <a href="#S"><strong>S</strong></a>
| <a href="#T"><strong>T</strong></a>
@@ -211,6 +217,8 @@
</li>
<li><a href="code.html#module-airflow.operators">airflow.operators (module)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.allocate_ids">allocate_ids() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.models.TaskInstance.are_dependencies_met">are_dependencies_met() (airflow.models.TaskInstance method)</a>
</li>
<li><a href="code.html#airflow.models.TaskInstance.are_dependents_done">are_dependents_done() (airflow.models.TaskInstance method)</a>
@@ -229,8 +237,28 @@
</li>
<li><a href="code.html#airflow.operators.BashOperator">BashOperator (class in airflow.operators)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.begin_transaction">begin_transaction() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator">BigQueryCheckOperator (class in airflow.contrib.operators.bigquery_check_operator)</a>
+</li>
+ <li><a href="code.html#airflow.contrib.hooks.BigQueryHook">BigQueryHook (class in airflow.contrib.hooks)</a>
+
+ <ul>
+ <li><a href="integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook">(class in airflow.contrib.hooks.bigquery_hook)</a>
+</li>
+ </ul></li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator">BigQueryIntervalCheckOperator (class in airflow.contrib.operators.bigquery_check_operator)</a>
+</li>
+ <li><a href="code.html#airflow.contrib.operators.bigquery_operator.BigQueryOperator">BigQueryOperator (class in airflow.contrib.operators.bigquery_operator)</a>, <a href="integration.html#airflow.contrib.operators.bigquery_operator.BigQueryOperator">[1]</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator">BigQueryToBigQueryOperator (class in airflow.contrib.operators.bigquery_to_bigquery)</a>
+</li>
+ <li><a href="code.html#airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator">BigQueryToCloudStorageOperator (class in airflow.contrib.operators.bigquery_to_gcs)</a>, <a href="integration.html#airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator">[1]</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator">BigQueryValueCheckOperator (class in airflow.contrib.operators.bigquery_check_operator)</a>
+</li>
<li><a href="code.html#airflow.operators.BranchPythonOperator">BranchPythonOperator (class in airflow.operators)</a>
</li>
<li><a href="code.html#airflow.hooks.DbApiHook.bulk_dump">bulk_dump() (airflow.hooks.DbApiHook method)</a>
@@ -247,7 +275,17 @@
<h2 id="C">C</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.contrib.hooks.SSHHook.check_output">check_output() (airflow.contrib.hooks.SSHHook method)</a>
+ <li><a href="code.html#airflow.hooks.S3Hook.check_for_bucket">check_for_bucket() (airflow.hooks.S3Hook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.check_for_key">check_for_key() (airflow.hooks.S3Hook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.check_for_named_partition">check_for_named_partition() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.check_for_partition">check_for_partition() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.check_for_prefix">check_for_prefix() (airflow.hooks.S3Hook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.check_for_wildcard_key">check_for_wildcard_key() (airflow.hooks.S3Hook method)</a>
</li>
<li><a href="code.html#airflow.models.BaseOperator.clear">clear() (airflow.models.BaseOperator method)</a>
@@ -265,22 +303,40 @@
</li>
<li><a href="code.html#airflow.models.DagBag.collect_dags">collect_dags() (airflow.models.DagBag method)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.TaskInstance.command">command() (airflow.models.TaskInstance method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.TaskInstance.command_as_list">command_as_list() (airflow.models.TaskInstance method)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.commit">commit() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.models.DAG.concurrency_reached">concurrency_reached (airflow.models.DAG attribute)</a>
</li>
<li><a href="code.html#airflow.models.Connection">Connection (class in airflow.models)</a>
</li>
- <li><a href="code.html#airflow.models.DAG.crawl_for_tasks">crawl_for_tasks() (airflow.models.DAG method)</a>
+ <li><a href="code.html#airflow.operators.SlackAPIOperator.construct_api_call_params">construct_api_call_params() (airflow.operators.SlackAPIOperator method)</a>
+</li>
+ <li><a href="code.html#airflow.operators.HiveToDruidTransfer.construct_ingest_query">construct_ingest_query() (airflow.operators.HiveToDruidTransfer method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.copy">copy() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.copy">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
+ <li><a href="code.html#airflow.hooks.PostgresHook.copy_expert">copy_expert() (airflow.hooks.PostgresHook method)</a>
</li>
<li><a href="code.html#airflow.models.DAG.create_dagrun">create_dagrun() (airflow.models.DAG method)</a>
</li>
<li><a href="code.html#airflow.contrib.hooks.FTPHook.create_directory">create_directory() (airflow.contrib.hooks.FTPHook method)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.create_job">create_job() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.create_model">create_model() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.create_version">create_version() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
<li><a href="code.html#airflow.models.TaskInstance.current_state">current_state() (airflow.models.TaskInstance method)</a>
</li>
</ul></td>
@@ -297,26 +353,68 @@
</li>
<li><a href="code.html#airflow.models.DagBag.dagbag_report">dagbag_report() (airflow.models.DagBag method)</a>
</li>
+ <li><a href="code.html#airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator">DatabricksSubmitRunOperator (class in airflow.contrib.operators.databricks_operator)</a>, <a href="integration.html#airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator">[1]</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook">DataFlowHook (class in airflow.contrib.hooks.gcp_dataflow_hook)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator">DataFlowJavaOperator (class in airflow.contrib.operators.dataflow_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator">DataFlowPythonOperator (class in airflow.contrib.operators.dataflow_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataproc_operator.DataProcHadoopOperator">DataProcHadoopOperator (class in airflow.contrib.operators.dataproc_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataproc_operator.DataProcHiveOperator">DataProcHiveOperator (class in airflow.contrib.operators.dataproc_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataproc_operator.DataProcPigOperator">DataProcPigOperator (class in airflow.contrib.operators.dataproc_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataproc_operator.DataProcPySparkOperator">DataProcPySparkOperator (class in airflow.contrib.operators.dataproc_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataproc_operator.DataProcSparkOperator">DataProcSparkOperator (class in airflow.contrib.operators.dataproc_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator">DataProcSparkSqlOperator (class in airflow.contrib.operators.dataproc_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook">DatastoreHook (class in airflow.contrib.hooks.datastore_hook)</a>
+</li>
<li><a href="code.html#airflow.hooks.DbApiHook">DbApiHook (class in airflow.hooks)</a>
</li>
<li><a href="code.html#airflow.models.DAG.deactivate_stale_dags">deactivate_stale_dags() (airflow.models.DAG static method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DAG.deactivate_unknown_dags">deactivate_unknown_dags() (airflow.models.DAG static method)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.delete">delete() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.delete">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
<li><a href="code.html#airflow.contrib.hooks.FTPHook.delete_directory">delete_directory() (airflow.contrib.hooks.FTPHook method)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.contrib.hooks.FTPHook.delete_file">delete_file() (airflow.contrib.hooks.FTPHook method)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.delete_operation">delete_operation() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.delete_version">delete_version() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
<li><a href="code.html#airflow.models.BaseOperator.deps">deps (airflow.models.BaseOperator attribute)</a>
</li>
<li><a href="code.html#airflow.contrib.hooks.FTPHook.describe_directory">describe_directory() (airflow.contrib.hooks.FTPHook method)</a>
</li>
<li><a href="code.html#airflow.models.BaseOperator.detect_downstream_cycle">detect_downstream_cycle() (airflow.models.BaseOperator method)</a>
</li>
+ <li><a href="code.html#airflow.operators.docker_operator.DockerOperator">DockerOperator (class in airflow.operators.docker_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.download">download() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.download">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
<li><a href="code.html#airflow.models.BaseOperator.downstream_list">downstream_list (airflow.models.BaseOperator attribute)</a>
</li>
+ <li><a href="code.html#airflow.hooks.DruidHook">DruidHook (class in airflow.hooks)</a>
+</li>
<li><a href="code.html#airflow.macros.ds_add">ds_add() (in module airflow.macros)</a>
</li>
<li><a href="code.html#airflow.macros.ds_format">ds_format() (in module airflow.macros)</a>
@@ -329,18 +427,38 @@
<h2 id="E">E</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="code.html#airflow.contrib.operators.ecs_operator.ECSOperator">ECSOperator (class in airflow.contrib.operators.ecs_operator)</a>, <a href="integration.html#airflow.contrib.operators.ecs_operator.ECSOperator">[1]</a>
+</li>
<li><a href="code.html#airflow.operators.EmailOperator">EmailOperator (class in airflow.operators)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator">EmrAddStepsOperator (class in airflow.contrib.operators.emr_add_steps_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator">EmrCreateJobFlowOperator (class in airflow.contrib.operators.emr_create_job_flow_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.emr_hook.EmrHook">EmrHook (class in airflow.contrib.hooks.emr_hook)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator">EmrTerminateJobFlowOperator (class in airflow.contrib.operators.emr_terminate_job_flow_operator)</a>
+</li>
<li><a href="code.html#airflow.models.TaskInstance.error">error() (airflow.models.TaskInstance method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.BaseOperator.execute">execute() (airflow.models.BaseOperator method)</a>
<ul>
<li><a href="code.html#airflow.operators.BashOperator.execute">(airflow.operators.BashOperator method)</a>
</li>
+ <li><a href="code.html#airflow.operators.SlackAPIOperator.execute">(airflow.operators.SlackAPIOperator method)</a>
+</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.exists">exists() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.exists">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.export_to_storage_bucket">export_to_storage_bucket() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.operators.ExternalTaskSensor">ExternalTaskSensor (class in airflow.operators)</a>
</li>
<li><a href="code.html#airflow.models.Connection.extra_dejson">extra_dejson (airflow.models.Connection attribute)</a>
@@ -353,6 +471,8 @@
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DAG.filepath">filepath (airflow.models.DAG attribute)</a>
</li>
+ <li><a href="code.html#airflow.contrib.operators.file_to_wasb.FileToWasbOperator">FileToWasbOperator (class in airflow.contrib.operators.file_to_wasb)</a>, <a href="integration.html#airflow.contrib.operators.file_to_wasb.FileToWasbOperator">[1]</a>
+</li>
<li><a href="code.html#airflow.operators.HdfsSensor.filter_for_filesize">filter_for_filesize() (airflow.operators.HdfsSensor static method)</a>
</li>
</ul></td>
@@ -375,9 +495,25 @@
</li>
<li><a href="code.html#airflow.models.DAG.get_active_runs">get_active_runs() (airflow.models.DAG method)</a>
</li>
- <li><a href="code.html#airflow.contrib.hooks.FTPHook.get_conn">get_conn() (airflow.contrib.hooks.FTPHook method)</a>
+ <li><a href="code.html#airflow.hooks.S3Hook.get_bucket">get_bucket() (airflow.hooks.S3Hook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_conn">get_conn() (airflow.contrib.hooks.bigquery_hook.BigQueryHook method)</a>
<ul>
+ <li><a href="code.html#airflow.contrib.hooks.BigQueryHook.get_conn">(airflow.contrib.hooks.BigQueryHook method)</a>
+</li>
+ <li><a href="code.html#airflow.contrib.hooks.FTPHook.get_conn">(airflow.contrib.hooks.FTPHook method)</a>
+</li>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.get_conn">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.get_conn">(airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook.get_conn">(airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.get_conn">(airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.get_conn">(airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+</li>
<li><a href="code.html#airflow.hooks.DbApiHook.get_conn">(airflow.hooks.DbApiHook method)</a>
</li>
<li><a href="code.html#airflow.hooks.HttpHook.get_conn">(airflow.hooks.HttpHook method)</a>
@@ -399,8 +535,8 @@
<li><a href="code.html#airflow.models.TaskInstance.get_dagrun">(airflow.models.TaskInstance method)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.get_databases">get_databases() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.models.BaseOperator.get_direct_relatives">get_direct_relatives() (airflow.models.BaseOperator method)</a>
</li>
<li><a href="code.html#airflow.hooks.DbApiHook.get_first">get_first() (airflow.hooks.DbApiHook method)</a>
@@ -411,24 +547,72 @@
</ul></li>
<li><a href="code.html#airflow.models.BaseOperator.get_flat_relatives">get_flat_relatives() (airflow.models.BaseOperator method)</a>
</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.get_key">get_key() (airflow.hooks.S3Hook method)</a>
+</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DAG.get_last_dagrun">get_last_dagrun() (airflow.models.DAG method)</a>
</li>
- <li><a href="code.html#airflow.hooks.DbApiHook.get_pandas_df">get_pandas_df() (airflow.hooks.DbApiHook method)</a>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.get_metastore_client">get_metastore_client() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.get_model">get_model() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
+ <li><a href="code.html#airflow.models.DAG.get_num_active_runs">get_num_active_runs() (airflow.models.DAG method)</a>
+</li>
+ <li><a href="code.html#airflow.models.DAG.get_num_task_instances">get_num_task_instances() (airflow.models.DAG static method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.get_operation">get_operation() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_pandas_df">get_pandas_df() (airflow.contrib.hooks.bigquery_hook.BigQueryHook method)</a>
<ul>
+ <li><a href="code.html#airflow.contrib.hooks.BigQueryHook.get_pandas_df">(airflow.contrib.hooks.BigQueryHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.DbApiHook.get_pandas_df">(airflow.hooks.DbApiHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveServer2Hook.get_pandas_df">(airflow.hooks.HiveServer2Hook method)</a>
+</li>
<li><a href="code.html#airflow.hooks.PrestoHook.get_pandas_df">(airflow.hooks.PrestoHook method)</a>
</li>
</ul></li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.get_partitions">get_partitions() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.hooks.DbApiHook.get_records">get_records() (airflow.hooks.DbApiHook method)</a>
<ul>
+ <li><a href="code.html#airflow.hooks.HiveServer2Hook.get_records">(airflow.hooks.HiveServer2Hook method)</a>
+</li>
<li><a href="code.html#airflow.hooks.PrestoHook.get_records">(airflow.hooks.PrestoHook method)</a>
</li>
</ul></li>
+ <li><a href="code.html#airflow.models.DAG.get_run_dates">get_run_dates() (airflow.models.DAG method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_service">get_service() (airflow.contrib.hooks.bigquery_hook.BigQueryHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.BigQueryHook.get_service">(airflow.contrib.hooks.BigQueryHook method)</a>
+</li>
+ </ul></li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.get_table">get_table() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.get_tables">get_tables() (airflow.hooks.HiveMetastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.models.BaseOperator.get_task_instances">get_task_instances() (airflow.models.BaseOperator method)</a>
</li>
<li><a href="code.html#airflow.models.DAG.get_template_env">get_template_env() (airflow.models.DAG method)</a>
</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.get_wildcard_key">get_wildcard_key() (airflow.hooks.S3Hook method)</a>
+</li>
+ <li><a href="code.html#airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator">GoogleCloudStorageDownloadOperator (class in airflow.contrib.operators.gcs_download_operator)</a>, <a href="integration.html#airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator">[1]</a>
+</li>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook">GoogleCloudStorageHook (class in airflow.contrib.hooks)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook">(class in airflow.contrib.hooks.gcs_hook)</a>, <a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook">[1]</a>
+</li>
+ </ul></li>
+ <li><a href="integration.html#airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator">GoogleCloudStorageToBigQueryOperator (class in airflow.contrib.operators.gcs_to_bq)</a>
+</li>
</ul></td>
</tr></table>
@@ -441,12 +625,24 @@
</li>
<li><a href="code.html#airflow.contrib.operators.hipchat_operator.HipChatAPIOperator">HipChatAPIOperator (class in airflow.contrib.operators.hipchat_operator)</a>
</li>
+ <li><a href="code.html#airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator">HipChatAPISendRoomNotificationOperator (class in airflow.contrib.operators.hipchat_operator)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveCliHook">HiveCliHook (class in airflow.hooks)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook">HiveMetastoreHook (class in airflow.hooks)</a>
+</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator">HipChatAPISendRoomNotificationOperator (class in airflow.contrib.operators.hipchat_operator)</a>
+ <li><a href="code.html#airflow.operators.HiveOperator">HiveOperator (class in airflow.operators)</a>
</li>
<li><a href="code.html#airflow.operators.HivePartitionSensor">HivePartitionSensor (class in airflow.operators)</a>
</li>
+ <li><a href="code.html#airflow.hooks.HiveServer2Hook">HiveServer2Hook (class in airflow.hooks)</a>
+</li>
+ <li><a href="code.html#airflow.operators.HiveToDruidTransfer">HiveToDruidTransfer (class in airflow.operators)</a>
+</li>
+ <li><a href="code.html#airflow.operators.HiveToMySqlTransfer">HiveToMySqlTransfer (class in airflow.operators)</a>
+</li>
<li><a href="code.html#airflow.hooks.HttpHook">HttpHook (class in airflow.hooks)</a>
</li>
<li><a href="code.html#airflow.operators.HttpSensor">HttpSensor (class in airflow.operators)</a>
@@ -457,16 +653,30 @@
<h2 id="I">I</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.import_from_storage_bucket">import_from_storage_bucket() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.models.TaskInstance.init_on_load">init_on_load() (airflow.models.TaskInstance method)</a>
</li>
- <li><a href="code.html#airflow.hooks.DbApiHook.insert_rows">insert_rows() (airflow.hooks.DbApiHook method)</a>
+ <li><a href="integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.insert_rows">insert_rows() (airflow.contrib.hooks.bigquery_hook.BigQueryHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.BigQueryHook.insert_rows">(airflow.contrib.hooks.BigQueryHook method)</a>
</li>
+ <li><a href="code.html#airflow.hooks.DbApiHook.insert_rows">(airflow.hooks.DbApiHook method)</a>
+</li>
+ </ul></li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DAG.is_paused">is_paused (airflow.models.DAG attribute)</a>
</li>
<li><a href="code.html#airflow.models.TaskInstance.is_premature">is_premature (airflow.models.TaskInstance attribute)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.is_updated_after">is_updated_after() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.is_updated_after">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
</ul></td>
</tr></table>
@@ -487,30 +697,70 @@
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DAG.latest_execution_date">latest_execution_date (airflow.models.DAG attribute)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list">list() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.list">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
+ <li><a href="code.html#airflow.contrib.hooks.FTPHook.list_directory">list_directory() (airflow.contrib.hooks.FTPHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.list_keys">list_keys() (airflow.hooks.S3Hook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.S3Hook.list_prefixes">list_prefixes() (airflow.hooks.S3Hook method)</a>
+</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.contrib.hooks.FTPHook.list_directory">list_directory() (airflow.contrib.hooks.FTPHook method)</a>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.list_versions">list_versions() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveCliHook.load_df">load_df() (airflow.hooks.HiveCliHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveCliHook.load_file">load_file() (airflow.hooks.HiveCliHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.hooks.S3Hook.load_file">(airflow.hooks.S3Hook method)</a>
+</li>
+ </ul></li>
+ <li><a href="code.html#airflow.hooks.S3Hook.load_string">load_string() (airflow.hooks.S3Hook method)</a>
</li>
<li><a href="code.html#airflow.executors.LocalExecutor">LocalExecutor (class in airflow.executors)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.lookup">lookup() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
</ul></td>
</tr></table>
<h2 id="M">M</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.macros.hive.max_partition">max_partition() (in module airflow.macros.hive)</a>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.max_partition">max_partition() (airflow.hooks.HiveMetastoreHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.macros.hive.max_partition">(in module airflow.macros.hive)</a>
</li>
+ </ul></li>
<li><a href="code.html#airflow.contrib.executors.mesos_executor.MesosExecutor">MesosExecutor (class in airflow.contrib.executors.mesos_executor)</a>
</li>
+ <li><a href="code.html#airflow.operators.MetastorePartitionSensor">MetastorePartitionSensor (class in airflow.operators)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator">MLEngineBatchPredictionOperator (class in airflow.contrib.operators.mlengine_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook">MLEngineHook (class in airflow.contrib.hooks.gcp_mlengine_hook)</a>
+</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.operators.MetastorePartitionSensor">MetastorePartitionSensor (class in airflow.operators)</a>
+ <li><a href="integration.html#airflow.contrib.operators.mlengine_operator.MLEngineModelOperator">MLEngineModelOperator (class in airflow.contrib.operators.mlengine_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator">MLEngineTrainingOperator (class in airflow.contrib.operators.mlengine_operator)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator">MLEngineVersionOperator (class in airflow.contrib.operators.mlengine_operator)</a>
</li>
<li><a href="code.html#airflow.hooks.MySqlHook">MySqlHook (class in airflow.hooks)</a>
</li>
<li><a href="code.html#airflow.operators.MySqlOperator">MySqlOperator (class in airflow.operators)</a>
</li>
+ <li><a href="code.html#airflow.operators.MySqlToHiveTransfer">MySqlToHiveTransfer (class in airflow.operators)</a>
+</li>
</ul></td>
</tr></table>
@@ -539,20 +789,24 @@
<h2 id="P">P</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.models.TaskInstance.pool_full">pool_full() (airflow.models.TaskInstance method)</a>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.poll_operation_until_done">poll_operation_until_done() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
</li>
- <li><a href="code.html#airflow.contrib.hooks.SSHHook.Popen">Popen() (airflow.contrib.hooks.SSHHook method)</a>
+ <li><a href="code.html#airflow.models.TaskInstance.pool_full">pool_full() (airflow.models.TaskInstance method)</a>
</li>
<li><a href="code.html#airflow.models.BaseOperator.post_execute">post_execute() (airflow.models.BaseOperator method)</a>
</li>
+ <li><a href="code.html#airflow.hooks.PostgresHook">PostgresHook (class in airflow.hooks)</a>
+</li>
+ <li><a href="code.html#airflow.operators.PostgresOperator">PostgresOperator (class in airflow.operators)</a>
+</li>
<li><a href="code.html#airflow.models.BaseOperator.pre_execute">pre_execute() (airflow.models.BaseOperator method)</a>
</li>
<li><a href="code.html#airflow.models.BaseOperator.prepare_template">prepare_template() (airflow.models.BaseOperator method)</a>
</li>
- <li><a href="code.html#airflow.operators.PrestoCheckOperator">PrestoCheckOperator (class in airflow.operators)</a>
-</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="code.html#airflow.operators.PrestoCheckOperator">PrestoCheckOperator (class in airflow.operators)</a>
+</li>
<li><a href="code.html#airflow.hooks.PrestoHook">PrestoHook (class in airflow.hooks)</a>
</li>
<li><a href="code.html#airflow.operators.PrestoIntervalCheckOperator">PrestoIntervalCheckOperator (class in airflow.operators)</a>
@@ -568,13 +822,23 @@
</ul></td>
</tr></table>
+<h2 id="Q">Q</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="code.html#airflow.contrib.operators.QuboleOperator">QuboleOperator (class in airflow.contrib.operators)</a>
+</li>
+ </ul></td>
+</tr></table>
+
<h2 id="R">R</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
- <li><a href="code.html#airflow.macros.random">random() (in module airflow.macros)</a>
+ <li><a href="code.html#airflow.hooks.S3Hook.read_key">read_key() (airflow.hooks.S3Hook method)</a>
</li>
<li><a href="code.html#airflow.models.TaskInstance.ready_for_retry">ready_for_retry() (airflow.models.TaskInstance method)</a>
</li>
+ <li><a href="integration.html#airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer">RedshiftToS3Transfer (class in airflow.operators.redshift_to_s3_operator)</a>
+</li>
<li><a href="code.html#airflow.models.TaskInstance.refresh_from_db">refresh_from_db() (airflow.models.TaskInstance method)</a>
</li>
<li><a href="code.html#airflow.contrib.hooks.FTPHook.rename">rename() (airflow.contrib.hooks.FTPHook method)</a>
@@ -587,6 +851,8 @@
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.rollback">rollback() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
<li><a href="code.html#airflow.hooks.DbApiHook.run">run() (airflow.hooks.DbApiHook method)</a>
<ul>
@@ -598,44 +864,62 @@
</li>
<li><a href="code.html#airflow.models.DAG.run">(airflow.models.DAG method)</a>
</li>
- <li><a href="code.html#airflow.models.TaskInstance.run">(airflow.models.TaskInstance method)</a>
-</li>
</ul></li>
<li><a href="code.html#airflow.hooks.HttpHook.run_and_check">run_and_check() (airflow.hooks.HttpHook method)</a>
</li>
+ <li><a href="code.html#airflow.hooks.HiveCliHook.run_cli">run_cli() (airflow.hooks.HiveCliHook method)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook.run_query">run_query() (airflow.contrib.hooks.datastore_hook.DatastoreHook method)</a>
+</li>
</ul></td>
</tr></table>
<h2 id="S">S</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.operators.s3_file_transform_operator.S3FileTransformOperator">S3FileTransformOperator (class in airflow.operators.s3_file_transform_operator)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.S3Hook">S3Hook (class in airflow.hooks)</a>
+
+ <ul>
+ <li><a href="integration.html#airflow.hooks.S3_hook.S3Hook">(class in airflow.hooks.S3_hook)</a>
+</li>
+ </ul></li>
<li><a href="code.html#airflow.operators.S3KeySensor">S3KeySensor (class in airflow.operators)</a>
</li>
+ <li><a href="code.html#airflow.operators.S3ToHiveTransfer">S3ToHiveTransfer (class in airflow.operators)</a>
+
+ <ul>
+ <li><a href="integration.html#airflow.operators.s3_to_hive_operator.S3ToHiveTransfer">(class in airflow.operators.s3_to_hive_operator)</a>
+</li>
+ </ul></li>
<li><a href="code.html#airflow.models.BaseOperator.schedule_interval">schedule_interval (airflow.models.BaseOperator attribute)</a>
</li>
<li><a href="code.html#airflow.executors.SequentialExecutor">SequentialExecutor (class in airflow.executors)</a>
</li>
+ <li><a href="integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.set_default_version">set_default_version() (airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook method)</a>
+</li>
<li><a href="code.html#airflow.models.DAG.set_dependency">set_dependency() (airflow.models.DAG method)</a>
</li>
<li><a href="code.html#airflow.models.BaseOperator.set_downstream">set_downstream() (airflow.models.BaseOperator method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.BaseOperator.set_upstream">set_upstream() (airflow.models.BaseOperator method)</a>
</li>
<li><a href="code.html#airflow.operators.ShortCircuitOperator">ShortCircuitOperator (class in airflow.operators)</a>
</li>
<li><a href="code.html#airflow.operators.SimpleHttpOperator">SimpleHttpOperator (class in airflow.operators)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DagBag.size">size() (airflow.models.DagBag method)</a>
</li>
- <li><a href="code.html#airflow.hooks.SqliteHook">SqliteHook (class in airflow.hooks)</a>
+ <li><a href="code.html#airflow.operators.SlackAPIOperator">SlackAPIOperator (class in airflow.operators)</a>
</li>
- <li><a href="code.html#airflow.operators.SqlSensor">SqlSensor (class in airflow.operators)</a>
+ <li><a href="code.html#airflow.operators.SlackAPIPostOperator">SlackAPIPostOperator (class in airflow.operators)</a>
</li>
- <li><a href="code.html#airflow.contrib.operators.SSHExecuteOperator">SSHExecuteOperator (class in airflow.contrib.operators)</a>
+ <li><a href="code.html#airflow.hooks.SqliteHook">SqliteHook (class in airflow.hooks)</a>
</li>
- <li><a href="code.html#airflow.contrib.hooks.SSHHook">SSHHook (class in airflow.contrib.hooks)</a>
+ <li><a href="code.html#airflow.operators.SqlSensor">SqlSensor (class in airflow.operators)</a>
</li>
<li><a href="code.html#airflow.contrib.hooks.FTPHook.store_file">store_file() (airflow.contrib.hooks.FTPHook method)</a>
</li>
@@ -643,7 +927,7 @@
</li>
<li><a href="code.html#airflow.models.DAG.subdags">subdags (airflow.models.DAG attribute)</a>
</li>
- <li><a href="code.html#airflow.models.DAG.sync_to_db">sync_to_db() (airflow.models.DAG static method)</a>
+ <li><a href="code.html#airflow.models.DAG.sync_to_db">sync_to_db() (airflow.models.DAG method)</a>
</li>
</ul></td>
</tr></table>
@@ -651,19 +935,29 @@
<h2 id="T">T</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook.table_exists">table_exists() (airflow.contrib.hooks.bigquery_hook.BigQueryHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.BigQueryHook.table_exists">(airflow.contrib.hooks.BigQueryHook method)</a>
+</li>
+ <li><a href="code.html#airflow.hooks.HiveMetastoreHook.table_exists">(airflow.hooks.HiveMetastoreHook method)</a>
+</li>
+ </ul></li>
<li><a href="code.html#airflow.models.TaskInstance">TaskInstance (class in airflow.models)</a>
</li>
+ <li><a href="code.html#airflow.hooks.HiveCliHook.test_hql">test_hql() (airflow.hooks.HiveCliHook method)</a>
+</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.operators.TimeSensor">TimeSensor (class in airflow.operators)</a>
</li>
<li><a href="code.html#airflow.models.DAG.topological_sort">topological_sort() (airflow.models.DAG method)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.DAG.tree_view">tree_view() (airflow.models.DAG method)</a>
</li>
<li><a href="code.html#airflow.operators.TriggerDagRunOperator">TriggerDagRunOperator (class in airflow.operators)</a>
</li>
- <li><a href="code.html#airflow.contrib.hooks.SSHHook.tunnel">tunnel() (airflow.contrib.hooks.SSHHook method)</a>
+ <li><a href="code.html#airflow.models.TaskInstance.try_number">try_number (airflow.models.TaskInstance attribute)</a>
</li>
</ul></td>
</tr></table>
@@ -671,6 +965,14 @@
<h2 id="U">U</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.upload">upload() (airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook method)</a>
+
+ <ul>
+ <li><a href="code.html#airflow.contrib.hooks.GoogleCloudStorageHook.upload">(airflow.contrib.hooks.GoogleCloudStorageHook method)</a>
+</li>
+ </ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="code.html#airflow.models.BaseOperator.upstream_list">upstream_list (airflow.models.BaseOperator attribute)</a>
</li>
</ul></td>
@@ -679,6 +981,14 @@
<h2 id="W">W</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.sensors.wasb_sensor.WasbBlobSensor">WasbBlobSensor (class in airflow.contrib.sensors.wasb_sensor)</a>
+</li>
+ <li><a href="integration.html#airflow.contrib.hooks.wasb_hook.WasbHook">WasbHook (class in airflow.contrib.hooks.wasb_hook)</a>
+</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="integration.html#airflow.contrib.sensors.wasb_sensor.WasbPrefixSensor">WasbPrefixSensor (class in airflow.contrib.sensors.wasb_sensor)</a>
+</li>
<li><a href="code.html#airflow.operators.WebHdfsSensor">WebHdfsSensor (class in airflow.operators)</a>
</li>
</ul></td>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/index.html
----------------------------------------------------------------------
diff --git a/index.html b/index.html
index eec44c1..9c53c75 100644
--- a/index.html
+++ b/index.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -225,7 +230,7 @@ unit of work and continuity.</p>
</li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="start.html#what-s-next">What’s Next?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="start.html#what-s-next">What’s Next?</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a><ul>
@@ -235,7 +240,7 @@ unit of work and continuity.</p>
</li>
<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a><ul>
<li class="toctree-l2"><a class="reference internal" href="tutorial.html#example-pipeline-definition">Example Pipeline definition</a></li>
-<li class="toctree-l2"><a class="reference internal" href="tutorial.html#it-s-a-dag-definition-file">It’s a DAG definition file</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#it-s-a-dag-definition-file">It’s a DAG definition file</a></li>
<li class="toctree-l2"><a class="reference internal" href="tutorial.html#importing-modules">Importing Modules</a></li>
<li class="toctree-l2"><a class="reference internal" href="tutorial.html#default-arguments">Default Arguments</a></li>
<li class="toctree-l2"><a class="reference internal" href="tutorial.html#instantiate-a-dag">Instantiate a DAG</a></li>
@@ -250,7 +255,7 @@ unit of work and continuity.</p>
<li class="toctree-l3"><a class="reference internal" href="tutorial.html#backfill">Backfill</a></li>
</ul>
</li>
-<li class="toctree-l2"><a class="reference internal" href="tutorial.html#what-s-next">What’s Next?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#what-s-next">What’s Next?</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a><ul>
@@ -258,6 +263,7 @@ unit of work and continuity.</p>
<li class="toctree-l2"><a class="reference internal" href="configuration.html#setting-up-a-backend">Setting up a Backend</a></li>
<li class="toctree-l2"><a class="reference internal" href="configuration.html#connections">Connections</a></li>
<li class="toctree-l2"><a class="reference internal" href="configuration.html#scaling-out-with-celery">Scaling Out with Celery</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#scaling-out-with-dask">Scaling Out with Dask</a></li>
<li class="toctree-l2"><a class="reference internal" href="configuration.html#logs">Logs</a></li>
<li class="toctree-l2"><a class="reference internal" href="configuration.html#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li>
<li class="toctree-l2"><a class="reference internal" href="configuration.html#integration-with-systemd">Integration with systemd</a></li>
@@ -324,7 +330,118 @@ unit of work and continuity.</p>
</li>
</ul>
</li>
-<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="cli.html#Positional Arguments">Positional Arguments</a></li>
+<li class="toctree-l2"><a class="reference internal" href="cli.html#Sub-commands:">Sub-commands:</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#resetdb">resetdb</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#render">render</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat1">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat1">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#variables">variables</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat2">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#connections">connections</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat3">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#pause">pause</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat2">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat4">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#task_failed_deps">task_failed_deps</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat3">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat5">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#version">version</a></li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#trigger_dag">trigger_dag</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat4">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat6">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#initdb">initdb</a></li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#test">test</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat5">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat7">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#unpause">unpause</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat6">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat8">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#dag_state">dag_state</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat7">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat9">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#run">run</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat8">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat10">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#list_tasks">list_tasks</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat9">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat11">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#backfill">backfill</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat10">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat12">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#list_dags">list_dags</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat13">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#kerberos">kerberos</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat11">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat14">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#worker">worker</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat15">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#webserver">webserver</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat16">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#flower">flower</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat17">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#scheduler">scheduler</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat18">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#task_state">task_state</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat12">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat19">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#pool">pool</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat20">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#serve_logs">serve_logs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#clear">clear</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Positional Arguments_repeat13">Positional Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="cli.html#Named Arguments_repeat21">Named Arguments</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="cli.html#upgradedb">upgradedb</a></li>
+</ul>
+</li>
+</ul>
+</li>
<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a><ul>
<li class="toctree-l2"><a class="reference internal" href="scheduler.html#dag-runs">DAG Runs</a></li>
<li class="toctree-l2"><a class="reference internal" href="scheduler.html#backfill-and-catchup">Backfill and Catchup</a></li>
@@ -349,7 +466,11 @@ unit of work and continuity.</p>
<li class="toctree-l2"><a class="reference internal" href="security.html#multi-tenancy">Multi-tenancy</a></li>
<li class="toctree-l2"><a class="reference internal" href="security.html#kerberos">Kerberos</a><ul>
<li class="toctree-l3"><a class="reference internal" href="security.html#limitations">Limitations</a></li>
-<li class="toctree-l3"><a class="reference internal" href="security.html#enabling-kerberos">Enabling kerberos</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#enabling-kerberos">Enabling kerberos</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="security.html#airflow">Airflow</a></li>
+<li class="toctree-l4"><a class="reference internal" href="security.html#hadoop">Hadoop</a></li>
+</ul>
+</li>
<li class="toctree-l3"><a class="reference internal" href="security.html#using-kerberos-authentication">Using kerberos authentication</a></li>
</ul>
</li>
@@ -364,8 +485,9 @@ unit of work and continuity.</p>
</li>
</ul>
</li>
-<li class="toctree-l2"><a class="reference internal" href="security.html#ssl">SSL</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="security.html#impersonation">Impersonation</a></li>
+<li class="toctree-l2"><a class="reference internal" href="security.html#ssl">SSL</a></li>
+<li class="toctree-l2"><a class="reference internal" href="security.html#impersonation">Impersonation</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="security.html#default-impersonation">Default Impersonation</a></li>
</ul>
</li>
</ul>
@@ -377,8 +499,46 @@ unit of work and continuity.</p>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="integration.html#aws-amazon-webservices">AWS: Amazon Webservices</a></li>
+<li class="toctree-l2"><a class="reference internal" href="integration.html#azure-microsoft-azure">Azure: Microsoft Azure</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#azure-blob-storage">Azure Blob Storage</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#wasbblobsensor">WasbBlobSensor</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#wasbprefixsensor">WasbPrefixSensor</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#filetowasboperator">FileToWasbOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#wasbhook">WasbHook</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="integration.html#aws-amazon-web-services">AWS: Amazon Web Services</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#aws-emr">AWS EMR</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#emraddstepsoperator">EmrAddStepsOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#emrcreatejobflowoperator">EmrCreateJobFlowOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#emrterminatejobflowoperator">EmrTerminateJobFlowOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#emrhook">EmrHook</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#aws-s3">AWS S3</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#s3filetransformoperator">S3FileTransformOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#s3tohivetransfer">S3ToHiveTransfer</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#s3hook">S3Hook</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#aws-ec2-container-service">AWS EC2 Container Service</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#ecsoperator">ECSOperator</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#aws-redshift">AWS RedShift</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#redshifttos3transfer">RedshiftToS3Transfer</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="integration.html#databricks">Databricks</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#databrickssubmitrunoperator">DatabricksSubmitRunOperator</a></li>
+</ul>
+</li>
<li class="toctree-l2"><a class="reference internal" href="integration.html#gcp-google-cloud-platform">GCP: Google Cloud Platform</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#logging">Logging</a></li>
<li class="toctree-l3"><a class="reference internal" href="integration.html#bigquery">BigQuery</a><ul>
<li class="toctree-l4"><a class="reference internal" href="integration.html#bigquery-operators">BigQuery Operators</a></li>
<li class="toctree-l4"><a class="reference internal" href="integration.html#bigqueryhook">BigQueryHook</a></li>
@@ -391,11 +551,15 @@ unit of work and continuity.</p>
</li>
<li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-dataproc">Cloud DataProc</a><ul>
<li class="toctree-l4"><a class="reference internal" href="integration.html#dataproc-operators">DataProc Operators</a></li>
-<li class="toctree-l4"><a class="reference internal" href="integration.html#dataprocpysparkoperator">DataProcPySparkOperator</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-datastore">Cloud Datastore</a><ul>
-<li class="toctree-l4"><a class="reference internal" href="integration.html#datastore-operators">Datastore Operators</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#datastorehook">DatastoreHook</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-ml-engine">Cloud ML Engine</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#cloud-ml-engine-operators">Cloud ML Engine Operators</a></li>
+<li class="toctree-l4"><a class="reference internal" href="integration.html#cloud-ml-engine-hook">Cloud ML Engine Hook</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="integration.html#cloud-storage">Cloud Storage</a><ul>
@@ -408,10 +572,10 @@ unit of work and continuity.</p>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="faq.html#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
-<li class="toctree-l2"><a class="reference internal" href="faq.html#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="faq.html#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="faq.html#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
<li class="toctree-l2"><a class="reference internal" href="faq.html#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto">Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?</a></li>
-<li class="toctree-l2"><a class="reference internal" href="faq.html#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="faq.html#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li>
<li class="toctree-l2"><a class="reference internal" href="faq.html#how-can-i-create-dags-dynamically">How can I create DAGs dynamically?</a></li>
<li class="toctree-l2"><a class="reference internal" href="faq.html#what-are-all-the-airflow-run-commands-in-my-process-list">What are all the <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span></code> commands in my process list?</a></li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/installation.html
----------------------------------------------------------------------
diff --git a/installation.html b/installation.html
index 5dd9401..455ffd4 100644
--- a/installation.html
+++ b/installation.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -171,29 +176,29 @@
<div class="section" id="getting-airflow">
<h2>Getting Airflow<a class="headerlink" href="#getting-airflow" title="Permalink to this headline">¶</a></h2>
<p>The easiest way to install the latest stable version of Airflow is with <code class="docutils literal"><span class="pre">pip</span></code>:</p>
-<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install airflow
+<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install apache-airflow
</pre></div>
</div>
<p>You can also install Airflow with support for extra features like <code class="docutils literal"><span class="pre">s3</span></code> or <code class="docutils literal"><span class="pre">postgres</span></code>:</p>
-<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install <span class="s2">"airflow[s3, postgres]"</span>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install <span class="s2">"apache-airflow[s3, postgres]"</span>
</pre></div>
</div>
</div>
<div class="section" id="extra-packages">
<h2>Extra Packages<a class="headerlink" href="#extra-packages" title="Permalink to this headline">¶</a></h2>
-<p>The <code class="docutils literal"><span class="pre">airflow</span></code> PyPI basic package only installs what’s needed to get started.
+<p>The <code class="docutils literal"><span class="pre">apache-airflow</span></code> PyPI basic package only installs what’s needed to get started.
Subpackages can be installed depending on what will be useful in your
-environment. For instance, if you don’t need connectivity with Postgres,
-you won’t have to go through the trouble of installing the <code class="docutils literal"><span class="pre">postgres-devel</span></code>
+environment. For instance, if you don’t need connectivity with Postgres,
+you won’t have to go through the trouble of installing the <code class="docutils literal"><span class="pre">postgres-devel</span></code>
yum package, or whatever equivalent applies on the distribution you are using.</p>
<p>Behind the scenes, Airflow does conditional imports of operators that require
these extra dependencies.</p>
-<p>Here’s the list of the subpackages and what they enable:</p>
+<p>Here’s the list of the subpackages and what they enable:</p>
<table border="1" class="docutils">
<colgroup>
-<col width="15%" />
-<col width="37%" />
-<col width="49%" />
+<col width="14%" />
+<col width="42%" />
+<col width="45%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">subpackage</th>
@@ -203,110 +208,114 @@ these extra dependencies.</p>
</thead>
<tbody valign="top">
<tr class="row-even"><td>all</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[all]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[all]</span></code></td>
<td>All Airflow features known to man</td>
</tr>
<tr class="row-odd"><td>all_dbs</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[all_dbs]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[all_dbs]</span></code></td>
<td>All databases integrations</td>
</tr>
<tr class="row-even"><td>async</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[async]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[async]</span></code></td>
<td>Async worker classes for gunicorn</td>
</tr>
<tr class="row-odd"><td>devel</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[devel]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[devel]</span></code></td>
<td>Minimum dev tools requirements</td>
</tr>
<tr class="row-even"><td>devel_hadoop</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[devel_hadoop]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[devel_hadoop]</span></code></td>
<td>Airflow + dependencies on the Hadoop stack</td>
</tr>
<tr class="row-odd"><td>celery</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[celery]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[celery]</span></code></td>
<td>CeleryExecutor</td>
</tr>
<tr class="row-even"><td>crypto</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[crypto]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[crypto]</span></code></td>
<td>Encrypt connection passwords in metadata db</td>
</tr>
<tr class="row-odd"><td>druid</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[druid]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[druid]</span></code></td>
<td>Druid.io related operators & hooks</td>
</tr>
<tr class="row-even"><td>gcp_api</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[gcp_api]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[gcp_api]</span></code></td>
<td>Google Cloud Platform hooks and operators
(using <code class="docutils literal"><span class="pre">google-api-python-client</span></code>)</td>
</tr>
<tr class="row-odd"><td>jdbc</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[jdbc]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[jdbc]</span></code></td>
<td>JDBC hooks and operators</td>
</tr>
<tr class="row-even"><td>hdfs</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[hdfs]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[hdfs]</span></code></td>
<td>HDFS hooks and operators</td>
</tr>
<tr class="row-odd"><td>hive</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[hive]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[hive]</span></code></td>
<td>All Hive related operators</td>
</tr>
<tr class="row-even"><td>kerberos</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[kerberos]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[kerberos]</span></code></td>
<td>kerberos integration for kerberized hadoop</td>
</tr>
<tr class="row-odd"><td>ldap</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[ldap]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[ldap]</span></code></td>
<td>ldap authentication for users</td>
</tr>
<tr class="row-even"><td>mssql</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[mssql]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[mssql]</span></code></td>
<td>Microsoft SQL operators and hook,
support as an Airflow backend</td>
</tr>
<tr class="row-odd"><td>mysql</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[mysql]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[mysql]</span></code></td>
<td>MySQL operators and hook, support as
an Airflow backend</td>
</tr>
<tr class="row-even"><td>password</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[password]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[password]</span></code></td>
<td>Password Authentication for users</td>
</tr>
<tr class="row-odd"><td>postgres</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[postgres]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[postgres]</span></code></td>
<td>Postgres operators and hook, support
as an Airflow backend</td>
</tr>
<tr class="row-even"><td>qds</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[qds]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[qds]</span></code></td>
<td>Enable QDS (qubole data services) support</td>
</tr>
<tr class="row-odd"><td>rabbitmq</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[rabbitmq]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[rabbitmq]</span></code></td>
<td>Rabbitmq support as a Celery backend</td>
</tr>
<tr class="row-even"><td>s3</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[s3]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[s3]</span></code></td>
<td><code class="docutils literal"><span class="pre">S3KeySensor</span></code>, <code class="docutils literal"><span class="pre">S3PrefixSensor</span></code></td>
</tr>
<tr class="row-odd"><td>samba</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[samba]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[samba]</span></code></td>
<td><code class="docutils literal"><span class="pre">Hive2SambaOperator</span></code></td>
</tr>
<tr class="row-even"><td>slack</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[slack]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[slack]</span></code></td>
<td><code class="docutils literal"><span class="pre">SlackAPIPostOperator</span></code></td>
</tr>
<tr class="row-odd"><td>vertica</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[vertica]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[vertica]</span></code></td>
<td>Vertica hook
support as an Airflow backend</td>
</tr>
<tr class="row-even"><td>cloudant</td>
-<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[cloudant]</span></code></td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[cloudant]</span></code></td>
<td>Cloudant hook</td>
</tr>
+<tr class="row-odd"><td>redis</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[redis]</span></code></td>
+<td>Redis hooks and sensors</td>
+</tr>
</tbody>
</table>
</div>
[28/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/dataproc_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/dataproc_operator.html b/_modules/airflow/contrib/operators/dataproc_operator.html
new file mode 100644
index 0000000..d49719e
--- /dev/null
+++ b/_modules/airflow/contrib/operators/dataproc_operator.html
@@ -0,0 +1,1139 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.dataproc_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.dataproc_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.dataproc_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">import</span> <span class="nn">time</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_dataproc_hook</span> <span class="k">import</span> <span class="n">DataProcHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.version</span> <span class="k">import</span> <span class="n">version</span>
+<span class="kn">from</span> <span class="nn">googleapiclient.errors</span> <span class="k">import</span> <span class="n">HttpError</span>
+
+
+<span class="k">class</span> <span class="nc">DataprocClusterCreateOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new cluster on Google Cloud Dataproc. The operator will wait until the</span>
+<span class="sd"> creation is successful or an error occurs in the creation process.</span>
+
+<span class="sd"> The parameters allow to configure the cluster. Please refer to</span>
+
+<span class="sd"> https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters</span>
+
+<span class="sd"> for a detailed explanation on the different parameters. Most of the configuration</span>
+<span class="sd"> parameters detailed in the link are available as a parameter to this operator.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'cluster_name'</span><span class="p">,]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">cluster_name</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span>
+ <span class="n">num_workers</span><span class="p">,</span>
+ <span class="n">zone</span><span class="p">,</span>
+ <span class="n">storage_bucket</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">init_actions_uris</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">metadata</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">image_version</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">properties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">master_machine_type</span><span class="o">=</span><span class="s1">'n1-standard-4'</span><span class="p">,</span>
+ <span class="n">master_disk_size</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span>
+ <span class="n">worker_machine_type</span><span class="o">=</span><span class="s1">'n1-standard-4'</span><span class="p">,</span>
+ <span class="n">worker_disk_size</span><span class="o">=</span><span class="mi">500</span><span class="p">,</span>
+ <span class="n">num_preemptible_workers</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
+ <span class="n">labels</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">region</span><span class="o">=</span><span class="s1">'global'</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">service_account</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">service_account_scopes</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataprocClusterCreateOperator.</span>
+
+<span class="sd"> For more info on the creation of a cluster through the API, have a look at:</span>
+
+<span class="sd"> https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters</span>
+
+<span class="sd"> :param cluster_name: The name of the DataProc cluster to create.</span>
+<span class="sd"> :type cluster_name: string</span>
+<span class="sd"> :param project_id: The ID of the google cloud project in which</span>
+<span class="sd"> to create the cluster</span>
+<span class="sd"> :type project_id: string</span>
+<span class="sd"> :param num_workers: The # of workers to spin up</span>
+<span class="sd"> :type num_workers: int</span>
+<span class="sd"> :param storage_bucket: The storage bucket to use, setting to None lets dataproc</span>
+<span class="sd"> generate a custom one for you</span>
+<span class="sd"> :type storage_bucket: string</span>
+<span class="sd"> :param init_actions_uris: List of GCS uri's containing</span>
+<span class="sd"> dataproc initialization scripts</span>
+<span class="sd"> :type init_actions_uris: list[string]</span>
+<span class="sd"> :param metadata: dict of key-value google compute engine metadata entries</span>
+<span class="sd"> to add to all instances</span>
+<span class="sd"> :type metadata: dict</span>
+<span class="sd"> :param image_version: the version of software inside the Dataproc cluster</span>
+<span class="sd"> :type image_version: string</span>
+<span class="sd"> :param properties: dict of properties to set on</span>
+<span class="sd"> config files (e.g. spark-defaults.conf), see</span>
+<span class="sd"> https://cloud.google.com/dataproc/docs/reference/rest/v1/ \</span>
+<span class="sd"> projects.regions.clusters#SoftwareConfig</span>
+<span class="sd"> :type properties: dict</span>
+<span class="sd"> :param master_machine_type: Compute engine machine type to use for the master node</span>
+<span class="sd"> :type master_machine_type: string</span>
+<span class="sd"> :param master_disk_size: Disk size for the master node</span>
+<span class="sd"> :type int</span>
+<span class="sd"> :param worker_machine_type:Compute engine machine type to use for the worker nodes</span>
+<span class="sd"> :type worker_machine_type: string</span>
+<span class="sd"> :param worker_disk_size: Disk size for the worker nodes</span>
+<span class="sd"> :type worker_disk_size: int</span>
+<span class="sd"> :param num_preemptible_workers: The # of preemptible worker nodes to spin up</span>
+<span class="sd"> :type num_preemptible_workers: int</span>
+<span class="sd"> :param labels: dict of labels to add to the cluster</span>
+<span class="sd"> :type labels: dict</span>
+<span class="sd"> :param zone: The zone where the cluster will be located</span>
+<span class="sd"> :type zone: string</span>
+<span class="sd"> :param region: leave as 'global', might become relevant in the future</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> :param service_account: The service account of the dataproc instances.</span>
+<span class="sd"> :type service_account: string</span>
+<span class="sd"> :param service_account_scopes: The URIs of service account scopes to be included.</span>
+<span class="sd"> :type service_account_scopes: list[string]</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataprocClusterCreateOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span> <span class="o">=</span> <span class="n">cluster_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">num_workers</span> <span class="o">=</span> <span class="n">num_workers</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">num_preemptible_workers</span> <span class="o">=</span> <span class="n">num_preemptible_workers</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">storage_bucket</span> <span class="o">=</span> <span class="n">storage_bucket</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">init_actions_uris</span> <span class="o">=</span> <span class="n">init_actions_uris</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span> <span class="o">=</span> <span class="n">metadata</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">image_version</span> <span class="o">=</span> <span class="n">image_version</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">properties</span> <span class="o">=</span> <span class="n">properties</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">master_machine_type</span> <span class="o">=</span> <span class="n">master_machine_type</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">master_disk_size</span> <span class="o">=</span> <span class="n">master_disk_size</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">worker_machine_type</span> <span class="o">=</span> <span class="n">worker_machine_type</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">worker_disk_size</span> <span class="o">=</span> <span class="n">worker_disk_size</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">labels</span> <span class="o">=</span> <span class="n">labels</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">zone</span> <span class="o">=</span> <span class="n">zone</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">region</span> <span class="o">=</span> <span class="n">region</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">service_account</span> <span class="o">=</span> <span class="n">service_account</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">service_account_scopes</span> <span class="o">=</span> <span class="n">service_account_scopes</span>
+
+ <span class="k">def</span> <span class="nf">_get_cluster_list_for_project</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">):</span>
+ <span class="n">result</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">regions</span><span class="p">()</span><span class="o">.</span><span class="n">clusters</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">region</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">region</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'clusters'</span><span class="p">,</span> <span class="p">[])</span>
+
+ <span class="k">def</span> <span class="nf">_get_cluster</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">):</span>
+ <span class="n">cluster_list</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cluster_list_for_project</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+ <span class="n">cluster</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">cluster_list</span> <span class="k">if</span> <span class="n">c</span><span class="p">[</span><span class="s1">'clusterName'</span><span class="p">]</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">cluster</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">cluster</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="k">return</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">_get_cluster_state</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">):</span>
+ <span class="n">cluster</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cluster</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+ <span class="k">if</span> <span class="s1">'status'</span> <span class="ow">in</span> <span class="n">cluster</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">cluster</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'state'</span><span class="p">]</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">None</span>
+
+ <span class="k">def</span> <span class="nf">_cluster_ready</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">state</span><span class="p">,</span> <span class="n">service</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">state</span> <span class="o">==</span> <span class="s1">'RUNNING'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">if</span> <span class="n">state</span> <span class="o">==</span> <span class="s1">'ERROR'</span><span class="p">:</span>
+ <span class="n">cluster</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cluster</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">error_details</span> <span class="o">=</span> <span class="n">cluster</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'details'</span><span class="p">]</span>
+ <span class="k">except</span> <span class="ne">KeyError</span><span class="p">:</span>
+ <span class="n">error_details</span> <span class="o">=</span> <span class="s1">'Unknown error in cluster creation, '</span> \
+ <span class="s1">'check Google Cloud console for details.'</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="n">error_details</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">False</span>
+
+ <span class="k">def</span> <span class="nf">_wait_for_done</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">):</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="n">state</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cluster_state</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">state</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No state for cluster '</span><span class="si">%s</span><span class="s2">'"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">)</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"State for cluster '</span><span class="si">%s</span><span class="s2">' is </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">,</span> <span class="n">state</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_cluster_ready</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">service</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Cluster '</span><span class="si">%s</span><span class="s2">' successfully created"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span>
+ <span class="p">)</span>
+ <span class="k">return</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_build_cluster_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">zone_uri</span> <span class="o">=</span> \
+ <span class="s1">'https://www.googleapis.com/compute/v1/projects/</span><span class="si">{}</span><span class="s1">/zones/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">zone</span>
+ <span class="p">)</span>
+ <span class="n">master_type_uri</span> <span class="o">=</span> \
+ <span class="s2">"https://www.googleapis.com/compute/v1/projects/</span><span class="si">{}</span><span class="s2">/zones/</span><span class="si">{}</span><span class="s2">/machineTypes/</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">zone</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">master_machine_type</span>
+ <span class="p">)</span>
+ <span class="n">worker_type_uri</span> <span class="o">=</span> \
+ <span class="s2">"https://www.googleapis.com/compute/v1/projects/</span><span class="si">{}</span><span class="s2">/zones/</span><span class="si">{}</span><span class="s2">/machineTypes/</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">zone</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">worker_machine_type</span>
+ <span class="p">)</span>
+ <span class="n">cluster_data</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'projectId'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="s1">'clusterName'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">,</span>
+ <span class="s1">'config'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'gceClusterConfig'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'zoneUri'</span><span class="p">:</span> <span class="n">zone_uri</span>
+ <span class="p">},</span>
+ <span class="s1">'masterConfig'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'numInstances'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="s1">'machineTypeUri'</span><span class="p">:</span> <span class="n">master_type_uri</span><span class="p">,</span>
+ <span class="s1">'diskConfig'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'bootDiskSizeGb'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">master_disk_size</span>
+ <span class="p">}</span>
+ <span class="p">},</span>
+ <span class="s1">'workerConfig'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'numInstances'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_workers</span><span class="p">,</span>
+ <span class="s1">'machineTypeUri'</span><span class="p">:</span> <span class="n">worker_type_uri</span><span class="p">,</span>
+ <span class="s1">'diskConfig'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'bootDiskSizeGb'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">worker_disk_size</span>
+ <span class="p">}</span>
+ <span class="p">},</span>
+ <span class="s1">'secondaryWorkerConfig'</span><span class="p">:</span> <span class="p">{},</span>
+ <span class="s1">'softwareConfig'</span><span class="p">:</span> <span class="p">{}</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_preemptible_workers</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'secondaryWorkerConfig'</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'numInstances'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_preemptible_workers</span><span class="p">,</span>
+ <span class="s1">'machineTypeUri'</span><span class="p">:</span> <span class="n">worker_type_uri</span><span class="p">,</span>
+ <span class="s1">'diskConfig'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'bootDiskSizeGb'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">worker_disk_size</span>
+ <span class="p">},</span>
+ <span class="s1">'isPreemptible'</span><span class="p">:</span> <span class="kc">True</span>
+ <span class="p">}</span>
+
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'labels'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">labels</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">labels</span> <span class="k">else</span> <span class="p">{}</span>
+ <span class="c1"># Dataproc labels must conform to the following regex:</span>
+ <span class="c1"># [a-z]([-a-z0-9]*[a-z0-9])? (current airflow version string follows</span>
+ <span class="c1"># semantic versioning spec: x.y.z).</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'labels'</span><span class="p">]</span><span class="o">.</span><span class="n">update</span><span class="p">({</span><span class="s1">'airflow-version'</span><span class="p">:</span>
+ <span class="s1">'v'</span> <span class="o">+</span> <span class="n">version</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">)})</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">storage_bucket</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'configBucket'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">storage_bucket</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'gceClusterConfig'</span><span class="p">][</span><span class="s1">'metadata'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metadata</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">image_version</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'softwareConfig'</span><span class="p">][</span><span class="s1">'imageVersion'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">image_version</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">properties</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'softwareConfig'</span><span class="p">][</span><span class="s1">'properties'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">properties</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">init_actions_uris</span><span class="p">:</span>
+ <span class="n">init_actions_dict</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="p">{</span><span class="s1">'executableFile'</span><span class="p">:</span> <span class="n">uri</span><span class="p">}</span> <span class="k">for</span> <span class="n">uri</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">init_actions_uris</span>
+ <span class="p">]</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'initializationActions'</span><span class="p">]</span> <span class="o">=</span> <span class="n">init_actions_dict</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_account</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'gceClusterConfig'</span><span class="p">][</span><span class="s1">'serviceAccount'</span><span class="p">]</span> <span class="o">=</span>\
+ <span class="bp">self</span><span class="o">.</span><span class="n">service_account</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_account_scopes</span><span class="p">:</span>
+ <span class="n">cluster_data</span><span class="p">[</span><span class="s1">'config'</span><span class="p">][</span><span class="s1">'gceClusterConfig'</span><span class="p">][</span><span class="s1">'serviceAccountScopes'</span><span class="p">]</span> <span class="o">=</span>\
+ <span class="bp">self</span><span class="o">.</span><span class="n">service_account_scopes</span>
+ <span class="k">return</span> <span class="n">cluster_data</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Creating cluster: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataProcHook</span><span class="p">(</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span>
+ <span class="p">)</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cluster</span><span class="p">(</span><span class="n">service</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Cluster </span><span class="si">%s</span><span class="s1"> already exists... Checking status...'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_done</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">True</span>
+
+ <span class="n">cluster_data</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_build_cluster_data</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">regions</span><span class="p">()</span><span class="o">.</span><span class="n">clusters</span><span class="p">()</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">region</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">region</span><span class="p">,</span>
+ <span class="n">body</span><span class="o">=</span><span class="n">cluster_data</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">except</span> <span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="c1"># probably two cluster start commands at the same time</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_cluster</span><span class="p">(</span><span class="n">service</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Cluster </span><span class="si">{}</span><span class="s1"> already exists... Checking status...'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_done</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">e</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_done</span><span class="p">(</span><span class="n">service</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">DataprocClusterDeleteOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Delete a cluster on Google Cloud Dataproc. The operator will wait until the</span>
+<span class="sd"> cluster is destroyed.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'cluster_name'</span><span class="p">]</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">cluster_name</span><span class="p">,</span>
+ <span class="n">project_id</span><span class="p">,</span>
+ <span class="n">region</span><span class="o">=</span><span class="s1">'global'</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Delete a cluster on Google Cloud Dataproc.</span>
+
+<span class="sd"> :param cluster_name: The name of the cluster to create.</span>
+<span class="sd"> :type cluster_name: string</span>
+<span class="sd"> :param project_id: The ID of the google cloud project in which</span>
+<span class="sd"> the cluster runs</span>
+<span class="sd"> :type project_id: string</span>
+<span class="sd"> :param region: leave as 'global', might become relevant in the future</span>
+<span class="sd"> :type region: string</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataprocClusterDeleteOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span> <span class="o">=</span> <span class="n">cluster_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">project_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">region</span> <span class="o">=</span> <span class="n">region</span>
+
+ <span class="k">def</span> <span class="nf">_wait_for_done</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">,</span> <span class="n">operation_name</span><span class="p">):</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">regions</span><span class="p">()</span><span class="o">.</span><span class="n">operations</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">operation_name</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="s1">'done'</span> <span class="ow">in</span> <span class="n">response</span> <span class="ow">and</span> <span class="n">response</span><span class="p">[</span><span class="s1">'done'</span><span class="p">]:</span>
+ <span class="k">if</span> <span class="s1">'error'</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">response</span><span class="p">[</span><span class="s1">'error'</span><span class="p">]))</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">15</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Deleting cluster: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">)</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataProcHook</span><span class="p">(</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span>
+ <span class="p">)</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="n">response</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">regions</span><span class="p">()</span><span class="o">.</span><span class="n">clusters</span><span class="p">()</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span>
+ <span class="n">projectId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span>
+ <span class="n">region</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">region</span><span class="p">,</span>
+ <span class="n">clusterName</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="n">operation_name</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Cluster delete operation name: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">operation_name</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_done</span><span class="p">(</span><span class="n">service</span><span class="p">,</span> <span class="n">operation_name</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="DataProcPigOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.dataproc_operator.DataProcPigOperator">[docs]</a><span class="k">class</span> <span class="nc">DataProcPigOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Start a Pig query Job on a Cloud DataProc cluster. The parameters of the operation</span>
+<span class="sd"> will be passed to the cluster.</span>
+
+<span class="sd"> It's a good practice to define dataproc_* parameters in the default_args of the dag</span>
+<span class="sd"> like the cluster name and UDFs.</span>
+
+<span class="sd"> ```</span>
+<span class="sd"> default_args = {</span>
+<span class="sd"> 'cluster_name': 'cluster-1',</span>
+<span class="sd"> 'dataproc_pig_jars': [</span>
+<span class="sd"> 'gs://example/udf/jar/datafu/1.2.0/datafu.jar',</span>
+<span class="sd"> 'gs://example/udf/jar/gpig/1.2/gpig.jar'</span>
+<span class="sd"> ]</span>
+<span class="sd"> }</span>
+<span class="sd"> ```</span>
+
+<span class="sd"> You can pass a pig script as string or file reference. Use variables to pass on</span>
+<span class="sd"> variables for the pig script to be resolved on the cluster or use the parameters to</span>
+<span class="sd"> be resolved in the script as template parameters.</span>
+
+<span class="sd"> ```</span>
+<span class="sd"> t1 = DataProcPigOperator(</span>
+<span class="sd"> task_id='dataproc_pig',</span>
+<span class="sd"> query='a_pig_script.pig',</span>
+<span class="sd"> variables={'out': 'gs://example/output/{{ds}}'},</span>
+<span class="sd"> dag=dag)</span>
+<span class="sd"> ```</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'query'</span><span class="p">,</span> <span class="s1">'variables'</span><span class="p">,</span> <span class="s1">'job_name'</span><span class="p">,</span> <span class="s1">'cluster_name'</span><span class="p">]</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.pg'</span><span class="p">,</span> <span class="s1">'.pig'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#0273d4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">query</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">query_uri</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">variables</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">job_name</span><span class="o">=</span><span class="s1">'{{task.task_id}}_{{ds_nodash}}'</span><span class="p">,</span>
+ <span class="n">cluster_name</span><span class="o">=</span><span class="s1">'cluster-1'</span><span class="p">,</span>
+ <span class="n">dataproc_pig_properties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">dataproc_pig_jars</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataProcPigOperator.</span>
+
+<span class="sd"> For more detail on about job submission have a look at the reference:</span>
+
+<span class="sd"> https://cloud.google.com/dataproc/reference/rest/v1/projects.regions.jobs</span>
+
+<span class="sd"> :param query: The query or reference to the query file (pg or pig extension).</span>
+<span class="sd"> :type query: string</span>
+<span class="sd"> :param query_uri: The uri of a pig script on Cloud Storage.</span>
+<span class="sd"> :type query_uri: string</span>
+<span class="sd"> :param variables: Map of named parameters for the query.</span>
+<span class="sd"> :type variables: dict</span>
+<span class="sd"> :param job_name: The job name used in the DataProc cluster. This name by default</span>
+<span class="sd"> is the task_id appended with the execution data, but can be templated. The</span>
+<span class="sd"> name will always be appended with a random number to avoid name clashes.</span>
+<span class="sd"> :type job_name: string</span>
+<span class="sd"> :param cluster_name: The name of the DataProc cluster.</span>
+<span class="sd"> :type cluster_name: string</span>
+<span class="sd"> :param dataproc_pig_properties: Map for the Pig properties. Ideal to put in</span>
+<span class="sd"> default arguments</span>
+<span class="sd"> :type dataproc_pig_properties: dict</span>
+<span class="sd"> :param dataproc_pig_jars: URIs to jars provisioned in Cloud Storage (example: for</span>
+<span class="sd"> UDFs and libs) and are ideal to put in default arguments.</span>
+<span class="sd"> :type dataproc_pig_jars: list</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataProcPigOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query</span> <span class="o">=</span> <span class="n">query</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query_uri</span> <span class="o">=</span> <span class="n">query_uri</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">variables</span> <span class="o">=</span> <span class="n">variables</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">job_name</span> <span class="o">=</span> <span class="n">job_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span> <span class="o">=</span> <span class="n">cluster_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_properties</span> <span class="o">=</span> <span class="n">dataproc_pig_properties</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_jars</span> <span class="o">=</span> <span class="n">dataproc_pig_jars</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataProcHook</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="n">job</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_job_template</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">,</span> <span class="s2">"pigJob"</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_properties</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_query_uri</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">query_uri</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_variables</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">variables</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_jar_file_uris</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dataproc_jars</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">set_job_name</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">job_name</span><span class="p">)</span>
+
+ <span class="n">hook</span><span class="o">.</span><span class="n">submit</span><span class="p">(</span><span class="n">hook</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job</span><span class="o">.</span><span class="n">build</span><span class="p">())</span></div>
+
+
+<div class="viewcode-block" id="DataProcHiveOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.dataproc_operator.DataProcHiveOperator">[docs]</a><span class="k">class</span> <span class="nc">DataProcHiveOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Start a Hive query Job on a Cloud DataProc cluster.</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'query'</span><span class="p">,</span> <span class="s1">'variables'</span><span class="p">,</span> <span class="s1">'job_name'</span><span class="p">,</span> <span class="s1">'cluster_name'</span><span class="p">]</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.q'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#0273d4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">query</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">query_uri</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">variables</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">job_name</span><span class="o">=</span><span class="s1">'{{task.task_id}}_{{ds_nodash}}'</span><span class="p">,</span>
+ <span class="n">cluster_name</span><span class="o">=</span><span class="s1">'cluster-1'</span><span class="p">,</span>
+ <span class="n">dataproc_hive_properties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">dataproc_hive_jars</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataProcHiveOperator.</span>
+
+<span class="sd"> :param query: The query or reference to the query file (q extension).</span>
+<span class="sd"> :type query: string</span>
+<span class="sd"> :param query_uri: The uri of a hive script on Cloud Storage.</span>
+<span class="sd"> :type query_uri: string</span>
+<span class="sd"> :param variables: Map of named parameters for the query.</span>
+<span class="sd"> :type variables: dict</span>
+<span class="sd"> :param job_name: The job name used in the DataProc cluster. This name by default</span>
+<span class="sd"> is the task_id appended with the execution data, but can be templated. The</span>
+<span class="sd"> name will always be appended with a random number to avoid name clashes.</span>
+<span class="sd"> :type job_name: string</span>
+<span class="sd"> :param cluster_name: The name of the DataProc cluster.</span>
+<span class="sd"> :type cluster_name: string</span>
+<span class="sd"> :param dataproc_hive_properties: Map for the Pig properties. Ideal to put in</span>
+<span class="sd"> default arguments</span>
+<span class="sd"> :type dataproc_hive_properties: dict</span>
+<span class="sd"> :param dataproc_hive_jars: URIs to jars provisioned in Cloud Storage (example: for</span>
+<span class="sd"> UDFs and libs) and are ideal to put in default arguments.</span>
+<span class="sd"> :type dataproc_hive_jars: list</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataProcHiveOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query</span> <span class="o">=</span> <span class="n">query</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query_uri</span> <span class="o">=</span> <span class="n">query_uri</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">variables</span> <span class="o">=</span> <span class="n">variables</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">job_name</span> <span class="o">=</span> <span class="n">job_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span> <span class="o">=</span> <span class="n">cluster_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_properties</span> <span class="o">=</span> <span class="n">dataproc_hive_properties</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_jars</span> <span class="o">=</span> <span class="n">dataproc_hive_jars</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataProcHook</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+
+ <span class="n">job</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_job_template</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">,</span> <span class="s2">"hiveJob"</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_properties</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_query_uri</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">query_uri</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_variables</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">variables</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_jar_file_uris</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dataproc_jars</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">set_job_name</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">job_name</span><span class="p">)</span>
+
+ <span class="n">hook</span><span class="o">.</span><span class="n">submit</span><span class="p">(</span><span class="n">hook</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job</span><span class="o">.</span><span class="n">build</span><span class="p">())</span></div>
+
+
+<div class="viewcode-block" id="DataProcSparkSqlOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator">[docs]</a><span class="k">class</span> <span class="nc">DataProcSparkSqlOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Start a Spark SQL query Job on a Cloud DataProc cluster.</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'query'</span><span class="p">,</span> <span class="s1">'variables'</span><span class="p">,</span> <span class="s1">'job_name'</span><span class="p">,</span> <span class="s1">'cluster_name'</span><span class="p">]</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.q'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#0273d4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">query</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">query_uri</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">variables</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">job_name</span><span class="o">=</span><span class="s1">'{{task.task_id}}_{{ds_nodash}}'</span><span class="p">,</span>
+ <span class="n">cluster_name</span><span class="o">=</span><span class="s1">'cluster-1'</span><span class="p">,</span>
+ <span class="n">dataproc_spark_properties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">dataproc_spark_jars</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataProcSparkSqlOperator.</span>
+
+<span class="sd"> :param query: The query or reference to the query file (q extension).</span>
+<span class="sd"> :type query: string</span>
+<span class="sd"> :param query_uri: The uri of a spark sql script on Cloud Storage.</span>
+<span class="sd"> :type query_uri: string</span>
+<span class="sd"> :param variables: Map of named parameters for the query.</span>
+<span class="sd"> :type variables: dict</span>
+<span class="sd"> :param job_name: The job name used in the DataProc cluster. This name by default</span>
+<span class="sd"> is the task_id appended with the execution data, but can be templated. The</span>
+<span class="sd"> name will always be appended with a random number to avoid name clashes.</span>
+<span class="sd"> :type job_name: string</span>
+<span class="sd"> :param cluster_name: The name of the DataProc cluster.</span>
+<span class="sd"> :type cluster_name: string</span>
+<span class="sd"> :param dataproc_spark_properties: Map for the Pig properties. Ideal to put in</span>
+<span class="sd"> default arguments</span>
+<span class="sd"> :type dataproc_spark_properties: dict</span>
+<span class="sd"> :param dataproc_spark_jars: URIs to jars provisioned in Cloud Storage (example:</span>
+<span class="sd"> for UDFs and libs) and are ideal to put in default arguments.</span>
+<span class="sd"> :type dataproc_spark_jars: list</span>
+<span class="sd"> :param gcp_conn_id: The connection ID to use connecting to Google Cloud Platform.</span>
+<span class="sd"> :type gcp_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DataProcSparkSqlOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span> <span class="o">=</span> <span class="n">gcp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query</span> <span class="o">=</span> <span class="n">query</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query_uri</span> <span class="o">=</span> <span class="n">query_uri</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">variables</span> <span class="o">=</span> <span class="n">variables</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">job_name</span> <span class="o">=</span> <span class="n">job_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span> <span class="o">=</span> <span class="n">cluster_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_properties</span> <span class="o">=</span> <span class="n">dataproc_spark_properties</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_jars</span> <span class="o">=</span> <span class="n">dataproc_spark_jars</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">DataProcHook</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">gcp_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+
+ <span class="n">job</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">create_job_template</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cluster_name</span><span class="p">,</span> <span class="s2">"sparkSqlJob"</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dataproc_properties</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">query</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_query_uri</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">query_uri</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_query</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">query</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_variables</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">variables</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">add_jar_file_uris</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dataproc_jars</span><span class="p">)</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">set_job_name</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">job_name</span><span class="p">)</span>
+
+ <span class="n">hook</span><span class="o">.</span><span class="n">submit</span><span class="p">(</span><span class="n">hook</span><span class="o">.</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job</span><span class="o">.</span><span class="n">build</span><span class="p">())</span></div>
+
+
+<div class="viewcode-block" id="DataProcSparkOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.dataproc_operator.DataProcSparkOperator">[docs]</a><span class="k">class</span> <span class="nc">DataProcSparkOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Start a Spark Job on a Cloud DataProc cluster.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'arguments'</span><span class="p">,</span> <span class="s1">'job_name'</span><span class="p">,</span> <span class="s1">'cluster_name'</span><span class="p">]</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#0273d4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">main_jar</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">main_class</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">arguments</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">archives</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">files</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">job_name</span><span class="o">=</span><span class="s1">'{{task.task_id}}_{{ds_nodash}}'</span><span class="p">,</span>
+ <span class="n">cluster_name</span><span class="o">=</span><span class="s1">'cluster-1'</span><span class="p">,</span>
+ <span class="n">dataproc_spark_properties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">dataproc_spark_jars</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a new DataProcSparkOperator.</span>
+
+<span class="sd"> :param main_jar: URI of the job jar provisioned on Cloud Storage. (use this or</span>
+<span class="sd"> the main_class, not both together).</span>
+<span class="sd"> :type main_jar: string</span>
+<span class="sd"> :param main_class: Name of the job class. (use this or the main_jar, not both</span>
+<span class="sd"> together).</span>
+<span class="sd"> :type main_class: string</span>
+<span class="sd"> :param arguments: Arguments for the job.</span>
+<span class="sd"> :type arguments: list</span>
+<span class="sd"> :param archives: List of archived files that will be unpacked in the work</span>
+<span class="sd"> directory. Should be stored in Cloud Storage.</span>
+<span class="sd"> :type archives: list</span>
+<span class="sd"> :param files: List of files to be copied to the working directory</span>
+<span class="sd"> :type files: list</span>
+<span class="sd"> :param job_name: The job name used in the DataProc cluster. This name by default</span>
+<span class="sd"> is the task_id appended with the execution data, but can be templated. The</span>
+<span class="sd"> name will always be appended with a random number to avoid name clashes.</span>
+<span class="sd"> :type job_name: string</span>
+<span class="sd"> :param cluster_name: The name of the DataProc cluster.</span>
+<span class="sd"> :type cluster_name: string</span>
+<span class="sd"> :param dataproc_spark_properties: Map for the Pig properties. Ideal to put in</span>
+<span class="sd"> default arguments</span>
+<span class="sd"> :type dataproc_spark_properties: dict</span>
+<span class="sd"> :param dataproc_spark_jars: URIs to jars provisioned in Cloud Storage (example:</span>
+<span class="sd"> for UDFs and libs) and are ideal to put in de
<TRUNCATED>
[11/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/s3_to_hive_operator.html
----------------------------------------------------------------------
diff --git a/_modules/s3_to_hive_operator.html b/_modules/s3_to_hive_operator.html
index ad7715e..840a81b 100644
--- a/_modules/s3_to_hive_operator.html
+++ b/_modules/s3_to_hive_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>s3_to_hive_operator</li>
+
+ <li>s3_to_hive_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,16 +181,21 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">next</span>
-<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">zip</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">next</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">zip</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="k">import</span> <span class="n">TemporaryDirectory</span>
+<span class="kn">import</span> <span class="nn">gzip</span>
+<span class="kn">import</span> <span class="nn">bz2</span>
+<span class="kn">import</span> <span class="nn">tempfile</span>
+<span class="kn">import</span> <span class="nn">os</span>
-<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="kn">import</span> <span class="n">S3Hook</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.S3_hook</span> <span class="k">import</span> <span class="n">S3Hook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveCliHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.compression</span> <span class="k">import</span> <span class="n">uncompress_file</span>
<div class="viewcode-block" id="S3ToHiveTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.S3ToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">S3ToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -205,10 +240,15 @@
<span class="sd"> :type wildcard_match: bool</span>
<span class="sd"> :param delimiter: field delimiter in the file</span>
<span class="sd"> :type delimiter: str</span>
-<span class="sd"> :param s3_conn_id: source s3 connection</span>
-<span class="sd"> :type s3_conn_id: str</span>
-<span class="sd"> :param hive_conn_id: destination hive connection</span>
-<span class="sd"> :type hive_conn_id: str</span>
+<span class="sd"> :param aws_conn_id: source s3 connection</span>
+<span class="sd"> :type aws_conn_id: str</span>
+<span class="sd"> :param hive_cli_conn_id: destination hive connection</span>
+<span class="sd"> :type hive_cli_conn_id: str</span>
+<span class="sd"> :param input_compressed: Boolean to determine if file decompression is</span>
+<span class="sd"> required to process headers</span>
+<span class="sd"> :type input_compressed: bool</span>
+<span class="sd"> :param tblproperties: TBLPROPERTIES of the hive table being created</span>
+<span class="sd"> :type tblproperties: dict</span>
<span class="sd"> """</span>
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'s3_key'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
@@ -222,16 +262,18 @@
<span class="n">field_dict</span><span class="p">,</span>
<span class="n">hive_table</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
- <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
- <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">headers</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">check_headers</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">wildcard_match</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">check_headers</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">wildcard_match</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'aws_default'</span><span class="p">,</span>
<span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="n">input_compressed</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">S3ToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3ToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span> <span class="o">=</span> <span class="n">s3_key</span>
<span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span> <span class="o">=</span> <span class="n">field_dict</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
@@ -243,29 +285,42 @@
<span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span> <span class="o">=</span> <span class="n">check_headers</span>
<span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">input_compressed</span> <span class="o">=</span> <span class="n">input_compressed</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span> <span class="o">=</span> <span class="n">tblproperties</span>
+
+ <span class="k">if</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span> <span class="ow">and</span>
+ <span class="ow">not</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">)):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"To check_headers provide "</span> <span class="o">+</span>
+ <span class="s2">"field_dict and headers"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="c1"># Downloading file from S3</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Downloading S3 file"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Downloading S3 file"</span><span class="p">)</span>
+
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">):</span>
- <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"No key matches {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"No key matches </span><span class="si">{0}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
<span class="n">s3_key_object</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">):</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
- <span class="s2">"The key {0} does not exists"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
+ <span class="s2">"The key </span><span class="si">{0}</span><span class="s2"> does not exists"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
<span class="n">s3_key_object</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">)</span>
- <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping S3 key {0} contents to local"</span>
- <span class="s2">" file {1}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">s3_key_object</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
- <span class="n">s3_key_object</span><span class="o">.</span><span class="n">get_contents_to_file</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
+ <span class="n">root</span><span class="p">,</span> <span class="n">file_ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">s3_key_object</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'tmps32hive_'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">,</span>\
+ <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="n">mode</span><span class="o">=</span><span class="s2">"wb"</span><span class="p">,</span>
+ <span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">,</span>
+ <span class="n">suffix</span><span class="o">=</span><span class="n">file_ext</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping S3 key </span><span class="si">{0}</span><span class="s2"> contents to local file </span><span class="si">{1}</span><span class="s2">"</span>
+ <span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">s3_key_object</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
+ <span class="n">s3_key_object</span><span class="o">.</span><span class="n">download_fileobj</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file </span><span class="si">%s</span><span class="s2"> into Hive"</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
<span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
@@ -273,41 +328,100 @@
<span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
<span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
- <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
- <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmpf</span><span class="p">:</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span><span class="p">:</span>
- <span class="n">header_l</span> <span class="o">=</span> <span class="n">tmpf</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
- <span class="n">header_line</span> <span class="o">=</span> <span class="n">header_l</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
- <span class="n">header_list</span> <span class="o">=</span> <span class="n">header_line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">)</span>
- <span class="n">field_names</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
- <span class="n">test_field_match</span> <span class="o">=</span> <span class="p">[</span><span class="n">h1</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">h2</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="k">for</span> <span class="n">h1</span><span class="p">,</span> <span class="n">h2</span>
- <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">header_list</span><span class="p">,</span> <span class="n">field_names</span><span class="p">)]</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="n">test_field_match</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Headers do not match field names"</span>
- <span class="s2">"File headers:</span><span class="se">\n</span><span class="s2"> {header_list}</span><span class="se">\n</span><span class="s2">"</span>
- <span class="s2">"Field names: </span><span class="se">\n</span><span class="s2"> {field_names}</span><span class="se">\n</span><span class="s2">"</span>
- <span class="s2">""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
- <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Headers do not match the "</span>
- <span class="s2">"field_dict keys"</span><span class="p">)</span>
- <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_no_headers</span><span class="p">:</span>
- <span class="n">tmpf</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
- <span class="nb">next</span><span class="p">(</span><span class="n">tmpf</span><span class="p">)</span>
- <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">tmpf</span><span class="p">:</span>
- <span class="n">f_no_headers</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
- <span class="n">f_no_headers</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file without headers into Hive"</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
- <span class="n">f_no_headers</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
- <span class="n">field_dict</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="p">,</span>
- <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
- <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
- <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
- <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span></div>
+ <span class="c1"># Decompressing file</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">input_compressed</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Uncompressing file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
+ <span class="n">fn_uncompressed</span> <span class="o">=</span> <span class="n">uncompress_file</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="n">file_ext</span><span class="p">,</span>
+ <span class="n">tmp_dir</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Uncompressed to </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">fn_uncompressed</span><span class="p">)</span>
+ <span class="c1"># uncompressed file available now so deleting</span>
+ <span class="c1"># compressed file to save disk space</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">fn_uncompressed</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span>
+
+ <span class="c1"># Testing if header matches field_dict</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Matching file header against field_dict"</span><span class="p">)</span>
+ <span class="n">header_list</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_top_row_as_list</span><span class="p">(</span><span class="n">fn_uncompressed</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">_match_headers</span><span class="p">(</span><span class="n">header_list</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Header check failed"</span><span class="p">)</span>
+
+ <span class="c1"># Deleting top header row</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Removing header from file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">fn_uncompressed</span><span class="p">)</span>
+ <span class="n">headless_file</span> <span class="o">=</span> <span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_delete_top_row_and_compress</span><span class="p">(</span><span class="n">fn_uncompressed</span><span class="p">,</span>
+ <span class="n">file_ext</span><span class="p">,</span>
+ <span class="n">tmp_dir</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Headless file </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">headless_file</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file </span><span class="si">%s</span><span class="s2"> into Hive"</span><span class="p">,</span> <span class="n">headless_file</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span><span class="n">headless_file</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tblproperties</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_get_top_row_as_list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_name</span><span class="p">):</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_name</span><span class="p">,</span> <span class="s1">'rt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">header_line</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
+ <span class="n">header_list</span> <span class="o">=</span> <span class="n">header_line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">header_list</span>
+
+ <span class="k">def</span> <span class="nf">_match_headers</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">header_list</span><span class="p">):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">header_list</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Unable to retrieve header row from file"</span><span class="p">)</span>
+ <span class="n">field_names</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="o">.</span><span class="n">keys</span><span class="p">()</span>
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">field_names</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="n">header_list</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Headers count mismatch"</span>
+ <span class="s2">"File headers:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{header_list}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">"Field names: </span><span class="se">\n</span><span class="s2"> </span><span class="si">{field_names}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="n">test_field_match</span> <span class="o">=</span> <span class="p">[</span><span class="n">h1</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">h2</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">h1</span><span class="p">,</span> <span class="n">h2</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">header_list</span><span class="p">,</span> <span class="n">field_names</span><span class="p">)]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="n">test_field_match</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Headers do not match field names"</span>
+ <span class="s2">"File headers:</span><span class="se">\n</span><span class="s2"> </span><span class="si">{header_list}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">"Field names: </span><span class="se">\n</span><span class="s2"> </span><span class="si">{field_names}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">True</span>
+
+ <span class="k">def</span> <span class="nf">_delete_top_row_and_compress</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">input_file_name</span><span class="p">,</span>
+ <span class="n">output_file_ext</span><span class="p">,</span>
+ <span class="n">dest_dir</span><span class="p">):</span>
+ <span class="c1"># When output_file_ext is not defined, file is not compressed</span>
+ <span class="n">open_fn</span> <span class="o">=</span> <span class="nb">open</span>
+ <span class="k">if</span> <span class="n">output_file_ext</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'.gz'</span><span class="p">:</span>
+ <span class="n">open_fn</span> <span class="o">=</span> <span class="n">gzip</span><span class="o">.</span><span class="n">GzipFile</span>
+ <span class="k">elif</span> <span class="n">output_file_ext</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'.bz2'</span><span class="p">:</span>
+ <span class="n">open_fn</span> <span class="o">=</span> <span class="n">bz2</span><span class="o">.</span><span class="n">BZ2File</span>
+
+ <span class="n">os_fh_output</span><span class="p">,</span> <span class="n">fn_output</span> <span class="o">=</span> \
+ <span class="n">tempfile</span><span class="o">.</span><span class="n">mkstemp</span><span class="p">(</span><span class="n">suffix</span><span class="o">=</span><span class="n">output_file_ext</span><span class="p">,</span> <span class="nb">dir</span><span class="o">=</span><span class="n">dest_dir</span><span class="p">)</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">input_file_name</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_in</span><span class="p">,</span>\
+ <span class="n">open_fn</span><span class="p">(</span><span class="n">fn_output</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_out</span><span class="p">:</span>
+ <span class="n">f_in</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
+ <span class="nb">next</span><span class="p">(</span><span class="n">f_in</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f_in</span><span class="p">:</span>
+ <span class="n">f_out</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">fn_output</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -340,7 +454,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
[04/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/integration.html
----------------------------------------------------------------------
diff --git a/integration.html b/integration.html
index a55aa41..2f4ff20 100644
--- a/integration.html
+++ b/integration.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -97,8 +102,46 @@
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Integration</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="#aws-amazon-webservices">AWS: Amazon Webservices</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#azure-microsoft-azure">Azure: Microsoft Azure</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#azure-blob-storage">Azure Blob Storage</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#wasbblobsensor">WasbBlobSensor</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#wasbprefixsensor">WasbPrefixSensor</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#filetowasboperator">FileToWasbOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#wasbhook">WasbHook</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#aws-amazon-web-services">AWS: Amazon Web Services</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#aws-emr">AWS EMR</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#emraddstepsoperator">EmrAddStepsOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#emrcreatejobflowoperator">EmrCreateJobFlowOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#emrterminatejobflowoperator">EmrTerminateJobFlowOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#emrhook">EmrHook</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#aws-s3">AWS S3</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#s3filetransformoperator">S3FileTransformOperator</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#s3tohivetransfer">S3ToHiveTransfer</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#s3hook">S3Hook</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#aws-ec2-container-service">AWS EC2 Container Service</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#ecsoperator">ECSOperator</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#aws-redshift">AWS RedShift</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#redshifttos3transfer">RedshiftToS3Transfer</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#databricks">Databricks</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#databrickssubmitrunoperator">DatabricksSubmitRunOperator</a></li>
+</ul>
+</li>
<li class="toctree-l2"><a class="reference internal" href="#gcp-google-cloud-platform">GCP: Google Cloud Platform</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#logging">Logging</a></li>
<li class="toctree-l3"><a class="reference internal" href="#bigquery">BigQuery</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#bigquery-operators">BigQuery Operators</a></li>
<li class="toctree-l4"><a class="reference internal" href="#bigqueryhook">BigQueryHook</a></li>
@@ -111,11 +154,15 @@
</li>
<li class="toctree-l3"><a class="reference internal" href="#cloud-dataproc">Cloud DataProc</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#dataproc-operators">DataProc Operators</a></li>
-<li class="toctree-l4"><a class="reference internal" href="#dataprocpysparkoperator">DataProcPySparkOperator</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#cloud-datastore">Cloud Datastore</a><ul>
-<li class="toctree-l4"><a class="reference internal" href="#datastore-operators">Datastore Operators</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#datastorehook">DatastoreHook</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#cloud-ml-engine">Cloud ML Engine</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#cloud-ml-engine-operators">Cloud ML Engine Operators</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#cloud-ml-engine-hook">Cloud ML Engine Hook</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#cloud-storage">Cloud Storage</a><ul>
@@ -195,18 +242,601 @@
<div class="section" id="integration">
<h1>Integration<a class="headerlink" href="#integration" title="Permalink to this headline">¶</a></h1>
<ul class="simple">
-<li><a class="reference internal" href="#aws"><span class="std std-ref">AWS: Amazon Webservices</span></a></li>
+<li><a class="reference internal" href="#azure"><span class="std std-ref">Azure: Microsoft Azure</span></a></li>
+<li><a class="reference internal" href="#aws"><span class="std std-ref">AWS: Amazon Web Services</span></a></li>
+<li><span class="xref std std-ref">Databricks</span></li>
<li><a class="reference internal" href="#gcp"><span class="std std-ref">GCP: Google Cloud Platform</span></a></li>
</ul>
-<div class="section" id="aws-amazon-webservices">
-<span id="aws"></span><h2>AWS: Amazon Webservices<a class="headerlink" href="#aws-amazon-webservices" title="Permalink to this headline">¶</a></h2>
-<p>—</p>
+<div class="section" id="azure-microsoft-azure">
+<span id="azure"></span><h2>Azure: Microsoft Azure<a class="headerlink" href="#azure-microsoft-azure" title="Permalink to this headline">¶</a></h2>
+<p>Airflow has limited support for Microsoft Azure: interfaces exist only for Azure Blob
+Storage. Note that the Hook, Sensor and Operator are in the contrib section.</p>
+<div class="section" id="azure-blob-storage">
+<h3>Azure Blob Storage<a class="headerlink" href="#azure-blob-storage" title="Permalink to this headline">¶</a></h3>
+<p>All classes communicate via the Window Azure Storage Blob protocol. Make sure that a
+Airflow connection of type <cite>wasb</cite> exists. Authorization can be done by supplying a
+login (=Storage account name) and password (=KEY), or login and SAS token in the extra
+field (see connection <cite>wasb_default</cite> for an example).</p>
+<ul class="simple">
+<li><a class="reference internal" href="#wasbblobsensor"><span class="std std-ref">WasbBlobSensor</span></a>: Checks if a blob is present on Azure Blob storage.</li>
+<li><a class="reference internal" href="#wasbprefixsensor"><span class="std std-ref">WasbPrefixSensor</span></a>: Checks if blobs matching a prefix are present on Azure Blob storage.</li>
+<li><a class="reference internal" href="#filetowasboperator"><span class="std std-ref">FileToWasbOperator</span></a>: Uploads a local file to a container as a blob.</li>
+<li><a class="reference internal" href="#wasbhook"><span class="std std-ref">WasbHook</span></a>: Interface with Azure Blob Storage.</li>
+</ul>
+<div class="section" id="wasbblobsensor">
+<span id="id1"></span><h4>WasbBlobSensor<a class="headerlink" href="#wasbblobsensor" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.sensors.wasb_sensor.WasbBlobSensor">
+<em class="property">class </em><code class="descclassname">airflow.contrib.sensors.wasb_sensor.</code><code class="descname">WasbBlobSensor</code><span class="sig-paren">(</span><em>container_name</em>, <em>blob_name</em>, <em>wasb_conn_id='wasb_default'</em>, <em>check_options=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/sensors/wasb_sensor.html#WasbBlobSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.sensors.wasb_sensor.WasbBlobSensor" title="Permalink to this definition">¶</a></dt>
+<dd><p>Waits for a blob to arrive on Azure Blob Storage.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>container_name</strong> (<em>str</em>) – Name of the container.</li>
+<li><strong>blob_name</strong> (<em>str</em>) – Name of the blob.</li>
+<li><strong>wasb_conn_id</strong> (<em>str</em>) – Reference to the wasb connection.</li>
+<li><strong>check_options</strong> (<em>dict</em>) – Optional keyword arguments that
+<cite>WasbHook.check_for_blob()</cite> takes.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="wasbprefixsensor">
+<span id="id2"></span><h4>WasbPrefixSensor<a class="headerlink" href="#wasbprefixsensor" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.sensors.wasb_sensor.WasbPrefixSensor">
+<em class="property">class </em><code class="descclassname">airflow.contrib.sensors.wasb_sensor.</code><code class="descname">WasbPrefixSensor</code><span class="sig-paren">(</span><em>container_name</em>, <em>prefix</em>, <em>wasb_conn_id='wasb_default'</em>, <em>check_options=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/sensors/wasb_sensor.html#WasbPrefixSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.sensors.wasb_sensor.WasbPrefixSensor" title="Permalink to this definition">¶</a></dt>
+<dd><p>Waits for blobs matching a prefix to arrive on Azure Blob Storage.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>container_name</strong> (<em>str</em>) – Name of the container.</li>
+<li><strong>prefix</strong> (<em>str</em>) – Prefix of the blob.</li>
+<li><strong>wasb_conn_id</strong> (<em>str</em>) – Reference to the wasb connection.</li>
+<li><strong>check_options</strong> (<em>dict</em>) – Optional keyword arguments that
+<cite>WasbHook.check_for_prefix()</cite> takes.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="filetowasboperator">
+<span id="id3"></span><h4>FileToWasbOperator<a class="headerlink" href="#filetowasboperator" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.operators.file_to_wasb.FileToWasbOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.file_to_wasb.</code><code class="descname">FileToWasbOperator</code><span class="sig-paren">(</span><em>file_path</em>, <em>container_name</em>, <em>blob_name</em>, <em>wasb_conn_id='wasb_default'</em>, <em>load_options=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/file_to_wasb.html#FileToWasbOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.file_to_wasb.FileToWasbOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Uploads a file to Azure Blob Storage.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>file_path</strong> (<em>str</em>) – Path to the file to load.</li>
+<li><strong>container_name</strong> (<em>str</em>) – Name of the container.</li>
+<li><strong>blob_name</strong> (<em>str</em>) – Name of the blob.</li>
+<li><strong>wasb_conn_id</strong> (<em>str</em>) – Reference to the wasb connection.</li>
+<li><strong>load_options</strong> (<em>dict</em>) – Optional keyword arguments that
+<cite>WasbHook.load_file()</cite> takes.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="wasbhook">
+<span id="id4"></span><h4>WasbHook<a class="headerlink" href="#wasbhook" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.hooks.wasb_hook.WasbHook">
+<em class="property">class </em><code class="descclassname">airflow.contrib.hooks.wasb_hook.</code><code class="descname">WasbHook</code><span class="sig-paren">(</span><em>wasb_conn_id='wasb_default'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/wasb_hook.html#WasbHook"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.wasb_hook.WasbHook" title="Permalink to this definition">¶</a></dt>
+<dd><p>Interacts with Azure Blob Storage through the wasb:// protocol.</p>
+<p>Additional options passed in the ‘extra’ field of the connection will be
+passed to the <cite>BlockBlockService()</cite> constructor. For example, authenticate
+using a SAS token by adding {“sas_token”: “YOUR_TOKEN”}.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>wasb_conn_id</strong> (<em>str</em>) – Reference to the wasb connection.</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+</div>
+</div>
+<div class="section" id="aws-amazon-web-services">
+<span id="aws"></span><h2>AWS: Amazon Web Services<a class="headerlink" href="#aws-amazon-web-services" title="Permalink to this headline">¶</a></h2>
+<p>Airflow has extensive support for Amazon Web Services. But note that the Hooks, Sensors and
+Operators are in the contrib section.</p>
+<div class="section" id="aws-emr">
+<h3>AWS EMR<a class="headerlink" href="#aws-emr" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li><a class="reference internal" href="#emraddstepsoperator"><span class="std std-ref">EmrAddStepsOperator</span></a> : Adds steps to an existing EMR JobFlow.</li>
+<li><a class="reference internal" href="#emrcreatejobflowoperator"><span class="std std-ref">EmrCreateJobFlowOperator</span></a> : Creates an EMR JobFlow, reading the config from the EMR connection.</li>
+<li><a class="reference internal" href="#emrterminatejobflowoperator"><span class="std std-ref">EmrTerminateJobFlowOperator</span></a> : Terminates an EMR JobFlow.</li>
+<li><a class="reference internal" href="#emrhook"><span class="std std-ref">EmrHook</span></a> : Interact with AWS EMR.</li>
+</ul>
+<div class="section" id="emraddstepsoperator">
+<span id="id5"></span><h4>EmrAddStepsOperator<a class="headerlink" href="#emraddstepsoperator" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.emr_add_steps_operator.</code><code class="descname">EmrAddStepsOperator</code><span class="sig-paren">(</span><em>job_flow_id</em>, <em>aws_conn_id='s3_default'</em>, <em>steps=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/emr_add_steps_operator.html#EmrAddStepsOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>An operator that adds steps to an existing EMR job_flow.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>job_flow_id</strong> – id of the JobFlow to add steps to</li>
+<li><strong>aws_conn_id</strong> (<em>str</em>) – aws connection to uses</li>
+<li><strong>steps</strong> (<a class="reference internal" href="#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – boto3 style steps to be added to the jobflow</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="emrcreatejobflowoperator">
+<span id="id6"></span><h4>EmrCreateJobFlowOperator<a class="headerlink" href="#emrcreatejobflowoperator" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.emr_create_job_flow_operator.</code><code class="descname">EmrCreateJobFlowOperator</code><span class="sig-paren">(</span><em>aws_conn_id='s3_default'</em>, <em>emr_conn_id='emr_default'</em>, <em>job_flow_overrides=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/emr_create_job_flow_operator.html#EmrCreateJobFlowOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Creates an EMR JobFlow, reading the config from the EMR connection.
+A dictionary of JobFlow overrides can be passed that override the config from the connection.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>aws_conn_id</strong> (<em>str</em>) – aws connection to uses</li>
+<li><strong>emr_conn_id</strong> (<em>str</em>) – emr connection to use</li>
+<li><strong>job_flow_overrides</strong> – boto3 style arguments to override emr_connection extra</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="emrterminatejobflowoperator">
+<span id="id7"></span><h4>EmrTerminateJobFlowOperator<a class="headerlink" href="#emrterminatejobflowoperator" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.emr_terminate_job_flow_operator.</code><code class="descname">EmrTerminateJobFlowOperator</code><span class="sig-paren">(</span><em>job_flow_id</em>, <em>aws_conn_id='s3_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/emr_terminate_job_flow_operator.html#EmrTerminateJobFlowOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Operator to terminate EMR JobFlows.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>job_flow_id</strong> – id of the JobFlow to terminate</li>
+<li><strong>aws_conn_id</strong> (<em>str</em>) – aws connection to uses</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="emrhook">
+<span id="id8"></span><h4>EmrHook<a class="headerlink" href="#emrhook" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.hooks.emr_hook.EmrHook">
+<em class="property">class </em><code class="descclassname">airflow.contrib.hooks.emr_hook.</code><code class="descname">EmrHook</code><span class="sig-paren">(</span><em>emr_conn_id=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/emr_hook.html#EmrHook"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.emr_hook.EmrHook" title="Permalink to this definition">¶</a></dt>
+<dd><p>Interact with AWS EMR. emr_conn_id is only neccessary for using the create_job_flow method.</p>
+</dd></dl>
+
+</div>
+</div>
+<div class="section" id="aws-s3">
+<h3>AWS S3<a class="headerlink" href="#aws-s3" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li><a class="reference internal" href="#s3filetransformoperator"><span class="std std-ref">S3FileTransformOperator</span></a> : Copies data from a source S3 location to a temporary location on the local filesystem.</li>
+<li><a class="reference internal" href="#s3tohivetransfer"><span class="std std-ref">S3ToHiveTransfer</span></a> : Moves data from S3 to Hive. The operator downloads a file from S3, stores the file locally before loading it into a Hive table.</li>
+<li><a class="reference internal" href="#s3hook"><span class="std std-ref">S3Hook</span></a> : Interact with AWS S3.</li>
+</ul>
+<div class="section" id="s3filetransformoperator">
+<span id="id9"></span><h4>S3FileTransformOperator<a class="headerlink" href="#s3filetransformoperator" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.operators.s3_file_transform_operator.S3FileTransformOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.s3_file_transform_operator.</code><code class="descname">S3FileTransformOperator</code><span class="sig-paren">(</span><em>source_s3_key</em>, <em>dest_s3_key</em>, <em>transform_script</em>, <em>source_aws_conn_id='aws_default'</em>, <em>dest_aws_conn_id='aws_default'</em>, <em>replace=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/s3_file_transform_operator.html#S3FileTransformOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.s3_file_transform_operator.S3FileTransformOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Copies data from a source S3 location to a temporary location on the
+local filesystem. Runs a transformation on this file as specified by
+the transformation script and uploads the output to a destination S3
+location.</p>
+<p>The locations of the source and the destination files in the local
+filesystem is provided as an first and second arguments to the
+transformation script. The transformation script is expected to read the
+data from source , transform it and write the output to the local
+destination file. The operator then takes over control and uploads the
+local destination file to S3.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>source_s3_key</strong> (<em>str</em>) – The key to be retrieved from S3</li>
+<li><strong>source_aws_conn_id</strong> (<em>str</em>) – source s3 connection</li>
+<li><strong>dest_s3_key</strong> (<em>str</em>) – The key to be written from S3</li>
+<li><strong>dest_aws_conn_id</strong> (<em>str</em>) – destination s3 connection</li>
+<li><strong>replace</strong> (<em>bool</em>) – Replace dest S3 key if it already exists</li>
+<li><strong>transform_script</strong> (<em>str</em>) – location of the executable transformation script</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="s3tohivetransfer">
+<span id="id10"></span><h4>S3ToHiveTransfer<a class="headerlink" href="#s3tohivetransfer" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.operators.s3_to_hive_operator.S3ToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.s3_to_hive_operator.</code><code class="descname">S3ToHiveTransfer</code><span class="sig-paren">(</span><em>s3_key</em>, <em>field_dict</em>, <em>hive_table</em>, <em>delimiter='</em>, <em>'</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>headers=False</em>, <em>check_headers=False</em>, <em>wildcard_match=False</em>, <em>aws_conn_id='aws_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>input_compressed=False</em>, <em>tblproperties=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/s3_to_hive_operator.html#S3ToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.s3_to_hive_operator.S3ToHiveTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Moves data from S3 to Hive. The operator downloads a file from S3,
+stores the file locally before loading it into a Hive table.
+If the <code class="docutils literal"><span class="pre">create</span></code> or <code class="docutils literal"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal"><span class="pre">True</span></code>,
+a <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata from.</p>
+<p>Note that the table generated in Hive uses <code class="docutils literal"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the tables gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>s3_key</strong> (<em>str</em>) – The key to be retrieved from S3</li>
+<li><strong>field_dict</strong> (<em>dict</em>) – A dictionary of the fields name in the file
+as keys and their Hive types as values</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a
+specific database</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every
+execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns
+and values</li>
+<li><strong>headers</strong> (<em>bool</em>) – whether the file contains column names on the first
+line</li>
+<li><strong>check_headers</strong> (<em>bool</em>) – whether the column names on the first line should be
+checked against the keys of field_dict</li>
+<li><strong>wildcard_match</strong> (<em>bool</em>) – whether the s3_key should be interpreted as a Unix
+wildcard pattern</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>aws_conn_id</strong> (<em>str</em>) – source s3 connection</li>
+<li><strong>hive_cli_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>input_compressed</strong> (<em>bool</em>) – Boolean to determine if file decompression is
+required to process headers</li>
+<li><strong>tblproperties</strong> (<em>dict</em>) – TBLPROPERTIES of the hive table being created</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="s3hook">
+<span id="id11"></span><h4>S3Hook<a class="headerlink" href="#s3hook" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.hooks.S3_hook.S3Hook">
+<em class="property">class </em><code class="descclassname">airflow.hooks.S3_hook.</code><code class="descname">S3Hook</code><span class="sig-paren">(</span><em>aws_conn_id='aws_default'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/hooks/S3_hook.html#S3Hook"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.hooks.S3_hook.S3Hook" title="Permalink to this definition">¶</a></dt>
+<dd><p>Interact with AWS S3, using the boto3 library.</p>
+</dd></dl>
+
+</div>
+</div>
+<div class="section" id="aws-ec2-container-service">
+<h3>AWS EC2 Container Service<a class="headerlink" href="#aws-ec2-container-service" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li><a class="reference internal" href="#ecsoperator"><span class="std std-ref">ECSOperator</span></a> : Execute a task on AWS EC2 Container Service.</li>
+</ul>
+<div class="section" id="ecsoperator">
+<span id="id12"></span><h4>ECSOperator<a class="headerlink" href="#ecsoperator" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.operators.ecs_operator.ECSOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.ecs_operator.</code><code class="descname">ECSOperator</code><span class="sig-paren">(</span><em>task_definition</em>, <em>cluster</em>, <em>overrides</em>, <em>aws_conn_id=None</em>, <em>region_name=None</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/ecs_operator.html#ECSOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.ecs_operator.ECSOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Execute a task on AWS EC2 Container Service</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
+<li><strong>task_definition</strong> (<em>str</em>) – the task definition name on EC2 Container Service</li>
+<li><strong>cluster</strong> (<em>str</em>) – the cluster name on EC2 Container Service</li>
+<li><strong>aws_conn_id</strong> (<em>str</em>) – connection id of AWS credentials / region name. If None,
+credential boto3 strategy will be used (<a class="reference external" href="http://boto3.readthedocs.io/en/latest/guide/configuration.html">http://boto3.readthedocs.io/en/latest/guide/configuration.html</a>).</li>
+<li><strong>region_name</strong> – region name to use in AWS Hook. Override the region_name in connection (if provided)</li>
+</ul>
+</td>
+</tr>
+<tr class="field-even field"><th class="field-name">Param:</th><td class="field-body"><p class="first">overrides: the same parameter that boto3 will receive:
+<a class="reference external" href="http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task">http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task</a></p>
+</td>
+</tr>
+<tr class="field-odd field"><th class="field-name">Type:</th><td class="field-body"><p class="first last">overrides: dict</p>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+</div>
+<div class="section" id="aws-redshift">
+<h3>AWS RedShift<a class="headerlink" href="#aws-redshift" title="Permalink to this headline">¶</a></h3>
+<ul class="simple">
+<li><a class="reference internal" href="#redshifttos3transfer"><span class="std std-ref">RedshiftToS3Transfer</span></a> : Executes an unload command to S3 as a CSV with headers.</li>
+</ul>
+<div class="section" id="redshifttos3transfer">
+<span id="id13"></span><h4>RedshiftToS3Transfer<a class="headerlink" href="#redshifttos3transfer" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.redshift_to_s3_operator.</code><code class="descname">RedshiftToS3Transfer</code><span class="sig-paren">(</span><em>schema</em>, <em>table</em>, <em>s3_bucket</em>, <em>s3_key</em>, <em>redshift_conn_id='redshift_default'</em>, <em>aws_conn_id='aws_default'</em>, <em>unload_options=()</em>, <em>autocommit=False</em>, <em>parameters=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/redshift_to_s3_operator.html#RedshiftToS3Transfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Executes an UNLOAD command to s3 as a CSV with headers
+:param schema: reference to a specific schema in redshift database
+:type schema: string
+:param table: reference to a specific table in redshift database
+:type table: string
+:param s3_bucket: reference to a specific S3 bucket
+:type s3_bucket: string
+:param s3_key: reference to a specific S3 key
+:type s3_key: string
+:param redshift_conn_id: reference to a specific redshift database
+:type redshift_conn_id: string
+:param aws_conn_id: reference to a specific S3 connection
+:type aws_conn_id: string
+:param options: reference to a list of UNLOAD options
+:type options: list</p>
+</dd></dl>
+
+</div>
+</div>
+</div>
+<div class="section" id="databricks">
+<span id="id14"></span><h2>Databricks<a class="headerlink" href="#databricks" title="Permalink to this headline">¶</a></h2>
+<p><a class="reference external" href="https://databricks.com/">Databricks</a> has contributed an Airflow operator which enables
+submitting runs to the Databricks platform. Internally the operator talks to the
+<code class="docutils literal"><span class="pre">api/2.0/jobs/runs/submit</span></code> <a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#runs-submit">endpoint</a>.</p>
+<div class="section" id="databrickssubmitrunoperator">
+<h3>DatabricksSubmitRunOperator<a class="headerlink" href="#databrickssubmitrunoperator" title="Permalink to this headline">¶</a></h3>
+<dl class="class">
+<dt id="airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.databricks_operator.</code><code class="descname">DatabricksSubmitRunOperator</code><span class="sig-paren">(</span><em>json=None</em>, <em>spark_jar_task=None</em>, <em>notebook_task=None</em>, <em>new_cluster=None</em>, <em>existing_cluster_id=None</em>, <em>libraries=None</em>, <em>run_name=None</em>, <em>timeout_seconds=None</em>, <em>databricks_conn_id='databricks_default'</em>, <em>polling_period_seconds=30</em>, <em>databricks_retry_limit=3</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/databricks_operator.html#DatabricksSubmitRunOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Submits an Spark job run to Databricks using the
+<a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#runs-submit">api/2.0/jobs/runs/submit</a>
+API endpoint.</p>
+<p>There are two ways to instantiate this operator.</p>
+<p>In the first way, you can take the JSON payload that you typically use
+to call the <code class="docutils literal"><span class="pre">api/2.0/jobs/runs/submit</span></code> endpoint and pass it directly
+to our <code class="docutils literal"><span class="pre">DatabricksSubmitRunOperator</span></code> through the <code class="docutils literal"><span class="pre">json</span></code> parameter.
+For example</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">json</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'new_cluster'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'spark_version'</span><span class="p">:</span> <span class="s1">'2.1.0-db3-scala2.11'</span><span class="p">,</span>
+ <span class="s1">'num_workers'</span><span class="p">:</span> <span class="mi">2</span>
+ <span class="p">},</span>
+ <span class="s1">'notebook_task'</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s1">'notebook_path'</span><span class="p">:</span> <span class="s1">'/Users/airflow@example.com/PrepareData'</span><span class="p">,</span>
+ <span class="p">},</span>
+<span class="p">}</span>
+<span class="n">notebook_run</span> <span class="o">=</span> <span class="n">DatabricksSubmitRunOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'notebook_run'</span><span class="p">,</span> <span class="n">json</span><span class="o">=</span><span class="n">json</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Another way to accomplish the same thing is to use the named parameters
+of the <code class="docutils literal"><span class="pre">DatabricksSubmitRunOperator</span></code> directly. Note that there is exactly
+one named parameter for each top level parameter in the <code class="docutils literal"><span class="pre">runs/submit</span></code>
+endpoint. In this method, your code would look like this:</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">new_cluster</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'spark_version'</span><span class="p">:</span> <span class="s1">'2.1.0-db3-scala2.11'</span><span class="p">,</span>
+ <span class="s1">'num_workers'</span><span class="p">:</span> <span class="mi">2</span>
+<span class="p">}</span>
+<span class="n">notebook_task</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'notebook_path'</span><span class="p">:</span> <span class="s1">'/Users/airflow@example.com/PrepareData'</span><span class="p">,</span>
+<span class="p">}</span>
+<span class="n">notebook_run</span> <span class="o">=</span> <span class="n">DatabricksSubmitRunOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'notebook_run'</span><span class="p">,</span>
+ <span class="n">new_cluster</span><span class="o">=</span><span class="n">new_cluster</span><span class="p">,</span>
+ <span class="n">notebook_task</span><span class="o">=</span><span class="n">notebook_task</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>In the case where both the json parameter <strong>AND</strong> the named parameters
+are provided, they will be merged together. If there are conflicts during the merge,
+the named parameters will take precedence and override the top level <code class="docutils literal"><span class="pre">json</span></code> keys.</p>
+<dl class="docutils">
+<dt>Currently the named parameters that <code class="docutils literal"><span class="pre">DatabricksSubmitRunOperator</span></code> supports are</dt>
+<dd><ul class="first last simple">
+<li><code class="docutils literal"><span class="pre">spark_jar_task</span></code></li>
+<li><code class="docutils literal"><span class="pre">notebook_task</span></code></li>
+<li><code class="docutils literal"><span class="pre">new_cluster</span></code></li>
+<li><code class="docutils literal"><span class="pre">existing_cluster_id</span></code></li>
+<li><code class="docutils literal"><span class="pre">libraries</span></code></li>
+<li><code class="docutils literal"><span class="pre">run_name</span></code></li>
+<li><code class="docutils literal"><span class="pre">timeout_seconds</span></code></li>
+</ul>
+</dd>
+</dl>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>json</strong> (<em>dict</em>) – <p>A JSON object containing API parameters which will be passed
+directly to the <code class="docutils literal"><span class="pre">api/2.0/jobs/runs/submit</span></code> endpoint. The other named parameters
+(i.e. <code class="docutils literal"><span class="pre">spark_jar_task</span></code>, <code class="docutils literal"><span class="pre">notebook_task</span></code>..) to this operator will
+be merged with this json dictionary if they are provided.
+If there are conflicts during the merge, the named parameters will
+take precedence and override the top level json keys. This field will be
+templated.</p>
+<div class="admonition seealso">
+<p class="first admonition-title">See also</p>
+<p class="last">For more information about templating see <a class="reference internal" href="concepts.html#jinja-templating"><span class="std std-ref">Jinja Templating</span></a>.
+<a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#runs-submit">https://docs.databricks.com/api/latest/jobs.html#runs-submit</a></p>
+</div>
+</li>
+<li><strong>spark_jar_task</strong> (<em>dict</em>) – <p>The main class and parameters for the JAR task. Note that
+the actual JAR is specified in the <code class="docutils literal"><span class="pre">libraries</span></code>.
+<em>EITHER</em> <code class="docutils literal"><span class="pre">spark_jar_task</span></code> <em>OR</em> <code class="docutils literal"><span class="pre">notebook_task</span></code> should be specified.
+This field will be templated.</p>
+<div class="admonition seealso">
+<p class="first admonition-title">See also</p>
+<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#jobssparkjartask">https://docs.databricks.com/api/latest/jobs.html#jobssparkjartask</a></p>
+</div>
+</li>
+<li><strong>notebook_task</strong> (<em>dict</em>) – <p>The notebook path and parameters for the notebook task.
+<em>EITHER</em> <code class="docutils literal"><span class="pre">spark_jar_task</span></code> <em>OR</em> <code class="docutils literal"><span class="pre">notebook_task</span></code> should be specified.
+This field will be templated.</p>
+<div class="admonition seealso">
+<p class="first admonition-title">See also</p>
+<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#jobsnotebooktask">https://docs.databricks.com/api/latest/jobs.html#jobsnotebooktask</a></p>
+</div>
+</li>
+<li><strong>new_cluster</strong> (<em>dict</em>) – <p>Specs for a new cluster on which this task will be run.
+<em>EITHER</em> <code class="docutils literal"><span class="pre">new_cluster</span></code> <em>OR</em> <code class="docutils literal"><span class="pre">existing_cluster_id</span></code> should be specified.
+This field will be templated.</p>
+<div class="admonition seealso">
+<p class="first admonition-title">See also</p>
+<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/jobs.html#jobsclusterspecnewcluster">https://docs.databricks.com/api/latest/jobs.html#jobsclusterspecnewcluster</a></p>
+</div>
+</li>
+<li><strong>existing_cluster_id</strong> (<em>string</em>) – ID for existing cluster on which to run this task.
+<em>EITHER</em> <code class="docutils literal"><span class="pre">new_cluster</span></code> <em>OR</em> <code class="docutils literal"><span class="pre">existing_cluster_id</span></code> should be specified.
+This field will be templated.</li>
+<li><strong>libraries</strong> (<em>list of dicts</em>) – <p>Libraries which this run will use.
+This field will be templated.</p>
+<div class="admonition seealso">
+<p class="first admonition-title">See also</p>
+<p class="last"><a class="reference external" href="https://docs.databricks.com/api/latest/libraries.html#managedlibrarieslibrary">https://docs.databricks.com/api/latest/libraries.html#managedlibrarieslibrary</a></p>
+</div>
+</li>
+<li><strong>run_name</strong> (<em>string</em>) – The run name used for this task.
+By default this will be set to the Airflow <code class="docutils literal"><span class="pre">task_id</span></code>. This <code class="docutils literal"><span class="pre">task_id</span></code> is a
+required parameter of the superclass <code class="docutils literal"><span class="pre">BaseOperator</span></code>.
+This field will be templated.</li>
+<li><strong>timeout_seconds</strong> (<em>int32</em>) – The timeout for this run. By default a value of 0 is used
+which means to have no timeout.
+This field will be templated.</li>
+<li><strong>databricks_conn_id</strong> (<em>string</em>) – The name of the Airflow connection to use.
+By default and in the common case this will be <code class="docutils literal"><span class="pre">databricks_default</span></code>. To use
+token based authentication, provide the key <code class="docutils literal"><span class="pre">token</span></code> in the extra field for the
+connection.</li>
+<li><strong>polling_period_seconds</strong> (<em>int</em>) – Controls the rate which we poll for the result of
+this run. By default the operator will poll every 30 seconds.</li>
+<li><strong>databricks_retry_limit</strong> (<em>int</em>) – Amount of times retry if the Databricks backend is
+unreachable. Its value must be greater than or equal to 1.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
</div>
<div class="section" id="gcp-google-cloud-platform">
<span id="gcp"></span><h2>GCP: Google Cloud Platform<a class="headerlink" href="#gcp-google-cloud-platform" title="Permalink to this headline">¶</a></h2>
<p>Airflow has extensive support for the Google Cloud Platform. But note that most Hooks and
Operators are in the contrib section. Meaning that they have a <em>beta</em> status, meaning that
they can have breaking changes between minor releases.</p>
+<div class="section" id="logging">
+<h3>Logging<a class="headerlink" href="#logging" title="Permalink to this headline">¶</a></h3>
+<p>Airflow can be configured to read and write task logs in Google cloud storage.
+Follow the steps below to enable Google cloud storage logging.</p>
+<ol class="arabic">
+<li><p class="first">Airflow’s logging system requires a custom .py file to be located in the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code>, so that it’s importable from Airflow. Start by creating a directory to store the config file. <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/config</span></code> is recommended.</p>
+</li>
+<li><p class="first">Create empty files called <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/config/log_config.py</span></code> and <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/config/__init__.py</span></code>.</p>
+</li>
+<li><p class="first">Copy the contents of <code class="docutils literal"><span class="pre">airflow/config_templates/airflow_local_settings.py</span></code> into the <code class="docutils literal"><span class="pre">log_config.py</span></code> file that was just created in the step above.</p>
+</li>
+<li><p class="first">Customize the following portions of the template:</p>
+<blockquote>
+<div><div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Add this variable to the top of the file. Note the trailing slash.</span>
+<span class="nv">GCS_LOG_FOLDER</span> <span class="o">=</span> <span class="s1">'gs://<bucket where logs should be persisted>/'</span>
+
+<span class="c1"># Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG</span>
+<span class="nv">LOGGING_CONFIG</span> <span class="o">=</span> ...
+
+<span class="c1"># Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG variable</span>
+<span class="s1">'gcs.task'</span>: <span class="o">{</span>
+ <span class="s1">'class'</span>: <span class="s1">'airflow.utils.log.gcs_task_handler.GCSTaskHandler'</span>,
+ <span class="s1">'formatter'</span>: <span class="s1">'airflow.task'</span>,
+ <span class="s1">'base_log_folder'</span>: os.path.expanduser<span class="o">(</span>BASE_LOG_FOLDER<span class="o">)</span>,
+ <span class="s1">'gcs_log_folder'</span>: GCS_LOG_FOLDER,
+ <span class="s1">'filename_template'</span>: FILENAME_TEMPLATE,
+<span class="o">}</span>,
+
+<span class="c1"># Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of 'file.task'.</span>
+<span class="s1">'loggers'</span>: <span class="o">{</span>
+ <span class="s1">'airflow.task'</span>: <span class="o">{</span>
+ <span class="s1">'handlers'</span>: <span class="o">[</span><span class="s1">'gcs.task'</span><span class="o">]</span>,
+ ...
+ <span class="o">}</span>,
+ <span class="s1">'airflow.task_runner'</span>: <span class="o">{</span>
+ <span class="s1">'handlers'</span>: <span class="o">[</span><span class="s1">'gcs.task'</span><span class="o">]</span>,
+ ...
+ <span class="o">}</span>,
+ <span class="s1">'airflow'</span>: <span class="o">{</span>
+ <span class="s1">'handlers'</span>: <span class="o">[</span><span class="s1">'console'</span><span class="o">]</span>,
+ ...
+ <span class="o">}</span>,
+<span class="o">}</span>
+</pre></div>
+</div>
+</div></blockquote>
+</li>
+<li><p class="first">Make sure a Google cloud platform connection hook has been defined in Airflow. The hook should have read and write access to the Google cloud storage bucket defined above in <code class="docutils literal"><span class="pre">GCS_LOG_FOLDER</span></code>.</p>
+</li>
+<li><p class="first">Update <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code> to contain:</p>
+<blockquote>
+<div><div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">task_log_reader</span> <span class="o">=</span> gcs.task
+<span class="nv">logging_config_class</span> <span class="o">=</span> log_config.LOGGING_CONFIG
+<span class="nv">remote_log_conn_id</span> <span class="o">=</span> <name of the Google cloud platform hook>
+</pre></div>
+</div>
+</div></blockquote>
+</li>
+<li><p class="first">Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.</p>
+</li>
+<li><p class="first">Verify that logs are showing up for newly executed tasks in the bucket you’ve defined.</p>
+</li>
+<li><p class="first">Verify that the Google cloud storage viewer is working in the UI. Pull up a newly executed task, and verify that you see something like:</p>
+<blockquote>
+<div><div class="highlight-bash"><div class="highlight"><pre><span></span>*** Reading remote log from gs://<bucket where logs should be persisted>/example_bash_operator/run_this_last/2017-10-03T00:00:00/16.log.
+<span class="o">[</span><span class="m">2017</span>-10-03 <span class="m">21</span>:57:50,056<span class="o">]</span> <span class="o">{</span>cli.py:377<span class="o">}</span> INFO - Running on host chrisr-00532
+<span class="o">[</span><span class="m">2017</span>-10-03 <span class="m">21</span>:57:50,093<span class="o">]</span> <span class="o">{</span>base_task_runner.py:115<span class="o">}</span> INFO - Running: <span class="o">[</span><span class="s1">'bash'</span>, <span class="s1">'-c'</span>, u<span class="s1">'airflow run example_bash_operator run_this_last 2017-10-03T00:00:00 --job_id 47 --raw -sd DAGS_FOLDER/example_dags/example_bash_operator.py'</span><span class="o">]</span>
+<span class="o">[</span><span class="m">2017</span>-10-03 <span class="m">21</span>:57:51,264<span class="o">]</span> <span class="o">{</span>base_task_runner.py:98<span class="o">}</span> INFO - Subtask: <span class="o">[</span><span class="m">2017</span>-10-03 <span class="m">21</span>:57:51,263<span class="o">]</span> <span class="o">{</span>__init__.py:45<span class="o">}</span> INFO - Using executor SequentialExecutor
+<span class="o">[</span><span class="m">2017</span>-10-03 <span class="m">21</span>:57:51,306<span class="o">]</span> <span class="o">{</span>base_task_runner.py:98<span class="o">}</span> INFO - Subtask: <span class="o">[</span><span class="m">2017</span>-10-03 <span class="m">21</span>:57:51,306<span class="o">]</span> <span class="o">{</span>models.py:186<span class="o">}</span> INFO - Filling up the DagBag from /airflow/dags/example_dags/example_bash_operator.py
+</pre></div>
+</div>
+</div></blockquote>
+</li>
+</ol>
+<p>Note the top line that says it’s reading from the remote log file.</p>
+<p>Please be aware that if you were persisting logs to Google cloud storage using the old-style airflow.cfg configuration method, the old logs will no longer be visible in the Airflow UI, though they’ll still exist in Google cloud storage. This is a backwards incompatbile change. If you are unhappy with it, you can change the <code class="docutils literal"><span class="pre">FILENAME_TEMPLATE</span></code> to reflect the old-style log filename format.</p>
+</div>
<div class="section" id="bigquery">
<h3>BigQuery<a class="headerlink" href="#bigquery" title="Permalink to this headline">¶</a></h3>
<div class="section" id="bigquery-operators">
@@ -220,26 +850,287 @@ they can have breaking changes between minor releases.</p>
<li><a class="reference internal" href="#bigquerytocloudstorageoperator"><span class="std std-ref">BigQueryToCloudStorageOperator</span></a> : Transfers a BigQuery table to a Google Cloud Storage bucket</li>
</ul>
<div class="section" id="bigquerycheckoperator">
-<span id="id1"></span><h5>BigQueryCheckOperator<a class="headerlink" href="#bigquerycheckoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id16"></span><h5>BigQueryCheckOperator<a class="headerlink" href="#bigquerycheckoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.bigquery_check_operator.</code><code class="descname">BigQueryCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>bigquery_conn_id='bigquery_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/bigquery_check_operator.html#BigQueryCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Performs checks against BigQuery. The <code class="docutils literal"><span class="pre">BigQueryCheckOperator</span></code> expects
+a sql query that will return a single row. Each value on that
+first row is evaluated using python <code class="docutils literal"><span class="pre">bool</span></code> casting. If any of the
+values return <code class="docutils literal"><span class="pre">False</span></code> the check is failed and errors out.</p>
+<p>Note that Python bool casting evals the following as <code class="docutils literal"><span class="pre">False</span></code>:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">False</span></code></li>
+<li><code class="docutils literal"><span class="pre">0</span></code></li>
+<li>Empty string (<code class="docutils literal"><span class="pre">""</span></code>)</li>
+<li>Empty list (<code class="docutils literal"><span class="pre">[]</span></code>)</li>
+<li>Empty dictionary or set (<code class="docutils literal"><span class="pre">{}</span></code>)</li>
+</ul>
+<p>Given a query like <code class="docutils literal"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
+the count <code class="docutils literal"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
+for instance, check that the table has the same number of rows as
+the source table upstream, or that the count of today’s partition is
+greater than yesterday’s partition, or that a set of metrics are less
+than 3 standard deviation for the 7 day average.</p>
+<p>This operator can be used as a data quality check in your pipeline, and
+depending on where you put it in your DAG, you have the choice to
+stop the critical path, preventing from
+publishing dubious data, or on the side and receive email alterts
+without stopping the progress of the DAG.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
+<li><strong>bigquery_conn_id</strong> (<em>string</em>) – reference to the BigQuery database</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
</div>
<div class="section" id="bigqueryvaluecheckoperator">
-<span id="id2"></span><h5>BigQueryValueCheckOperator<a class="headerlink" href="#bigqueryvaluecheckoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id17"></span><h5>BigQueryValueCheckOperator<a class="headerlink" href="#bigqueryvaluecheckoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.bigquery_check_operator.</code><code class="descname">BigQueryValueCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>pass_value</em>, <em>tolerance=None</em>, <em>bigquery_conn_id='bigquery_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/bigquery_check_operator.html#BigQueryValueCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Performs a simple value check using sql code.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>sql</strong> (<em>string</em>) – the sql to be executed</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
</div>
<div class="section" id="bigqueryintervalcheckoperator">
-<span id="id3"></span><h5>BigQueryIntervalCheckOperator<a class="headerlink" href="#bigqueryintervalcheckoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id18"></span><h5>BigQueryIntervalCheckOperator<a class="headerlink" href="#bigqueryintervalcheckoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.bigquery_check_operator.</code><code class="descname">BigQueryIntervalCheckOperator</code><span class="sig-paren">(</span><em>table</em>, <em>metrics_thresholds</em>, <em>date_filter_column='ds'</em>, <em>days_back=-7</em>, <em>bigquery_conn_id='bigquery_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/bigquery_check_operator.html#BigQueryIntervalCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Checks that the values of metrics given as SQL expressions are within
+a certain tolerance of the ones from days_back before.</p>
+<p>This method constructs a query like so:</p>
+<dl class="docutils">
+<dt>SELECT {metrics_threshold_dict_key} FROM {table}</dt>
+<dd>WHERE {date_filter_column}=<date></dd>
+</dl>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>table</strong> (<em>str</em>) – the table name</li>
+<li><strong>days_back</strong> (<em>int</em>) – number of days between ds and the ds we want to check
+against. Defaults to 7 days</li>
+<li><strong>metrics_threshold</strong> (<em>dict</em>) – a dictionary of ratios indexed by metrics, for
+example ‘COUNT(*)’: 1.5 would require a 50 percent or less difference
+between the current day, and the prior days_back.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
</div>
<div class="section" id="bigqueryoperator">
-<span id="id4"></span><h5>BigQueryOperator<a class="headerlink" href="#bigqueryoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id19"></span><h5>BigQueryOperator<a class="headerlink" href="#bigqueryoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.bigquery_operator.BigQueryOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.bigquery_operator.</code><code class="descname">BigQueryOperator</code><span class="sig-paren">(</span><em>bql</em>, <em>destination_dataset_table=False</em>, <em>write_disposition='WRITE_EMPTY'</em>, <em>allow_large_results=False</em>, <em>bigquery_conn_id='bigquery_default'</em>, <em>delegate_to=None</em>, <em>udf_config=False</em>, <em>use_legacy_sql=True</em>, <em>maximum_billing_tier=None</em>, <em>create_disposition='CREATE_IF_NEEDED'</em>, <em>query_params=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/bigquery_operator.html#BigQueryOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.bigquery_operator.BigQueryOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Executes BigQuery SQL queries in a specific BigQuery database</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>bql</strong> (<em>Can receive a str representing a sql statement</em><em>,
+</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+<li><strong>destination_dataset_table</strong> (<em>string</em>) – A dotted
+(<project>.|<project>:)<dataset>.<table> that, if set, will store the results
+of the query.</li>
+<li><strong>write_disposition</strong> (<em>string</em>) – Specifies the action that occurs if the destination table
+already exists. (default: ‘WRITE_EMPTY’)</li>
+<li><strong>create_disposition</strong> (<em>string</em>) – Specifies whether the job is allowed to create new tables.
+(default: ‘CREATE_IF_NEEDED’)</li>
+<li><strong>bigquery_conn_id</strong> (<em>string</em>) – reference to a specific BigQuery hook.</li>
+<li><strong>delegate_to</strong> (<em>string</em>) – The account to impersonate, if any.
+For this to work, the service account making the request must have domain-wide
+delegation enabled.</li>
+<li><strong>udf_config</strong> (<a class="reference internal" href="#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – The User Defined Function configuration for the query.
+See <a class="reference external" href="https://cloud.google.com/bigquery/user-defined-functions">https://cloud.google.com/bigquery/user-defined-functions</a> for details.</li>
+<li><strong>use_legacy_sql</strong> (<em>boolean</em>) – Whether to use legacy SQL (true) or standard SQL (false).</li>
+<li><strong>maximum_billing_tier</strong> (<em>integer</em>) – Positive integer that serves as a multiplier of the basic price.
+Defaults to None, in which case it uses the value set in the project.</li>
+<li><strong>query_params</strong> (<em>dict</em>) – a dictionary containing query parameter types and values, passed to
+BigQuery.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
</div>
<div class="section" id="bigquerytobigqueryoperator">
-<span id="id5"></span><h5>BigQueryToBigQueryOperator<a class="headerlink" href="#bigquerytobigqueryoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id20"></span><h5>BigQueryToBigQueryOperator<a class="headerlink" href="#bigquerytobigqueryoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.bigquery_to_bigquery.</code><code class="descname">BigQueryToBigQueryOperator</code><span class="sig-paren">(</span><em>source_project_dataset_tables</em>, <em>destination_project_dataset_table</em>, <em>write_disposition='WRITE_EMPTY'</em>, <em>create_disposition='CREATE_IF_NEEDED'</em>, <em>bigquery_conn_id='bigquery_default'</em>, <em>delegate_to=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/bigquery_to_bigquery.html#BigQueryToBigQueryOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Copies data from one BigQuery table to another. See here:</p>
+<p><a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy">https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy</a></p>
+<p>For more details about these parameters.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>source_project_dataset_tables</strong> (<em>list|string</em>) – One or more
+dotted (project:<a href="#id21"><span class="problematic" id="id22">|</span></a>project.)<dataset>.<table> BigQuery tables to use as the
+source data. If <project> is not included, project will be the project defined
+in the connection json. Use a list if there are multiple source tables.</li>
+<li><strong>destination_project_dataset_table</strong> (<em>string</em>) – The destination BigQuery
+table. Format is: (project:<a href="#id23"><span class="problematic" id="id24">|</span></a>project.)<dataset>.<table></li>
+<li><strong>write_disposition</strong> (<em>string</em>) – The write disposition if the table already exists.</li>
+<li><strong>create_disposition</strong> (<em>string</em>) – The create disposition if the table doesn’t exist.</li>
+<li><strong>bigquery_conn_id</strong> (<em>string</em>) – reference to a specific BigQuery hook.</li>
+<li><strong>delegate_to</strong> (<em>string</em>) – The account to impersonate, if any.
+For this to work, the service account making the request must have domain-wide
+delegation enabled.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
</div>
<div class="section" id="bigquerytocloudstorageoperator">
-<span id="id6"></span><h5>BigQueryToCloudStorageOperator<a class="headerlink" href="#bigquerytocloudstorageoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id25"></span><h5>BigQueryToCloudStorageOperator<a class="headerlink" href="#bigquerytocloudstorageoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.bigquery_to_gcs.</code><code class="descname">BigQueryToCloudStorageOperator</code><span class="sig-paren">(</span><em>source_project_dataset_table</em>, <em>destination_cloud_storage_uris</em>, <em>compression='NONE'</em>, <em>export_format='CSV'</em>, <em>field_delimiter='</em>, <em>'</em>, <em>print_header=True</em>, <em>bigquery_conn_id='bigquery_default'</em>, <em>delegate_to=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/bigquery_to_gcs.html#BigQueryToCloudStorageOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Transfers a BigQuery table to a Google Cloud Storage bucket.</p>
+<p>See here:</p>
+<p><a class="reference external" href="https://cloud.google.com/bigquery/docs/reference/v2/jobs">https://cloud.google.com/bigquery/docs/reference/v2/jobs</a></p>
+<p>For more details about these parameters.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>source_project_dataset_table</strong> (<em>string</em>) – The dotted
+(<project>.|<project>:)<dataset>.<table> BigQuery table to use as the source
+data. If <project> is not included, project will be the project defined in
+the connection json.</li>
+<li><strong>destination_cloud_storage_uris</strong> (<a class="reference internal" href="#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – The destination Google Cloud
+Storage URI (e.g. gs://some-bucket/some-file.txt). Follows
+convention defined here:
+https://cloud.google.com/bigquery/exporting-data-from-bigquery#exportingmultiple</li>
+<li><strong>compression</strong> (<em>string</em>) – Type of compression to use.</li>
+<li><strong>export_format</strong> – File format to export.</li>
+<li><strong>field_delimiter</strong> (<em>string</em>) – The delimiter to use when extracting to a CSV.</li>
+<li><strong>print_header</strong> (<em>boolean</em>) – Whether to print a header for a CSV file extract.</li>
+<li><strong>bigquery_conn_id</strong> (<em>string</em>) – reference to a specific BigQuery hook.</li>
+<li><strong>delegate_to</strong> (<em>string</em>) – The account to impersonate, if any.
+For this to work, the service account making the request must have domain-wide
+delegation enabled.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
</div>
</div>
<div class="section" id="bigqueryhook">
<h4>BigQueryHook<a class="headerlink" href="#bigqueryhook" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.hooks.bigquery_hook.BigQueryHook">
+<em class="property">class </em><code class="descclassname">airflow.contrib.hooks.bigquery_hook.</code><code class="descname">BigQueryHook</code><span class="sig-paren">(</span><em>bigquery_conn_id='bigquery_default'</em>, <em>delegate_to=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/bigquery_hook.html#BigQueryHook"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.bigquery_hook.BigQueryHook" title="Permalink to this definition">¶</a></dt>
+<dd><p>Interact with BigQuery. This hook uses the Google Cloud Platform
+connection.</p>
+<dl class="method">
+<dt id="airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_conn">
+<code class="descname">get_conn</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/bigquery_hook.html#BigQueryHook.get_conn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_conn" title="Permalink to this definition">¶</a></dt>
+<dd><p>Returns a BigQuery PEP 249 connection object.</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_pandas_df">
+<code class="descname">get_pandas_df</code><span class="sig-paren">(</span><em>bql</em>, <em>parameters=None</em>, <em>dialect='legacy'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/bigquery_hook.html#BigQueryHook.get_pandas_df"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_pandas_df" title="Permalink to this definition">¶</a></dt>
+<dd><p>Returns a Pandas DataFrame for the results produced by a BigQuery
+query. The DbApiHook method must be overridden because Pandas
+doesn’t support PEP 249 connections, except for SQLite. See:</p>
+<p><a class="reference external" href="https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447">https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447</a>
+<a class="reference external" href="https://github.com/pydata/pandas/issues/6900">https://github.com/pydata/pandas/issues/6900</a></p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>bql</strong> (<em>string</em>) – The BigQuery SQL to execute.</li>
+<li><strong>parameters</strong> (<em>mapping</em><em> or </em><em>iterable</em>) – The parameters to render the SQL query with (not used, leave to override superclass method)</li>
+<li><strong>dialect</strong> (<em>string in {'legacy'</em><em>, </em><em>'standard'}</em><em>, </em><em>default 'legacy'</em>) – Dialect of BigQuery SQL – legacy SQL or standard SQL</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_service">
+<code class="descname">get_service</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/bigquery_hook.html#BigQueryHook.get_service"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.bigquery_hook.BigQueryHook.get_service" title="Permalink to this definition">¶</a></dt>
+<dd><p>Returns a BigQuery service object.</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.contrib.hooks.bigquery_hook.BigQueryHook.insert_rows">
+<code class="descname">insert_rows</code><span class="sig-paren">(</span><em>table</em>, <em>rows</em>, <em>target_fields=None</em>, <em>commit_every=1000</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/bigquery_hook.html#BigQueryHook.insert_rows"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.bigquery_hook.BigQueryHook.insert_rows" title="Permalink to this definition">¶</a></dt>
+<dd><p>Insertion is currently unsupported. Theoretically, you could use
+BigQuery’s streaming API to insert rows into a table, but this hasn’t
+been implemented.</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.contrib.hooks.bigquery_hook.BigQueryHook.table_exists">
+<code class="descname">table_exists</code><span class="sig-paren">(</span><em>project_id</em>, <em>dataset_id</em>, <em>table_id</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/bigquery_hook.html#BigQueryHook.table_exists"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.bigquery_hook.BigQueryHook.table_exists" title="Permalink to this definition">¶</a></dt>
+<dd><p>Checks for the existence of a table in Google BigQuery.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>project_id</strong> – The Google cloud project in which to look for the table. The connection supplied to the hook</td>
+</tr>
+</tbody>
+</table>
+<p>must provide access to the specified project.
+:type project_id: string
+:param dataset_id: The name of the dataset in which to look for the table.</p>
+<blockquote>
+<div>storage bucket.</div></blockquote>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>table_id</strong> (<em>string</em>) – The name of the table to check the existence of.</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</dd></dl>
+
</div>
</div>
<div class="section" id="cloud-dataflow">
@@ -247,10 +1138,50 @@ they can have breaking changes between minor releases.</p>
<div class="section" id="dataflow-operators">
<h4>DataFlow Operators<a class="headerlink" href="#dataflow-operators" title="Permalink to this headline">¶</a></h4>
<ul class="simple">
-<li><a class="reference internal" href="#dataflowjavaoperator"><span class="std std-ref">DataFlowJavaOperator</span></a> :</li>
+<li><a class="reference internal" href="#dataflowjavaoperator"><span class="std std-ref">DataFlowJavaOperator</span></a> : launching Cloud Dataflow jobs written in Java.</li>
+<li><a class="reference internal" href="#dataflowpythonoperator"><span class="std std-ref">DataFlowPythonOperator</span></a> : launching Cloud Dataflow jobs written in python.</li>
</ul>
<div class="section" id="dataflowjavaoperator">
-<span id="id7"></span><h5>DataFlowJavaOperator<a class="headerlink" href="#dataflowjavaoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id26"></span><h5>DataFlowJavaOperator<a class="headerlink" href="#dataflowjavaoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.dataflow_operator.</code><code class="descname">DataFlowJavaOperator</code><span class="sig-paren">(</span><em>jar</em>, <em>dataflow_default_options=None</em>, <em>options=None</em>, <em>gcp_conn_id='google_cloud_default'</em>, <em>delegate_to=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/dataflow_operator.html#DataFlowJavaOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Start a Java Cloud DataFlow batch job. The parameters of the operation
+will be passed to the job.</p>
+<p>It’s a good practice to define dataflow_* parameters in the default_args of the dag
+like the project, zone and staging location.</p>
+<p><a href="#id27"><span class="problematic" id="id28">``</span></a>`
+default_args = {</p>
+<blockquote>
+<div><dl class="docutils">
+<dt>‘dataflow_default_options’: {</dt>
+<dd>‘project’: ‘my-gcp-project’,
+‘zone’: ‘europe-west1-d’,
+‘stagingLocation’: ‘gs://my-staging-bucket/staging/’</dd>
+</dl>
+<p>}</p>
+</div></blockquote>
+<p>You need to pass the path to your dataflow as a file reference with the <code class="docutils literal"><span class="pre">jar</span></code>
+parameter, the jar needs to be a self executing jar. Use <code class="docutils literal"><span class="pre">options</span></code> to pass on
+options to your job.</p>
+<p><a href="#id29"><span class="problematic" id="id30">``</span></a>`
+t1 = DataFlowOperation(</p>
+<blockquote>
+<div><p>task_id=’datapflow_example’,
+jar=’{{var.value.gcp_dataflow_base}}pipeline/build/libs/pipeline-example-1.0.jar’,
+options={</p>
+<blockquote>
+<div>‘autoscalingAlgorithm’: ‘BASIC’,
+‘maxNumWorkers’: ‘50’,
+‘start’: ‘{{ds}}’,
+‘partitionType’: ‘DAY’</div></blockquote>
+<p>},
+dag=my-dag)</p>
+</div></blockquote>
+<p><a href="#id31"><span class="problematic" id="id32">``</span></a><a href="#id33"><span class="problematic" id="id34">`</span></a></p>
+<p>Both <code class="docutils literal"><span class="pre">jar</span></code> and <code class="docutils literal"><span class="pre">options</span></code> are templated so you can use variables in them.</p>
+</dd></dl>
+
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
<span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
@@ -285,9 +1216,28 @@ they can have breaking changes between minor releases.</p>
</pre></div>
</div>
</div>
+<div class="section" id="dataflowpythonoperator">
+<span id="id35"></span><h5>DataFlowPythonOperator<a class="headerlink" href="#dataflowpythonoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.dataflow_operator.</code><code class="descname">DataFlowPythonOperator</code><span class="sig-paren">(</span><em>py_file</em>, <em>py_options=None</em>, <em>dataflow_default_options=None</em>, <em>options=None</em>, <em>gcp_conn_id='google_cloud_default'</em>, <em>delegate_to=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/dataflow_operator.html#DataFlowPythonOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+
+</div>
</div>
<div class="section" id="dataflowhook">
<h4>DataFlowHook<a class="headerlink" href="#dataflowhook" title="Permalink to this headline">¶</a></h4>
+<dl class="class">
+<dt id="airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook">
+<em class="property">class </em><code class="descclassname">airflow.contrib.hooks.gcp_dataflow_hook.</code><code class="descname">DataFlowHook</code><span class="sig-paren">(</span><em>gcp_conn_id='google_cloud_default'</em>, <em>delegate_to=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/gcp_dataflow_hook.html#DataFlowHook"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook" title="Permalink to this definition">¶</a></dt>
+<dd><dl class="method">
+<dt id="airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook.get_conn">
+<code class="descname">get_conn</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/hooks/gcp_dataflow_hook.html#DataFlowHook.get_conn"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook.get_conn" title="Permalink to this definition">¶</a></dt>
+<dd><p>Returns a Google Cloud Storage service object.</p>
+</dd></dl>
+
+</dd></dl>
+
</div>
</div>
<div class="section" id="cloud-dataproc">
@@ -303,29 +1253,566 @@ they can have breaking changes between minor releases.</p>
<li><a class="reference internal" href="#dataprocpysparkoperator"><span class="std std-ref">DataProcPySparkOperator</span></a> : Start a PySpark Job on a Cloud DataProc cluster.</li>
</ul>
<div class="section" id="dataprocpigoperator">
-<span id="id8"></span><h5>DataProcPigOperator<a class="headerlink" href="#dataprocpigoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id36"></span><h5>DataProcPigOperator<a class="headerlink" href="#dataprocpigoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.dataproc_operator.DataProcPigOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.dataproc_operator.</code><code class="descname">DataProcPigOperator</code><span class="sig-paren">(</span><em>query=None</em>, <em>query_uri=None</em>, <em>variables=None</em>, <em>job_name='{{task.task_id}}_{{ds_nodash}}'</em>, <em>cluster_name='cluster-1'</em>, <em>dataproc_pig_properties=None</em>, <em>dataproc_pig_jars=None</em>, <em>gcp_conn_id='google_cloud_default'</em>, <em>delegate_to=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/dataproc_operator.html#DataProcPigOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.dataproc_operator.DataProcPigOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Start a Pig query Job on a Cloud DataProc cluster. The parameters of the operation
+will be passed to the cluster.</p>
+<p>It’s a good practice to define dataproc_* parameters in the default_args of the dag
+like the cluster name and UDFs.</p>
+<p><a href="#id37"><span class="problematic" id="id38">``</span></a>`
+default_args = {</p>
+<blockquote>
+<div><p>‘cluster_name’: ‘cluster-1’,
+‘dataproc_pig_jars’: [</p>
+<blockquote>
+<div>‘gs://example/udf/jar/datafu/1.2.0/datafu.jar’,
+‘gs://example/udf/jar/gpig/1.2/gpig.jar’</div></blockquote>
+<p>]</p>
+</div></blockquote>
+<p>You can pass a pig script as string or file reference. Use variables to pass on
+variables for the pig script to be resolved on the cluster or use the parameters to
+be resolved in the script as template parameters.</p>
+<p><a href="#id39"><span class="problematic" id="id40">``</span></a>`
+t1 = DataProcPigOperator(</p>
+<blockquote>
+<div>task_id=’dataproc_pig’,
+query=’a_pig_script.pig’,
+variables={‘out’: ‘gs://example/output/{{ds}}’},</div></blockquote>
+<p>dag=dag)
+<a href="#id41"><span class="problematic" id="id42">``</span></a><a href="#id43"><span class="problematic" id="id44">`</span></a></p>
+</dd></dl>
+
</div>
<div class="section" id="dataprochiveoperator">
-<span id="id9"></span><h5>DataProcHiveOperator<a class="headerlink" href="#dataprochiveoperator" title="Permalink to this headline">¶</a></h5>
+<span id="id45"></span><h5>DataProcHiveOperator<a class="headerlink" href="#dataprochiveoperator" title="Permalink to this headline">¶</a></h5>
+<dl class="class">
+<dt id="airflow.contrib.operators.dataproc_operator.DataProcHiveOperator">
+<em class="property">class </em><code class="descclassname">airflow.contrib.operators.dataproc_operator.</code><code class="descname">DataProcHiveOperator</code><span class="sig-paren">(</span><em>query=None</em>, <em>query_uri=None</em>, <em>
<TRUNCATED>
[31/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/gcp_mlengine_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/gcp_mlengine_hook.html b/_modules/airflow/contrib/hooks/gcp_mlengine_hook.html
new file mode 100644
index 0000000..82450ae
--- /dev/null
+++ b/_modules/airflow/contrib/hooks/gcp_mlengine_hook.html
@@ -0,0 +1,500 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.hooks.gcp_mlengine_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.hooks.gcp_mlengine_hook</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.hooks.gcp_mlengine_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1">#</span>
+<span class="c1"># Licensed to the Apache Software Foundation (ASF) under one or more</span>
+<span class="c1"># contributor license agreements. See the NOTICE file distributed with</span>
+<span class="c1"># this work for additional information regarding copyright ownership.</span>
+<span class="c1"># The ASF licenses this file to You under the Apache License, Version 2.0</span>
+<span class="c1"># (the "License"); you may not use this file except in compliance with</span>
+<span class="c1"># the License. You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">import</span> <span class="nn">random</span>
+<span class="kn">import</span> <span class="nn">time</span>
+<span class="kn">from</span> <span class="nn">apiclient</span> <span class="k">import</span> <span class="n">errors</span>
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span>
+<span class="kn">from</span> <span class="nn">oauth2client.client</span> <span class="k">import</span> <span class="n">GoogleCredentials</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.log.logging_mixin</span> <span class="k">import</span> <span class="n">LoggingMixin</span>
+
+
+<span class="k">def</span> <span class="nf">_poll_with_exponential_delay</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">max_n</span><span class="p">,</span> <span class="n">is_done_func</span><span class="p">,</span> <span class="n">is_error_func</span><span class="p">):</span>
+ <span class="n">log</span> <span class="o">=</span> <span class="n">LoggingMixin</span><span class="p">()</span><span class="o">.</span><span class="n">log</span>
+
+ <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">max_n</span><span class="p">):</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">is_error_func</span><span class="p">(</span><span class="n">response</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s1">'The response contained an error: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
+ <span class="p">)</span>
+ <span class="k">elif</span> <span class="n">is_done_func</span><span class="p">(</span><span class="n">response</span><span class="p">):</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Operation is done: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">response</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">response</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">((</span><span class="mi">2</span><span class="o">**</span><span class="n">i</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="o">/</span> <span class="mi">1000</span><span class="p">))</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="o">!=</span> <span class="mi">429</span><span class="p">:</span>
+ <span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Something went wrong. Not retrying: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="nb">format</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="k">raise</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">((</span><span class="mi">2</span><span class="o">**</span><span class="n">i</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1000</span><span class="p">)</span> <span class="o">/</span> <span class="mi">1000</span><span class="p">))</span>
+
+
+<div class="viewcode-block" id="MLEngineHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook">[docs]</a><span class="k">class</span> <span class="nc">MLEngineHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">gcp_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_default'</span><span class="p">,</span> <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MLEngineHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">gcp_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+<div class="viewcode-block" id="MLEngineHook.get_conn"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a Google MLEngine service object.</span>
+<span class="sd"> """</span>
+ <span class="n">credentials</span> <span class="o">=</span> <span class="n">GoogleCredentials</span><span class="o">.</span><span class="n">get_application_default</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'ml'</span><span class="p">,</span> <span class="s1">'v1'</span><span class="p">,</span> <span class="n">credentials</span><span class="o">=</span><span class="n">credentials</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="MLEngineHook.create_job"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.create_job">[docs]</a> <span class="k">def</span> <span class="nf">create_job</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">job</span><span class="p">,</span> <span class="n">use_existing_job_fn</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Launches a MLEngine job and wait for it to reach a terminal state.</span>
+
+<span class="sd"> :param project_id: The Google Cloud project id within which MLEngine</span>
+<span class="sd"> job will be launched.</span>
+<span class="sd"> :type project_id: string</span>
+
+<span class="sd"> :param job: MLEngine Job object that should be provided to the MLEngine</span>
+<span class="sd"> API, such as:</span>
+<span class="sd"> {</span>
+<span class="sd"> 'jobId': 'my_job_id',</span>
+<span class="sd"> 'trainingInput': {</span>
+<span class="sd"> 'scaleTier': 'STANDARD_1',</span>
+<span class="sd"> ...</span>
+<span class="sd"> }</span>
+<span class="sd"> }</span>
+<span class="sd"> :type job: dict</span>
+
+<span class="sd"> :param use_existing_job_fn: In case that a MLEngine job with the same</span>
+<span class="sd"> job_id already exist, this method (if provided) will decide whether</span>
+<span class="sd"> we should use this existing job, continue waiting for it to finish</span>
+<span class="sd"> and returning the job object. It should accepts a MLEngine job</span>
+<span class="sd"> object, and returns a boolean value indicating whether it is OK to</span>
+<span class="sd"> reuse the existing job. If 'use_existing_job_fn' is not provided,</span>
+<span class="sd"> we by default reuse the existing MLEngine job.</span>
+<span class="sd"> :type use_existing_job_fn: function</span>
+
+<span class="sd"> :return: The MLEngine job object if the job successfully reach a</span>
+<span class="sd"> terminal state (which might be FAILED or CANCELLED state).</span>
+<span class="sd"> :rtype: dict</span>
+<span class="sd"> """</span>
+ <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">jobs</span><span class="p">()</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
+ <span class="n">parent</span><span class="o">=</span><span class="s1">'projects/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">project_id</span><span class="p">),</span>
+ <span class="n">body</span><span class="o">=</span><span class="n">job</span><span class="p">)</span>
+ <span class="n">job_id</span> <span class="o">=</span> <span class="n">job</span><span class="p">[</span><span class="s1">'jobId'</span><span class="p">]</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="c1"># 409 means there is an existing job with the same job ID.</span>
+ <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="mi">409</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">use_existing_job_fn</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">existing_job</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_job</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">use_existing_job_fn</span><span class="p">(</span><span class="n">existing_job</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
+ <span class="s1">'Job with job_id </span><span class="si">%s</span><span class="s1"> already exist, but it does '</span>
+ <span class="s1">'not match our expectation: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">job_id</span><span class="p">,</span> <span class="n">existing_job</span>
+ <span class="p">)</span>
+ <span class="k">raise</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Job with job_id </span><span class="si">%s</span><span class="s1"> already exist. Will waiting for it to finish'</span><span class="p">,</span>
+ <span class="n">job_id</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Failed to create MLEngine job: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="k">raise</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_job_done</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">_get_job</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Gets a MLEngine job based on the job name.</span>
+
+<span class="sd"> :return: MLEngine job object if succeed.</span>
+<span class="sd"> :rtype: dict</span>
+
+<span class="sd"> Raises:</span>
+<span class="sd"> apiclient.errors.HttpError: if HTTP error is returned from server</span>
+<span class="sd"> """</span>
+ <span class="n">job_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/jobs/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">jobs</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">job_name</span><span class="p">)</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="mi">429</span><span class="p">:</span>
+ <span class="c1"># polling after 30 seconds when quota failure occurs</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Failed to get MLEngine job: </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="k">raise</span>
+
+ <span class="k">def</span> <span class="nf">_wait_for_job_done</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">,</span> <span class="n">interval</span><span class="o">=</span><span class="mi">30</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for the Job to reach a terminal state.</span>
+
+<span class="sd"> This method will periodically check the job state until the job reach</span>
+<span class="sd"> a terminal state.</span>
+
+<span class="sd"> Raises:</span>
+<span class="sd"> apiclient.errors.HttpError: if HTTP error is returned when getting</span>
+<span class="sd"> the job</span>
+<span class="sd"> """</span>
+ <span class="k">assert</span> <span class="n">interval</span> <span class="o">></span> <span class="mi">0</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
+ <span class="n">job</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_job</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span> <span class="n">job_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">job</span><span class="p">[</span><span class="s1">'state'</span><span class="p">]</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'SUCCEEDED'</span><span class="p">,</span> <span class="s1">'FAILED'</span><span class="p">,</span> <span class="s1">'CANCELLED'</span><span class="p">]:</span>
+ <span class="k">return</span> <span class="n">job</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">interval</span><span class="p">)</span>
+
+<div class="viewcode-block" id="MLEngineHook.create_version"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.create_version">[docs]</a> <span class="k">def</span> <span class="nf">create_version</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">,</span> <span class="n">version_spec</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Creates the Version on Google Cloud ML Engine.</span>
+
+<span class="sd"> Returns the operation if the version was created successfully and</span>
+<span class="sd"> raises an error otherwise.</span>
+<span class="sd"> """</span>
+ <span class="n">parent_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/models/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">)</span>
+ <span class="n">create_request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">versions</span><span class="p">()</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
+ <span class="n">parent</span><span class="o">=</span><span class="n">parent_name</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">version_spec</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">create_request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="n">get_request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">operations</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">response</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+
+ <span class="k">return</span> <span class="n">_poll_with_exponential_delay</span><span class="p">(</span>
+ <span class="n">request</span><span class="o">=</span><span class="n">get_request</span><span class="p">,</span>
+ <span class="n">max_n</span><span class="o">=</span><span class="mi">9</span><span class="p">,</span>
+ <span class="n">is_done_func</span><span class="o">=</span><span class="k">lambda</span> <span class="n">resp</span><span class="p">:</span> <span class="n">resp</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'done'</span><span class="p">,</span> <span class="kc">False</span><span class="p">),</span>
+ <span class="n">is_error_func</span><span class="o">=</span><span class="k">lambda</span> <span class="n">resp</span><span class="p">:</span> <span class="n">resp</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'error'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="MLEngineHook.set_default_version"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.set_default_version">[docs]</a> <span class="k">def</span> <span class="nf">set_default_version</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">,</span> <span class="n">version_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Sets a version to be the default. Blocks until finished.</span>
+<span class="sd"> """</span>
+ <span class="n">full_version_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/models/</span><span class="si">{}</span><span class="s1">/versions/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">,</span> <span class="n">version_name</span><span class="p">)</span>
+ <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">versions</span><span class="p">()</span><span class="o">.</span><span class="n">setDefault</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">full_version_name</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="p">{})</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Successfully set version: </span><span class="si">%s</span><span class="s1"> to default'</span><span class="p">,</span> <span class="n">response</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">response</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Something went wrong: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
+ <span class="k">raise</span></div>
+
+<div class="viewcode-block" id="MLEngineHook.list_versions"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.list_versions">[docs]</a> <span class="k">def</span> <span class="nf">list_versions</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lists all available versions of a model. Blocks until finished.</span>
+<span class="sd"> """</span>
+ <span class="n">result</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="n">full_parent_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/models/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">)</span>
+ <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">versions</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">parent</span><span class="o">=</span><span class="n">full_parent_name</span><span class="p">,</span> <span class="n">pageSize</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
+
+ <span class="n">response</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="n">next_page_token</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'nextPageToken'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
+ <span class="n">result</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'versions'</span><span class="p">,</span> <span class="p">[]))</span>
+ <span class="k">while</span> <span class="n">next_page_token</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">next_request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">versions</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">parent</span><span class="o">=</span><span class="n">full_parent_name</span><span class="p">,</span>
+ <span class="n">pageToken</span><span class="o">=</span><span class="n">next_page_token</span><span class="p">,</span>
+ <span class="n">pageSize</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">next_request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="n">next_page_token</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'nextPageToken'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
+ <span class="n">result</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'versions'</span><span class="p">,</span> <span class="p">[]))</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">result</span></div>
+
+<div class="viewcode-block" id="MLEngineHook.delete_version"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.delete_version">[docs]</a> <span class="k">def</span> <span class="nf">delete_version</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">,</span> <span class="n">version_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Deletes the given version of a model. Blocks until finished.</span>
+<span class="sd"> """</span>
+ <span class="n">full_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/models/</span><span class="si">{}</span><span class="s1">/versions/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">,</span> <span class="n">version_name</span><span class="p">)</span>
+ <span class="n">delete_request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">versions</span><span class="p">()</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">full_name</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">delete_request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="n">get_request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">operations</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">response</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+
+ <span class="k">return</span> <span class="n">_poll_with_exponential_delay</span><span class="p">(</span>
+ <span class="n">request</span><span class="o">=</span><span class="n">get_request</span><span class="p">,</span>
+ <span class="n">max_n</span><span class="o">=</span><span class="mi">9</span><span class="p">,</span>
+ <span class="n">is_done_func</span><span class="o">=</span><span class="k">lambda</span> <span class="n">resp</span><span class="p">:</span> <span class="n">resp</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'done'</span><span class="p">,</span> <span class="kc">False</span><span class="p">),</span>
+ <span class="n">is_error_func</span><span class="o">=</span><span class="k">lambda</span> <span class="n">resp</span><span class="p">:</span> <span class="n">resp</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'error'</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="MLEngineHook.create_model"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.create_model">[docs]</a> <span class="k">def</span> <span class="nf">create_model</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">model</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Create a Model. Blocks until finished.</span>
+<span class="sd"> """</span>
+ <span class="k">assert</span> <span class="n">model</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">model</span><span class="p">[</span><span class="s1">'name'</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="s1">''</span>
+ <span class="n">project</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">project_id</span><span class="p">)</span>
+
+ <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
+ <span class="n">parent</span><span class="o">=</span><span class="n">project</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="n">model</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="MLEngineHook.get_model"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook.get_model">[docs]</a> <span class="k">def</span> <span class="nf">get_model</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Gets a Model. Blocks until finished.</span>
+<span class="sd"> """</span>
+ <span class="k">assert</span> <span class="n">model_name</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">model_name</span> <span class="ow">is</span> <span class="ow">not</span> <span class="s1">''</span>
+ <span class="n">full_model_name</span> <span class="o">=</span> <span class="s1">'projects/</span><span class="si">{}</span><span class="s1">/models/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">project_id</span><span class="p">,</span> <span class="n">model_name</span><span class="p">)</span>
+ <span class="n">request</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_mlengine</span><span class="o">.</span><span class="n">projects</span><span class="p">()</span><span class="o">.</span><span class="n">models</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">full_model_name</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">request</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">e</span><span class="o">.</span><span class="n">resp</span><span class="o">.</span><span class="n">status</span> <span class="o">==</span> <span class="mi">404</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Model was not found: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
+ <span class="k">return</span> <span class="kc">None</span>
+ <span class="k">raise</span></div></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/hooks/gcs_hook.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/hooks/gcs_hook.html b/_modules/airflow/contrib/hooks/gcs_hook.html
index 8edc584..6f3bc49 100644
--- a/_modules/airflow/contrib/hooks/gcs_hook.html
+++ b/_modules/airflow/contrib/hooks/gcs_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
<link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
<link rel="up" title="Module code" href="../../../index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../../../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../../../../index.html">Docs</a> »</li>
-
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
<li><a href="../../../index.html">Module code</a> »</li>
-
- <li>airflow.contrib.hooks.gcs_hook</li>
+
+ <li>airflow.contrib.hooks.gcs_hook</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,17 +181,14 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
<span class="c1">#</span>
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="k">import</span> <span class="n">build</span>
+<span class="kn">from</span> <span class="nn">apiclient.http</span> <span class="k">import</span> <span class="n">MediaFileUpload</span>
+<span class="kn">from</span> <span class="nn">googleapiclient</span> <span class="k">import</span> <span class="n">errors</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="kn">import</span> <span class="n">GoogleCloudBaseHook</span>
-<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="kn">import</span> <span class="n">build</span>
-<span class="kn">from</span> <span class="nn">apiclient.http</span> <span class="kn">import</span> <span class="n">MediaFileUpload</span>
-
-<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"google_cloud_storage"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcp_api_base_hook</span> <span class="k">import</span> <span class="n">GoogleCloudBaseHook</span>
-<div class="viewcode-block" id="GoogleCloudStorageHook"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Interact with Google Cloud Storage. This hook uses the Google Cloud Platform</span>
<span class="sd"> connection.</span>
@@ -169,17 +196,65 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_storage_default'</span><span class="p">,</span>
- <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.get_conn"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a Google Cloud Storage service object.</span>
<span class="sd"> """</span>
<span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'storage'</span><span class="p">,</span> <span class="s1">'v1'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'storage'</span><span class="p">,</span> <span class="s1">'v1'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
+
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.copy"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.copy">[docs]</a> <span class="k">def</span> <span class="nf">copy</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source_bucket</span><span class="p">,</span> <span class="n">source_object</span><span class="p">,</span> <span class="n">destination_bucket</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">destination_object</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Copies an object from a bucket to another, with renaming if requested.</span>
+
+<span class="sd"> destination_bucket or destination_object can be omitted, in which case</span>
+<span class="sd"> source bucket/object is used, but not both.</span>
+
+<span class="sd"> :param bucket: The bucket of the object to copy from.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The object to copy.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param destination_bucket: The destination of the object to copied to.</span>
+<span class="sd"> Can be omitted; then the same bucket is used.</span>
+<span class="sd"> :type destination_bucket: string</span>
+<span class="sd"> :param destination_object: The (renamed) path of the object if given.</span>
+<span class="sd"> Can be omitted; then the same name is used.</span>
+<span class="sd"> """</span>
+ <span class="n">destination_bucket</span> <span class="o">=</span> <span class="n">destination_bucket</span> <span class="ow">or</span> <span class="n">source_bucket</span>
+ <span class="n">destination_object</span> <span class="o">=</span> <span class="n">destination_object</span> <span class="ow">or</span> <span class="n">source_object</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">source_bucket</span> <span class="o">==</span> <span class="n">destination_bucket</span> <span class="ow">and</span>
+ <span class="n">source_object</span> <span class="o">==</span> <span class="n">destination_object</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
+ <span class="s1">'Either source/destination bucket or source/destination object '</span>
+ <span class="s1">'must be different, not both the same: bucket=</span><span class="si">%s</span><span class="s1">, object=</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span>
+ <span class="p">(</span><span class="n">source_bucket</span><span class="p">,</span> <span class="n">source_object</span><span class="p">))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">source_bucket</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">source_object</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'source_bucket and source_object cannot be empty.'</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">copy</span><span class="p">(</span><span class="n">sourceBucket</span><span class="o">=</span><span class="n">source_bucket</span><span class="p">,</span> <span class="n">sourceObject</span><span class="o">=</span><span class="n">source_object</span><span class="p">,</span>
+ <span class="n">destinationBucket</span><span class="o">=</span><span class="n">destination_bucket</span><span class="p">,</span>
+ <span class="n">destinationObject</span><span class="o">=</span><span class="n">destination_object</span><span class="p">,</span> <span class="n">body</span><span class="o">=</span><span class="s1">''</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div>
+
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.download"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.download">[docs]</a> <span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Get a file from Google Cloud Storage.</span>
@@ -198,12 +273,14 @@
<span class="c1"># Write the file to local file path, if requested.</span>
<span class="k">if</span> <span class="n">filename</span><span class="p">:</span>
- <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_fd</span><span class="p">:</span>
+ <span class="n">write_argument</span> <span class="o">=</span> <span class="s1">'wb'</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">downloaded_file_bytes</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">)</span> <span class="k">else</span> <span class="s1">'w'</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">write_argument</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_fd</span><span class="p">:</span>
<span class="n">file_fd</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">downloaded_file_bytes</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">downloaded_file_bytes</span>
+ <span class="k">return</span> <span class="n">downloaded_file_bytes</span></div>
- <span class="k">def</span> <span class="nf">upload</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mime_type</span><span class="o">=</span><span class="s1">'application/octet-stream'</span><span class="p">):</span>
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.upload"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.upload">[docs]</a> <span class="k">def</span> <span class="nf">upload</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mime_type</span><span class="o">=</span><span class="s1">'application/octet-stream'</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Uploads a local file to Google Cloud Storage.</span>
@@ -222,9 +299,145 @@
<span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
<span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="nb">object</span><span class="p">,</span> <span class="n">media_body</span><span class="o">=</span><span class="n">media</span><span class="p">)</span> \
<span class="o">.</span><span class="n">execute</span><span class="p">()</span></div>
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.exists"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.exists">[docs]</a> <span class="k">def</span> <span class="nf">exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks for the existence of a file in Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The Google cloud storage bucket where the object is.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The name of the object to check in the Google cloud</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div>
+
+ <span class="c1"># pylint:disable=redefined-builtin</span>
+<div class="viewcode-block" id="GoogleCloudStorageHook.is_updated_after"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.is_updated_after">[docs]</a> <span class="k">def</span> <span class="nf">is_updated_after</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">ts</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks if an object is updated in Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The Google cloud storage bucket where the object is.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The name of the object to check in the Google cloud</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param ts: The timestamp to check against.</span>
+<span class="sd"> :type ts: datetime</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="p">(</span><span class="n">service</span>
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span>
+ <span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">)</span>
+ <span class="o">.</span><span class="n">execute</span><span class="p">())</span>
+
+ <span class="k">if</span> <span class="s1">'updated'</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="kn">import</span> <span class="nn">dateutil.parser</span>
+ <span class="kn">import</span> <span class="nn">dateutil.tz</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">ts</span><span class="o">.</span><span class="n">tzinfo</span><span class="p">:</span>
+ <span class="n">ts</span> <span class="o">=</span> <span class="n">ts</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">tzinfo</span><span class="o">=</span><span class="n">dateutil</span><span class="o">.</span><span class="n">tz</span><span class="o">.</span><span class="n">tzutc</span><span class="p">())</span>
+
+ <span class="n">updated</span> <span class="o">=</span> <span class="n">dateutil</span><span class="o">.</span><span class="n">parser</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">response</span><span class="p">[</span><span class="s1">'updated'</span><span class="p">])</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Verify object date: </span><span class="si">%s</span><span class="s2"> > </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">updated</span><span class="p">,</span> <span class="n">ts</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">updated</span> <span class="o">></span> <span class="n">ts</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">True</span>
+
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">raise</span>
+
+ <span class="k">return</span> <span class="kc">False</span></div>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.delete"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.delete">[docs]</a> <span class="k">def</span> <span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">generation</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Delete an object if versioning is not enabled for the bucket, or if generation</span>
+<span class="sd"> parameter is used.</span>
+
+<span class="sd"> :param bucket: name of the bucket, where the object resides</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: name of the object to delete</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param generation: if present, permanently delete the object of this generation</span>
+<span class="sd"> :type generation: string</span>
+<span class="sd"> :return: True if succeeded</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">,</span> <span class="n">generation</span><span class="o">=</span><span class="n">generation</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+ <span class="k">return</span> <span class="kc">True</span>
+ <span class="k">except</span> <span class="n">errors</span><span class="o">.</span><span class="n">HttpError</span> <span class="k">as</span> <span class="n">ex</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ex</span><span class="o">.</span><span class="n">resp</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'404'</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">False</span>
+ <span class="k">raise</span></div>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.list"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list">[docs]</a> <span class="k">def</span> <span class="nf">list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="n">versions</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">maxResults</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> List all objects from the bucket with the give string prefix in name</span>
+
+<span class="sd"> :param bucket: bucket name</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param versions: if true, list all versions of the objects</span>
+<span class="sd"> :type versions: boolean</span>
+<span class="sd"> :param maxResults: max count of items to return in a single page of responses</span>
+<span class="sd"> :type maxResults: integer</span>
+<span class="sd"> :param prefix: prefix string which filters objects whose name begin with this prefix</span>
+<span class="sd"> :type prefix: string</span>
+<span class="sd"> :return: a stream of object names matching the filtering criteria</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="n">ids</span> <span class="o">=</span> <span class="nb">list</span><span class="p">()</span>
+ <span class="n">pageToken</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="k">while</span><span class="p">(</span><span class="kc">True</span><span class="p">):</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">objects</span><span class="p">()</span><span class="o">.</span><span class="n">list</span><span class="p">(</span>
+ <span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span>
+ <span class="n">versions</span><span class="o">=</span><span class="n">versions</span><span class="p">,</span>
+ <span class="n">maxResults</span><span class="o">=</span><span class="n">maxResults</span><span class="p">,</span>
+ <span class="n">pageToken</span><span class="o">=</span><span class="n">pageToken</span><span class="p">,</span>
+ <span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="s1">'items'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"No items found for prefix: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">prefix</span><span class="p">)</span>
+ <span class="k">break</span>
+
+ <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'items'</span><span class="p">]:</span>
+ <span class="k">if</span> <span class="n">item</span> <span class="ow">and</span> <span class="s1">'name'</span> <span class="ow">in</span> <span class="n">item</span><span class="p">:</span>
+ <span class="n">ids</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s1">'name'</span><span class="p">])</span>
+
+ <span class="k">if</span> <span class="s1">'nextPageToken'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+ <span class="c1"># no further pages of results, so stop the loop</span>
+ <span class="k">break</span>
+
+ <span class="n">pageToken</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="s1">'nextPageToken'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">pageToken</span><span class="p">:</span>
+ <span class="c1"># empty next page token</span>
+ <span class="k">break</span>
+ <span class="k">return</span> <span class="n">ids</span></div></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -257,7 +470,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../../../../_static/jquery.js"></script>
[03/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/license.html
----------------------------------------------------------------------
diff --git a/license.html b/license.html
index dd551bd..32e6ae2 100644
--- a/license.html
+++ b/license.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/objects.inv
----------------------------------------------------------------------
diff --git a/objects.inv b/objects.inv
index 6723914..f884cd0 100644
Binary files a/objects.inv and b/objects.inv differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/plugins.html
----------------------------------------------------------------------
diff --git a/plugins.html b/plugins.html
index be1cfbb..c956b7b 100644
--- a/plugins.html
+++ b/plugins.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -175,7 +180,7 @@ features to its core by simply dropping files in your
<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/plugins</span></code> folder.</p>
<p>The python modules in the <code class="docutils literal"><span class="pre">plugins</span></code> folder get imported,
and <strong>hooks</strong>, <strong>operators</strong>, <strong>macros</strong>, <strong>executors</strong> and web <strong>views</strong>
-get integrated to Airflow’s main collections and become available for use.</p>
+get integrated to Airflow’s main collections and become available for use.</p>
<div class="section" id="what-for">
<h2>What for?<a class="headerlink" href="#what-for" title="Permalink to this headline">¶</a></h2>
<p>Airflow offers a generic toolbox for working with data. Different
@@ -184,16 +189,16 @@ plugins can be a way for companies to customize their Airflow installation
to reflect their ecosystem.</p>
<p>Plugins can be used as an easy way to write, share and activate new sets of
features.</p>
-<p>There’s also a need for a set of more complex applications to interact with
+<p>There’s also a need for a set of more complex applications to interact with
different flavors of data and metadata.</p>
<p>Examples:</p>
<ul class="simple">
-<li>A set of tools to parse Hive logs and expose Hive metadata (CPU /IO / phases/ skew /...)</li>
+<li>A set of tools to parse Hive logs and expose Hive metadata (CPU /IO / phases/ skew /…)</li>
<li>An anomaly detection framework, allowing people to collect metrics, set thresholds and alerts</li>
<li>An auditing tool, helping understand who accesses what</li>
<li>A config-driven SLA monitoring tool, allowing you to set monitored tables and at what time
they should land, alert people, and expose visualizations of outages</li>
-<li>...</li>
+<li>…</li>
</ul>
</div>
<div class="section" id="why-build-on-top-of-airflow">
@@ -204,7 +209,7 @@ they should land, alert people, and expose visualizations of outages</li>
<li>A metadata database to store your models</li>
<li>Access to your databases, and knowledge of how to connect to them</li>
<li>An array of workers that your application can push workload to</li>
-<li>Airflow is deployed, you can just piggy back on it’s deployment logistics</li>
+<li>Airflow is deployed, you can just piggy back on it’s deployment logistics</li>
<li>Basic charting capabilities, underlying libraries and abstractions</li>
</ul>
</div>
@@ -212,7 +217,7 @@ they should land, alert people, and expose visualizations of outages</li>
<h2>Interface<a class="headerlink" href="#interface" title="Permalink to this headline">¶</a></h2>
<p>To create a plugin you will need to derive the
<code class="docutils literal"><span class="pre">airflow.plugins_manager.AirflowPlugin</span></code> class and reference the objects
-you want to plug into Airflow. Here’s what the class you need to derive
+you want to plug into Airflow. Here’s what the class you need to derive
looks like:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">AirflowPlugin</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="c1"># The name of your plugin (str)</span>
@@ -277,7 +282,7 @@ definitions in Airflow.</p>
<span class="c1"># Creating a flask blueprint to intergrate the templates and static folder</span>
<span class="n">bp</span> <span class="o">=</span> <span class="n">Blueprint</span><span class="p">(</span>
- <span class="s2">"test_plugin"</span><span class="p">,</span> <span class="n">__name__</span><span class="p">,</span>
+ <span class="s2">"test_plugin"</span><span class="p">,</span> <span class="vm">__name__</span><span class="p">,</span>
<span class="n">template_folder</span><span class="o">=</span><span class="s1">'templates'</span><span class="p">,</span> <span class="c1"># registers airflow/plugins/templates as a Jinja template folder</span>
<span class="n">static_folder</span><span class="o">=</span><span class="s1">'static'</span><span class="p">,</span>
<span class="n">static_url_path</span><span class="o">=</span><span class="s1">'/static/test_plugin'</span><span class="p">)</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/profiling.html
----------------------------------------------------------------------
diff --git a/profiling.html b/profiling.html
index 350c015..9438031 100644
--- a/profiling.html
+++ b/profiling.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -186,12 +191,12 @@ connections registered in Airflow.</p>
<h2>Charts<a class="headerlink" href="#charts" title="Permalink to this headline">¶</a></h2>
<p>A simple UI built on top of flask-admin and highcharts allows building
data visualizations and charts easily. Fill in a form with a label, SQL,
-chart type, pick a source database from your environment’s connectons,
+chart type, pick a source database from your environment’s connectons,
select a few other options, and save it for later use.</p>
<p>You can even use the same templating and macros available when writing
airflow pipelines, parameterizing your queries and modifying parameters
directly in the URL.</p>
-<p>These charts are basic, but they’re easy to create, modify and share.</p>
+<p>These charts are basic, but they’re easy to create, modify and share.</p>
<div class="section" id="chart-screenshot">
<h3>Chart Screenshot<a class="headerlink" href="#chart-screenshot" title="Permalink to this headline">¶</a></h3>
<img alt="_images/chart.png" src="_images/chart.png" />
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/project.html
----------------------------------------------------------------------
diff --git a/project.html b/project.html
index 1bc09d0..a60697e 100644
--- a/project.html
+++ b/project.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1 current"><a class="current reference internal" href="#">Project</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#history">History</a></li>
<li class="toctree-l2"><a class="reference internal" href="#committers">Committers</a></li>
@@ -175,13 +180,13 @@
<p>Airflow was started in October 2014 by Maxime Beauchemin at Airbnb.
It was open source from the very first commit and officially brought under
the Airbnb Github and announced in June 2015.</p>
-<p>The project joined the Apache Software Foundation’s incubation program in March 2016.</p>
+<p>The project joined the Apache Software Foundation’s incubation program in March 2016.</p>
</div>
<div class="section" id="committers">
<h2>Committers<a class="headerlink" href="#committers" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
-<li>@mistercrunch (Maxime “Max” Beauchemin)</li>
-<li>@r39132 (Siddharth “Sid” Anand)</li>
+<li>@mistercrunch (Maxime “Max” Beauchemin)</li>
+<li>@r39132 (Siddharth “Sid” Anand)</li>
<li>@criccomini (Chris Riccomini)</li>
<li>@bolkedebruin (Bolke de Bruin)</li>
<li>@artwr (Arthur Wiedmer)</li>
@@ -190,18 +195,18 @@ the Airbnb Github and announced in June 2015.</p>
<li>@aoen (Dan Davydov)</li>
<li>@syvineckruyk (Steven Yvinec-Kruyk)</li>
</ul>
-<p>For the full list of contributors, take a look at <a class="reference external" href="https://github.com/apache/incubator-airflow/graphs/contributors">Airflow’s Github
+<p>For the full list of contributors, take a look at <a class="reference external" href="https://github.com/apache/incubator-airflow/graphs/contributors">Airflow’s Github
Contributor page:</a></p>
</div>
<div class="section" id="resources-links">
<h2>Resources & links<a class="headerlink" href="#resources-links" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
-<li><a class="reference external" href="http://airflow.apache.org/">Airflow’s official documentation</a></li>
+<li><a class="reference external" href="http://airflow.apache.org/">Airflow’s official documentation</a></li>
<li>Mailing list (send emails to
<code class="docutils literal"><span class="pre">dev-subscribe@airflow.incubator.apache.org</span></code> and/or
<code class="docutils literal"><span class="pre">commits-subscribe@airflow.incubator.apache.org</span></code>
to subscribe to each)</li>
-<li><a class="reference external" href="https://issues.apache.org/jira/browse/AIRFLOW">Issues on Apache’s Jira</a></li>
+<li><a class="reference external" href="https://issues.apache.org/jira/browse/AIRFLOW">Issues on Apache’s Jira</a></li>
<li><a class="reference external" href="https://gitter.im/airbnb/airflow">Gitter (chat) Channel</a></li>
<li><a class="reference external" href="https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links">More resources and links to Airflow related content on the Wiki</a></li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/py-modindex.html
----------------------------------------------------------------------
diff --git a/py-modindex.html b/py-modindex.html
index 3f84d0f..4e57fc8 100644
--- a/py-modindex.html
+++ b/py-modindex.html
@@ -13,6 +13,8 @@
+
+
@@ -82,7 +84,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/scheduler.html
----------------------------------------------------------------------
diff --git a/scheduler.html b/scheduler.html
index 3f3aa9a..18f7a67 100644
--- a/scheduler.html
+++ b/scheduler.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -183,7 +188,7 @@ execute <code class="docutils literal"><span class="pre">airflow</span> <span cl
the run stamped <code class="docutils literal"><span class="pre">2016-01-01</span></code> will be trigger soon after <code class="docutils literal"><span class="pre">2016-01-01T23:59</span></code>.
In other words, the job instance is started once the period it covers
has ended.</p>
-<p><strong>Let’s Repeat That</strong> The scheduler runs your job one <code class="docutils literal"><span class="pre">schedule_interval</span></code> AFTER the
+<p><strong>Let’s Repeat That</strong> The scheduler runs your job one <code class="docutils literal"><span class="pre">schedule_interval</span></code> AFTER the
start date, at the END of the period.</p>
<p>The scheduler starts an instance of the executor specified in the your
<code class="docutils literal"><span class="pre">airflow.cfg</span></code>. If it happens to be the <code class="docutils literal"><span class="pre">LocalExecutor</span></code>, tasks will be
@@ -201,7 +206,7 @@ created. <code class="docutils literal"><span class="pre">schedule_interval</spa
preferably a
<a class="reference external" href="https://en.wikipedia.org/wiki/Cron#CRON_expression">cron expression</a> as
a <code class="docutils literal"><span class="pre">str</span></code>, or a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code> object. Alternatively, you can also
-use one of these cron “preset”:</p>
+use one of these cron “preset”:</p>
<table border="1" class="docutils">
<colgroup>
<col width="15%" />
@@ -216,7 +221,7 @@ use one of these cron “preset”:</p>
</thead>
<tbody valign="top">
<tr class="row-even"><td><code class="docutils literal"><span class="pre">None</span></code></td>
-<td>Don’t schedule, use for exclusively “externally triggered”
+<td>Don’t schedule, use for exclusively “externally triggered”
DAGs</td>
<td> </td>
</tr>
@@ -263,17 +268,41 @@ series of intervals which the scheduler turn into individual Dag Runs and execut
Airflow is that these DAG Runs are atomic, idempotent items, and the scheduler, by default, will examine
the lifetime of the DAG (from start to end/now, one interval at a time) and kick off a DAG Run for any
interval that has not been run (or has been cleared). This concept is called Catchup.</p>
-<p>If your DAG is written to handle it’s own catchup (IE not limited to the interval, but instead to “Now”
+<p>If your DAG is written to handle it’s own catchup (IE not limited to the interval, but instead to “Now”
for instance.), then you will want to turn catchup off (Either on the DAG itself with <code class="docutils literal"><span class="pre">dag.catchup</span> <span class="pre">=</span>
<span class="pre">False</span></code>) or by default at the configuration file level with <code class="docutils literal"><span class="pre">catchup_by_default</span> <span class="pre">=</span> <span class="pre">False</span></code>. What this
will do, is to instruct the scheduler to only create a DAG Run for the most current instance of the DAG
interval series.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="sd">"""</span>
+<span class="sd">Code that goes along with the Airflow tutorial located at:</span>
+<span class="sd">https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py</span>
+<span class="sd">"""</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">DAG</span>
+<span class="kn">from</span> <span class="nn">airflow.operators.bash_operator</span> <span class="k">import</span> <span class="n">BashOperator</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+
+
+<span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
+ <span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@example.com'</span><span class="p">],</span>
+ <span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="s1">'retry_delay'</span><span class="p">:</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">minutes</span><span class="o">=</span><span class="mi">5</span><span class="p">),</span>
+ <span class="s1">'schedule_interval'</span><span class="p">:</span> <span class="s1">'@hourly'</span><span class="p">,</span>
+<span class="p">}</span>
+
+<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">catchup</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span>
+</pre></div>
+</div>
<p>In the example above, if the DAG is picked up by the scheduler daemon on 2016-01-02 at 6 AM, (or from the
command line), a single DAG Run will be created, with an <code class="docutils literal"><span class="pre">execution_date</span></code> of 2016-01-01, and the next
one will be created just after midnight on the morning of 2016-01-03 with an execution date of 2016-01-02.</p>
<p>If the <code class="docutils literal"><span class="pre">dag.catchup</span></code> value had been True instead, the scheduler would have created a DAG Run for each
completed interval between 2015-12-01 and 2016-01-02 (but not yet one for 2016-01-02, as that interval
-hasn’t completed) and the scheduler will execute them sequentially. This behavior is great for atomic
+hasn’t completed) and the scheduler will execute them sequentially. This behavior is great for atomic
datasets that can easily be split into periods. Turning catchup off is great if your DAG Runs perform
backfill internally.</p>
</div>
@@ -282,7 +311,7 @@ backfill internally.</p>
<p>Note that <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> can also be created manually through the CLI while
running an <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">trigger_dag</span></code> command, where you can define a
specific <code class="docutils literal"><span class="pre">run_id</span></code>. The <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> created externally to the
-scheduler get associated to the trigger’s timestamp, and will be displayed
+scheduler get associated to the trigger’s timestamp, and will be displayed
in the UI alongside scheduled <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">runs</span></code>.</p>
</div>
<div class="section" id="to-keep-in-mind">
@@ -291,17 +320,28 @@ in the UI alongside scheduled <code class="docutils literal"><span class="pre">D
<li>The first <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code> is created based on the minimum <code class="docutils literal"><span class="pre">start_date</span></code> for the
tasks in your DAG.</li>
<li>Subsequent <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> are created by the scheduler process, based on
-your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>, sequentially.</li>
-<li>When clearing a set of tasks’ state in hope of getting them to re-run,
-it is important to keep in mind the <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code>‘s state too as it defines
+your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>, sequentially.</li>
+<li>When clearing a set of tasks’ state in hope of getting them to re-run,
+it is important to keep in mind the <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code>’s state too as it defines
whether the scheduler should look into triggering tasks for that run.</li>
</ul>
<p>Here are some of the ways you can <strong>unblock tasks</strong>:</p>
<ul class="simple">
-<li>From the UI, you can <strong>clear</strong> (as in delete the status of) individual task instances from the task instances dialog, while defining whether you want to includes the past/future and the upstream/downstream dependencies. Note that a confirmation window comes next and allows you to see the set you are about to clear.</li>
-<li>The CLI command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">clear</span> <span class="pre">-h</span></code> has lots of options when it comes to clearing task instance states, including specifying date ranges, targeting task_ids by specifying a regular expression, flags for including upstream and downstream relatives, and targeting task instances in specific states (<code class="docutils literal"><span class="pre">failed</span></code>, or <code class="docutils literal"><span class="pre">success</span></code>)</li>
-<li>Marking task instances as successful can be done through the UI. This is mostly to fix false negatives, or for instance when the fix has been applied outside of Airflow.</li>
-<li>The <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">backfill</span></code> CLI subcommand has a flag to <code class="docutils literal"><span class="pre">--mark_success</span></code> and allows selecting subsections of the DAG as well as specifying date ranges.</li>
+<li>From the UI, you can <strong>clear</strong> (as in delete the status of) individual task instances
+from the task instances dialog, while defining whether you want to includes the past/future
+and the upstream/downstream dependencies. Note that a confirmation window comes next and
+allows you to see the set you are about to clear. You can also clear all task instances
+associated with the dag.</li>
+<li>The CLI command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">clear</span> <span class="pre">-h</span></code> has lots of options when it comes to clearing task instance
+states, including specifying date ranges, targeting task_ids by specifying a regular expression,
+flags for including upstream and downstream relatives, and targeting task instances in specific
+states (<code class="docutils literal"><span class="pre">failed</span></code>, or <code class="docutils literal"><span class="pre">success</span></code>)</li>
+<li>Clearing a task instance will no longer delete the task instance record. Instead it updates
+max_tries and set the current task instance state to be None.</li>
+<li>Marking task instances as successful can be done through the UI. This is mostly to fix false negatives,
+or for instance when the fix has been applied outside of Airflow.</li>
+<li>The <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">backfill</span></code> CLI subcommand has a flag to <code class="docutils literal"><span class="pre">--mark_success</span></code> and allows selecting
+subsections of the DAG as well as specifying date ranges.</li>
</ul>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/search.html
----------------------------------------------------------------------
diff --git a/search.html b/search.html
index eed50cd..5121594 100644
--- a/search.html
+++ b/search.html
@@ -13,6 +13,8 @@
+
+
@@ -79,7 +81,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
[12/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/python_operator.html
----------------------------------------------------------------------
diff --git a/_modules/python_operator.html b/_modules/python_operator.html
index d2def55..5e5cdc6 100644
--- a/_modules/python_operator.html
+++ b/_modules/python_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -177,13 +182,20 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
-<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">TaskInstance</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="k">import</span> <span class="n">State</span>
+<span class="kn">import</span> <span class="nn">dill</span>
+<span class="kn">import</span> <span class="nn">inspect</span>
+<span class="kn">import</span> <span class="nn">os</span>
+<span class="kn">import</span> <span class="nn">pickle</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">import</span> <span class="nn">sys</span>
+<span class="kn">import</span> <span class="nn">types</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">SkipMixin</span>
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
-<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">settings</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="k">import</span> <span class="n">TemporaryDirectory</span>
+
+<span class="kn">from</span> <span class="nn">textwrap</span> <span class="k">import</span> <span class="n">dedent</span>
<div class="viewcode-block" id="PythonOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PythonOperator">[docs]</a><span class="k">class</span> <span class="nc">PythonOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -226,7 +238,9 @@
<span class="n">templates_dict</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">templates_exts</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">callable</span><span class="p">(</span><span class="n">python_callable</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'`python_callable` param must be callable'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span> <span class="o">=</span> <span class="n">python_callable</span>
<span class="bp">self</span><span class="o">.</span><span class="n">op_args</span> <span class="o">=</span> <span class="n">op_args</span> <span class="ow">or</span> <span class="p">[]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span> <span class="o">=</span> <span class="n">op_kwargs</span> <span class="ow">or</span> <span class="p">{}</span>
@@ -241,12 +255,15 @@
<span class="n">context</span><span class="p">[</span><span class="s1">'templates_dict'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">templates_dict</span>
<span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span> <span class="o">=</span> <span class="n">context</span>
- <span class="n">return_value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">op_args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done. Returned value was: "</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">return_value</span><span class="p">))</span>
- <span class="k">return</span> <span class="n">return_value</span></div>
+ <span class="n">return_value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">execute_callable</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done. Returned value was: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">return_value</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">return_value</span>
+
+ <span class="k">def</span> <span class="nf">execute_callable</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">op_args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span><span class="p">)</span></div>
-<div class="viewcode-block" id="BranchPythonOperator"><a class="viewcode-back" href="../code.html#airflow.operators.BranchPythonOperator">[docs]</a><span class="k">class</span> <span class="nc">BranchPythonOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">):</span>
+<div class="viewcode-block" id="BranchPythonOperator"><a class="viewcode-back" href="../code.html#airflow.operators.BranchPythonOperator">[docs]</a><span class="k">class</span> <span class="nc">BranchPythonOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">,</span> <span class="n">SkipMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Allows a workflow to "branch" or follow a single path following the</span>
<span class="sd"> execution of this task.</span>
@@ -267,23 +284,20 @@
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">branch</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">BranchPythonOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Following branch "</span> <span class="o">+</span> <span class="n">branch</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Marking other directly downstream tasks as skipped"</span><span class="p">)</span>
- <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
- <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span><span class="o">.</span><span class="n">downstream_list</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">task_id</span> <span class="o">!=</span> <span class="n">branch</span><span class="p">:</span>
- <span class="n">ti</span> <span class="o">=</span> <span class="n">TaskInstance</span><span class="p">(</span>
- <span class="n">task</span><span class="p">,</span> <span class="n">execution_date</span><span class="o">=</span><span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">execution_date</span><span class="p">)</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SKIPPED</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">end_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
- <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
- <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
- <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
-
-
-<div class="viewcode-block" id="ShortCircuitOperator"><a class="viewcode-back" href="../code.html#airflow.operators.ShortCircuitOperator">[docs]</a><span class="k">class</span> <span class="nc">ShortCircuitOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Following branch </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">branch</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Marking other directly downstream tasks as skipped"</span><span class="p">)</span>
+
+ <span class="n">downstream_tasks</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span><span class="o">.</span><span class="n">downstream_list</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Downstream task_ids </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">downstream_tasks</span><span class="p">)</span>
+
+ <span class="n">skip_tasks</span> <span class="o">=</span> <span class="p">[</span><span class="n">t</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">downstream_tasks</span> <span class="k">if</span> <span class="n">t</span><span class="o">.</span><span class="n">task_id</span> <span class="o">!=</span> <span class="n">branch</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">downstream_tasks</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">skip</span><span class="p">(</span><span class="n">context</span><span class="p">[</span><span class="s1">'dag_run'</span><span class="p">],</span> <span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">execution_date</span><span class="p">,</span> <span class="n">skip_tasks</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="ShortCircuitOperator"><a class="viewcode-back" href="../code.html#airflow.operators.ShortCircuitOperator">[docs]</a><span class="k">class</span> <span class="nc">ShortCircuitOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">,</span> <span class="n">SkipMixin</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Allows a workflow to continue only if a condition is met. Otherwise, the</span>
<span class="sd"> workflow "short-circuits" and downstream tasks are skipped.</span>
@@ -297,23 +311,220 @@
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">condition</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">ShortCircuitOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Condition result is </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">condition</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Condition result is </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">condition</span><span class="p">)</span>
+
<span class="k">if</span> <span class="n">condition</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Proceeding with downstream tasks...'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Proceeding with downstream tasks...'</span><span class="p">)</span>
<span class="k">return</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Skipping downstream tasks...'</span><span class="p">)</span>
+
+ <span class="n">downstream_tasks</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span><span class="o">.</span><span class="n">get_flat_relatives</span><span class="p">(</span><span class="n">upstream</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Downstream task_ids </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">downstream_tasks</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">downstream_tasks</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">skip</span><span class="p">(</span><span class="n">context</span><span class="p">[</span><span class="s1">'dag_run'</span><span class="p">],</span> <span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">execution_date</span><span class="p">,</span> <span class="n">downstream_tasks</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+
+<span class="k">class</span> <span class="nc">PythonVirtualenvOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Allows one to run a function in a virtualenv that is created and destroyed</span>
+<span class="sd"> automatically (with certain caveats).</span>
+
+<span class="sd"> The function must be defined using def, and not be part of a class. All imports</span>
+<span class="sd"> must happen inside the function and no variables outside of the scope may be referenced.</span>
+<span class="sd"> A global scope variable named virtualenv_string_args will be available (populated by</span>
+<span class="sd"> string_args). In addition, one can pass stuff through op_args and op_kwargs, and one</span>
+<span class="sd"> can use a return value.</span>
+
+<span class="sd"> Note that if your virtualenv runs in a different Python major version than Airflow,</span>
+<span class="sd"> you cannot use return values, op_args, or op_kwargs. You can use string_args though.</span>
+
+<span class="sd"> :param python_callable: A python function with no references to outside variables,</span>
+<span class="sd"> defined with def, which will be run in a virtualenv</span>
+<span class="sd"> :type python_callable: function</span>
+<span class="sd"> :param requirements: A list of requirements as specified in a pip install command</span>
+<span class="sd"> :type requirements: list(str)</span>
+<span class="sd"> :param python_version: The Python version to run the virtualenv with. Note that</span>
+<span class="sd"> both 2 and 2.7 are acceptable forms.</span>
+<span class="sd"> :type python_version: str</span>
+<span class="sd"> :param use_dill: Whether to use dill to serialize the args and result (pickle is default).</span>
+<span class="sd"> This allow more complex types but requires you to include dill in your requirements.</span>
+<span class="sd"> :type use_dill: bool</span>
+<span class="sd"> :param system_site_packages: Whether to include system_site_packages in your virtualenv.</span>
+<span class="sd"> See virtualenv documentation for more information.</span>
+<span class="sd"> :type system_site_packages: bool</span>
+<span class="sd"> :param op_args: A list of positional arguments to pass to python_callable.</span>
+<span class="sd"> :type op_kwargs: list</span>
+<span class="sd"> :param op_kwargs: A dict of keyword arguments to pass to python_callable.</span>
+<span class="sd"> :type op_kwargs: dict</span>
+<span class="sd"> :param string_args: Strings that are present in the global var virtualenv_string_args,</span>
+<span class="sd"> available to python_callable at runtime as a list(str). Note that args are split</span>
+<span class="sd"> by newline.</span>
+<span class="sd"> :type string_args: list(str)</span>
+
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">python_callable</span><span class="p">,</span> <span class="n">requirements</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">python_version</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">use_dill</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">system_site_packages</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">op_args</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">op_kwargs</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">string_args</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PythonVirtualenvOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
+ <span class="n">python_callable</span><span class="o">=</span><span class="n">python_callable</span><span class="p">,</span>
+ <span class="n">op_args</span><span class="o">=</span><span class="n">op_args</span><span class="p">,</span>
+ <span class="n">op_kwargs</span><span class="o">=</span><span class="n">op_kwargs</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">requirements</span> <span class="o">=</span> <span class="n">requirements</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">string_args</span> <span class="o">=</span> <span class="n">string_args</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">python_version</span> <span class="o">=</span> <span class="n">python_version</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">use_dill</span> <span class="o">=</span> <span class="n">use_dill</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">system_site_packages</span> <span class="o">=</span> <span class="n">system_site_packages</span>
+ <span class="c1"># check that dill is present if needed</span>
+ <span class="n">dill_in_requirements</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'dill'</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">requirements</span><span class="p">)</span>
+ <span class="k">if</span> <span class="p">(</span><span class="ow">not</span> <span class="n">system_site_packages</span><span class="p">)</span> <span class="ow">and</span> <span class="n">use_dill</span> <span class="ow">and</span> <span class="ow">not</span> <span class="nb">any</span><span class="p">(</span><span class="n">dill_in_requirements</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'If using dill, dill must be in the environment '</span> <span class="o">+</span>
+ <span class="s1">'either via system_site_packages or requirements'</span><span class="p">)</span>
+ <span class="c1"># check that a function is passed, and that it is not a lambda</span>
+ <span class="k">if</span> <span class="p">(</span><span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="p">,</span> <span class="n">types</span><span class="o">.</span><span class="n">FunctionType</span><span class="p">)</span>
+ <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">==</span> <span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'</span><span class="si">{}</span><span class="s1"> only supports functions for python_callable arg'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span><span class="p">)</span>
+ <span class="c1"># check that args are passed iff python major version matches</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">python_version</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
+ <span class="ow">and</span> <span class="nb">str</span><span class="p">(</span><span class="n">python_version</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="o">!=</span> <span class="nb">str</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
+ <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pass_op_args</span><span class="p">()):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Passing op_args or op_kwargs is not supported across "</span>
+ <span class="s2">"different Python major versions "</span>
+ <span class="s2">"for PythonVirtualenvOperator. Please use string_args."</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute_callable</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'venv'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">:</span>
+ <span class="c1"># generate filenames</span>
+ <span class="n">input_filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="s1">'script.in'</span><span class="p">)</span>
+ <span class="n">output_filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="s1">'script.out'</span><span class="p">)</span>
+ <span class="n">string_args_filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="s1">'string_args.txt'</span><span class="p">)</span>
+ <span class="n">script_filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="s1">'script.py'</span><span class="p">)</span>
+
+ <span class="c1"># set up virtualenv</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_execute_in_subprocess</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_generate_virtualenv_cmd</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">))</span>
+ <span class="n">cmd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_generate_pip_install_cmd</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">cmd</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_execute_in_subprocess</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">_write_args</span><span class="p">(</span><span class="n">input_filename</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_write_script</span><span class="p">(</span><span class="n">script_filename</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_write_string_args</span><span class="p">(</span><span class="n">string_args_filename</span><span class="p">)</span>
+
+ <span class="c1"># execute command in virtualenv</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_execute_in_subprocess</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_generate_python_cmd</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">,</span>
+ <span class="n">script_filename</span><span class="p">,</span>
+ <span class="n">input_filename</span><span class="p">,</span>
+ <span class="n">output_filename</span><span class="p">,</span>
+ <span class="n">string_args_filename</span><span class="p">))</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_read_result</span><span class="p">(</span><span class="n">output_filename</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_pass_op_args</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="c1"># we should only pass op_args if any are given to us</span>
+ <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">op_args</span><span class="p">)</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span>
+
+ <span class="k">def</span> <span class="nf">_execute_in_subprocess</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cmd</span><span class="p">):</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing cmd</span><span class="se">\n</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">cmd</span><span class="p">))</span>
+ <span class="n">output</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">check_output</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">STDOUT</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">output</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Got output</span><span class="se">\n</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">output</span><span class="p">))</span>
+ <span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Got error output</span><span class="se">\n</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">output</span><span class="p">))</span>
+ <span class="k">raise</span>
+
+ <span class="k">def</span> <span class="nf">_write_string_args</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
+ <span class="c1"># writes string_args to a file, which are read line by line</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">string_args</span><span class="p">)))</span>
+
+ <span class="k">def</span> <span class="nf">_write_args</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_filename</span><span class="p">):</span>
+ <span class="c1"># serialize args to file</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pass_op_args</span><span class="p">():</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">input_filename</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">arg_dict</span> <span class="o">=</span> <span class="p">({</span><span class="s1">'args'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">op_args</span><span class="p">,</span> <span class="s1">'kwargs'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span><span class="p">})</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">use_dill</span><span class="p">:</span>
+ <span class="n">dill</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">arg_dict</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">pickle</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">arg_dict</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_read_result</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">output_filename</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">stat</span><span class="p">(</span><span class="n">output_filename</span><span class="p">)</span><span class="o">.</span><span class="n">st_size</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">None</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">output_filename</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">use_dill</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">dill</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
+ <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Error deserializing result. Note that result deserialization "</span>
+ <span class="s2">"is not supported across major Python versions."</span><span class="p">)</span>
+ <span class="k">raise</span>
+
+ <span class="k">def</span> <span class="nf">_write_script</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">script_filename</span><span class="p">):</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">script_filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">python_code</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_generate_python_code</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Writing code to file</span><span class="se">\n</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">python_code</span><span class="p">))</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">python_code</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_generate_virtualenv_cmd</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tmp_dir</span><span class="p">):</span>
+ <span class="n">cmd</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'virtualenv'</span><span class="p">,</span> <span class="n">tmp_dir</span><span class="p">]</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">system_site_packages</span><span class="p">:</span>
+ <span class="n">cmd</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'--system-site-packages'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_version</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">cmd</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'--python=python</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">python_version</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">cmd</span>
+
+ <span class="k">def</span> <span class="nf">_generate_pip_install_cmd</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tmp_dir</span><span class="p">):</span>
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">requirements</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">return</span> <span class="p">[]</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="c1"># direct path alleviates need to activate</span>
+ <span class="n">cmd</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'</span><span class="si">{}</span><span class="s1">/bin/pip'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">),</span> <span class="s1">'install'</span><span class="p">]</span>
+ <span class="k">return</span> <span class="n">cmd</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">requirements</span>
+
+ <span class="k">def</span> <span class="nf">_generate_python_cmd</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tmp_dir</span><span class="p">,</span> <span class="n">script_filename</span><span class="p">,</span> <span class="n">input_filename</span><span class="p">,</span> <span class="n">output_filename</span><span class="p">,</span> <span class="n">string_args_filename</span><span class="p">):</span>
+ <span class="c1"># direct path alleviates need to activate</span>
+ <span class="k">return</span> <span class="p">[</span><span class="s1">'</span><span class="si">{}</span><span class="s1">/bin/python'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tmp_dir</span><span class="p">),</span> <span class="n">script_filename</span><span class="p">,</span> <span class="n">input_filename</span><span class="p">,</span> <span class="n">output_filename</span><span class="p">,</span> <span class="n">string_args_filename</span><span class="p">]</span>
+
+ <span class="k">def</span> <span class="nf">_generate_python_code</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">use_dill</span><span class="p">:</span>
+ <span class="n">pickling_library</span> <span class="o">=</span> <span class="s1">'dill'</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">pickling_library</span> <span class="o">=</span> <span class="s1">'pickle'</span>
+ <span class="n">fn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span>
+ <span class="c1"># dont try to read pickle if we didnt pass anything</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pass_op_args</span><span class="p">():</span>
+ <span class="n">load_args_line</span> <span class="o">=</span> <span class="s1">'with open(sys.argv[1], "rb") as f: arg_dict = </span><span class="si">{}</span><span class="s1">.load(f)'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">pickling_library</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Skipping downstream tasks...'</span><span class="p">)</span>
- <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
- <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span><span class="o">.</span><span class="n">downstream_list</span><span class="p">:</span>
- <span class="n">ti</span> <span class="o">=</span> <span class="n">TaskInstance</span><span class="p">(</span>
- <span class="n">task</span><span class="p">,</span> <span class="n">execution_date</span><span class="o">=</span><span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">execution_date</span><span class="p">)</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SKIPPED</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
- <span class="n">ti</span><span class="o">.</span><span class="n">end_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
- <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
- <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
- <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+ <span class="n">load_args_line</span> <span class="o">=</span> <span class="s1">'arg_dict = {"args": [], "kwargs": </span><span class="si">{}</span><span class="s1">}'</span>
+
+ <span class="c1"># no indents in original code so we can accept any type of indents in the original function</span>
+ <span class="c1"># we deserialize args, call function, serialize result if necessary</span>
+ <span class="k">return</span> <span class="n">dedent</span><span class="p">(</span><span class="s2">"""</span><span class="se">\</span>
+<span class="s2"> import </span><span class="si">{pickling_library}</span><span class="s2"></span>
+<span class="s2"> import sys</span>
+<span class="s2"> </span><span class="si">{load_args_code}</span><span class="s2"></span>
+<span class="s2"> args = arg_dict["args"]</span>
+<span class="s2"> kwargs = arg_dict["kwargs"]</span>
+<span class="s2"> with open(sys.argv[3], 'r') as f: virtualenv_string_args = list(map(lambda x: x.strip(), list(f)))</span>
+<span class="s2"> </span><span class="si">{python_callable_lines}</span><span class="s2"></span>
+<span class="s2"> res = </span><span class="si">{python_callable_name}</span><span class="s2">(*args, **kwargs)</span>
+<span class="s2"> with open(sys.argv[2], 'wb') as f: res is not None and </span><span class="si">{pickling_library}</span><span class="s2">.dump(res, f)</span>
+<span class="s2"> """</span><span class="p">)</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">load_args_code</span><span class="o">=</span><span class="n">load_args_line</span><span class="p">,</span>
+ <span class="n">python_callable_lines</span><span class="o">=</span><span class="n">dedent</span><span class="p">(</span><span class="n">inspect</span><span class="o">.</span><span class="n">getsource</span><span class="p">(</span><span class="n">fn</span><span class="p">)),</span>
+ <span class="n">python_callable_name</span><span class="o">=</span><span class="n">fn</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span>
+ <span class="n">pickling_library</span><span class="o">=</span><span class="n">pickling_library</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span>
+
</pre></div>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/qubole_operator.html
----------------------------------------------------------------------
diff --git a/_modules/qubole_operator.html b/_modules/qubole_operator.html
index fb33047..c6fdd28 100644
--- a/_modules/qubole_operator.html
+++ b/_modules/qubole_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>qubole_operator</li>
+
+ <li>qubole_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,9 +181,9 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.qubole_hook</span> <span class="kn">import</span> <span class="n">QuboleHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.qubole_hook</span> <span class="k">import</span> <span class="n">QuboleHook</span>
<div class="viewcode-block" id="QuboleOperator"><a class="viewcode-back" href="../code.html#airflow.contrib.operators.QuboleOperator">[docs]</a><span class="k">class</span> <span class="nc">QuboleOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -168,6 +198,7 @@
<span class="sd"> :tags: array of tags to be assigned with the command</span>
<span class="sd"> :cluster_label: cluster label on which the command will be executed</span>
<span class="sd"> :name: name to be given to command</span>
+<span class="sd"> :notify: whether to send email on command completion or not (default is False)</span>
<span class="sd"> **Arguments specific to command types**</span>
@@ -181,20 +212,28 @@
<span class="sd"> :script_location: s3 location containing query statement</span>
<span class="sd"> :macros: macro values which were used in query</span>
<span class="sd"> hadoopcmd:</span>
-<span class="sd"> :sub_commnad: must be one these ["jar", "s3distcp", "streaming"] followed by 1 or more args</span>
+<span class="sd"> :sub_commnad: must be one these ["jar", "s3distcp", "streaming"] followed by</span>
+<span class="sd"> 1 or more args</span>
<span class="sd"> shellcmd:</span>
<span class="sd"> :script: inline command with args</span>
<span class="sd"> :script_location: s3 location containing query statement</span>
-<span class="sd"> :files: list of files in s3 bucket as file1,file2 format. These files will be copied into the working directory where the qubole command is being executed.</span>
-<span class="sd"> :archives: list of archives in s3 bucket as archive1,archive2 format. These will be unarchived intothe working directory where the qubole command is being executed</span>
-<span class="sd"> :parameters: any extra args which need to be passed to script (only when script_location is supplied)</span>
+<span class="sd"> :files: list of files in s3 bucket as file1,file2 format. These files will be</span>
+<span class="sd"> copied into the working directory where the qubole command is being</span>
+<span class="sd"> executed.</span>
+<span class="sd"> :archives: list of archives in s3 bucket as archive1,archive2 format. These</span>
+<span class="sd"> will be unarchived intothe working directory where the qubole command is</span>
+<span class="sd"> being executed</span>
+<span class="sd"> :parameters: any extra args which need to be passed to script (only when</span>
+<span class="sd"> script_location is supplied)</span>
<span class="sd"> pigcmd:</span>
<span class="sd"> :script: inline query statement (latin_statements)</span>
<span class="sd"> :script_location: s3 location containing pig query</span>
-<span class="sd"> :parameters: any extra args which need to be passed to script (only when script_location is supplied</span>
+<span class="sd"> :parameters: any extra args which need to be passed to script (only when</span>
+<span class="sd"> script_location is supplied</span>
<span class="sd"> sparkcmd:</span>
<span class="sd"> :program: the complete Spark Program in Scala, SQL, Command, R, or Python</span>
-<span class="sd"> :cmdline: spark-submit command line, all required information must be specify in cmdline itself.</span>
+<span class="sd"> :cmdline: spark-submit command line, all required information must be specify</span>
+<span class="sd"> in cmdline itself.</span>
<span class="sd"> :sql: inline sql query</span>
<span class="sd"> :script_location: s3 location containing query statement</span>
<span class="sd"> :language: language of the program, Scala, SQL, Command, R, or Python</span>
@@ -215,7 +254,7 @@
<span class="sd"> :db_update_mode: allowinsert or updateonly</span>
<span class="sd"> :db_update_keys: columns used to determine the uniqueness of rows</span>
<span class="sd"> :export_dir: HDFS/S3 location from which data will be exported.</span>
-<span class="sd"> :fields_terminated_by: hex of the char used as column separator in the dataset.</span>
+<span class="sd"> :fields_terminated_by: hex of the char used as column separator in the dataset</span>
<span class="sd"> dbimportcmd:</span>
<span class="sd"> :mode: 1 (simple), 2 (advance)</span>
<span class="sd"> :hive_table: Name of the hive table</span>
@@ -223,17 +262,32 @@
<span class="sd"> :db_table: name of the db table</span>
<span class="sd"> :where_clause: where clause, if any</span>
<span class="sd"> :parallelism: number of parallel db connections to use for extracting data</span>
-<span class="sd"> :extract_query: SQL query to extract data from db. $CONDITIONS must be part of the where clause.</span>
+<span class="sd"> :extract_query: SQL query to extract data from db. $CONDITIONS must be part</span>
+<span class="sd"> of the where clause.</span>
<span class="sd"> :boundary_query: Query to be used get range of row IDs to be extracted</span>
<span class="sd"> :split_column: Column used as row ID to split data into ranges (mode 2)</span>
-<span class="sd"> .. note:: Following fields are template-supported : ``query``, ``script_location``, ``sub_command``, ``script``, ``files``,</span>
-<span class="sd"> ``archives``, ``program``, ``cmdline``, ``sql``, ``where_clause``, ``extract_query``, ``boundary_query``, ``macros``, ``tags``,</span>
-<span class="sd"> ``name``, ``parameters``. You can also use ``.txt`` files for template driven use cases.</span>
+<span class="sd"> .. note:: Following fields are template-supported : ``query``, ``script_location``,</span>
+<span class="sd"> ``sub_command``, ``script``, ``files``, ``archives``, ``program``, ``cmdline``,</span>
+<span class="sd"> ``sql``, ``where_clause``, ``extract_query``, ``boundary_query``, ``macros``,</span>
+<span class="sd"> ``tags``, ``name``, ``parameters``, ``dbtap_id``, ``hive_table``, ``db_table``,</span>
+<span class="sd"> ``split_column``, ``note_id``, ``db_update_keys``, ``export_dir``,</span>
+<span class="sd"> ``partition_spec``, ``qubole_conn_id``, ``arguments``, ``user_program_arguments``.</span>
+<span class="sd"> You can also use ``.txt`` files for template driven use cases.</span>
+
+<span class="sd"> .. note:: In QuboleOperator there is a default handler for task failures and retries,</span>
+<span class="sd"> which generally kills the command running at QDS for the corresponding task</span>
+<span class="sd"> instance. You can override this behavior by providing your own failure and retry</span>
+<span class="sd"> handler in task definition.</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'query'</span><span class="p">,</span> <span class="s1">'script_location'</span><span class="p">,</span> <span class="s1">'sub_command'</span><span class="p">,</span> <span class="s1">'script'</span><span class="p">,</span> <span class="s1">'files'</span><span class="p">,</span> <span class="s1">'archives'</span><span class="p">,</span> <span class="s1">'program'</span><span class="p">,</span> <span class="s1">'cmdline'</span><span class="p">,</span>
- <span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'where_clause'</span><span class="p">,</span> <span class="s1">'extract_query'</span><span class="p">,</span> <span class="s1">'boundary_query'</span><span class="p">,</span> <span class="s1">'macros'</span><span class="p">,</span> <span class="s1">'tags'</span><span class="p">,</span> <span class="s1">'name'</span><span class="p">,</span> <span class="s1">'parameters'</span><span class="p">)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'query'</span><span class="p">,</span> <span class="s1">'script_location'</span><span class="p">,</span> <span class="s1">'sub_command'</span><span class="p">,</span> <span class="s1">'script'</span><span class="p">,</span> <span class="s1">'files'</span><span class="p">,</span>
+ <span class="s1">'archives'</span><span class="p">,</span> <span class="s1">'program'</span><span class="p">,</span> <span class="s1">'cmdline'</span><span class="p">,</span> <span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'where_clause'</span><span class="p">,</span> <span class="s1">'tags'</span><span class="p">,</span>
+ <span class="s1">'extract_query'</span><span class="p">,</span> <span class="s1">'boundary_query'</span><span class="p">,</span> <span class="s1">'macros'</span><span class="p">,</span> <span class="s1">'name'</span><span class="p">,</span> <span class="s1">'parameters'</span><span class="p">,</span>
+ <span class="s1">'dbtap_id'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">,</span> <span class="s1">'db_table'</span><span class="p">,</span> <span class="s1">'split_column'</span><span class="p">,</span> <span class="s1">'note_id'</span><span class="p">,</span>
+ <span class="s1">'db_update_keys'</span><span class="p">,</span> <span class="s1">'export_dir'</span><span class="p">,</span> <span class="s1">'partition_spec'</span><span class="p">,</span> <span class="s1">'qubole_conn_id'</span><span class="p">,</span>
+ <span class="s1">'arguments'</span><span class="p">,</span> <span class="s1">'user_program_arguments'</span><span class="p">)</span>
+
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.txt'</span><span class="p">,)</span>
<span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#3064A1'</span>
<span class="n">ui_fgcolor</span> <span class="o">=</span> <span class="s1">'#fff'</span>
@@ -243,25 +297,32 @@
<span class="bp">self</span><span class="o">.</span><span class="n">args</span> <span class="o">=</span> <span class="n">args</span>
<span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span> <span class="o">=</span> <span class="n">kwargs</span>
<span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span><span class="p">[</span><span class="s1">'qubole_conn_id'</span><span class="p">]</span> <span class="o">=</span> <span class="n">qubole_conn_id</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">QuboleHook</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span><span class="p">)</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">QuboleOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">QuboleOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">on_failure_callback</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">on_failure_callback</span> <span class="o">=</span> <span class="n">QuboleHook</span><span class="o">.</span><span class="n">handle_failure_retry</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">on_retry_callback</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">on_retry_callback</span> <span class="o">=</span> <span class="n">QuboleHook</span><span class="o">.</span><span class="n">handle_failure_retry</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="c1"># Reinitiating the hook, as some template fields might have changed</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">QuboleHook</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ti</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">kill</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ti</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span><span class="o">.</span><span class="n">kill</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
- <span class="k">def</span> <span class="nf">get_results</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ti</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">fp</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">inline</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">delim</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">get_results</span><span class="p">(</span><span class="n">ti</span><span class="p">,</span> <span class="n">fp</span><span class="p">,</span> <span class="n">inline</span><span class="p">,</span> <span class="n">delim</span><span class="p">,</span> <span class="n">fetch</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">get_results</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ti</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">fp</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">inline</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">delim</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">fetch</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span><span class="o">.</span><span class="n">get_results</span><span class="p">(</span><span class="n">ti</span><span class="p">,</span> <span class="n">fp</span><span class="p">,</span> <span class="n">inline</span><span class="p">,</span> <span class="n">delim</span><span class="p">,</span> <span class="n">fetch</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_log</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ti</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">get_log</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span><span class="o">.</span><span class="n">get_log</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_jobs_id</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ti</span><span class="p">):</span>
- <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">get_jobs_id</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span><span class="o">.</span><span class="n">get_jobs_id</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="c1"># Reinitiating the hook, as some template fields might have changed</span>
+ <span class="k">return</span> <span class="n">QuboleHook</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__getattribute__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">QuboleOperator</span><span class="o">.</span><span class="n">template_fields</span><span class="p">:</span>
@@ -270,19 +331,19 @@
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="s1">''</span>
<span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="nb">object</span><span class="o">.</span><span class="n">__getattribute__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="nb">object</span><span class="o">.</span><span class="fm">__getattribute__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">QuboleOperator</span><span class="o">.</span><span class="n">template_fields</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">kwargs</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">else</span><span class="p">:</span>
- <span class="nb">object</span><span class="o">.</span><span class="n">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span></div>
-
-
-
+ <span class="nb">object</span><span class="o">.</span><span class="fm">__setattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -315,7 +376,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
[14/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/hive_operator.html
----------------------------------------------------------------------
diff --git a/_modules/hive_operator.html b/_modules/hive_operator.html
index 4d8c62c..b0e3265 100644
--- a/_modules/hive_operator.html
+++ b/_modules/hive_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>hive_operator</li>
+
+ <li>hive_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -150,14 +180,12 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">re</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.operator_helpers</span> <span class="kn">import</span> <span class="n">context_to_airflow_vars</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveCliHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.operator_helpers</span> <span class="k">import</span> <span class="n">context_to_airflow_vars</span>
<div class="viewcode-block" id="HiveOperator"><a class="viewcode-back" href="../code.html#airflow.operators.HiveOperator">[docs]</a><span class="k">class</span> <span class="nc">HiveOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -196,21 +224,21 @@
<span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span>
<span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
<span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
- <span class="n">hiveconf_jinja_translate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">script_begin_tag</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">run_as_owner</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">mapred_queue</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">mapred_queue_priority</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">mapred_job_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">hiveconf_jinja_translate</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">script_begin_tag</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">run_as_owner</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">mapred_queue</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mapred_queue_priority</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mapred_job_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HiveOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HiveOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hiveconf_jinja_translate</span> <span class="o">=</span> <span class="n">hiveconf_jinja_translate</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">script_begin_tag</span> <span class="o">=</span> <span class="n">script_begin_tag</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="n">run_as_owner</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dag</span><span class="o">.</span><span class="n">owner</span>
@@ -234,7 +262,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">hql</span> <span class="o">=</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">script_begin_tag</span><span class="p">)[</span><span class="mi">1</span><span class="p">:])</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
<span class="n">hive_conf</span><span class="o">=</span><span class="n">context_to_airflow_vars</span><span class="p">(</span><span class="n">context</span><span class="p">))</span>
@@ -248,6 +276,9 @@
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -280,7 +311,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/hive_to_druid.html
----------------------------------------------------------------------
diff --git a/_modules/hive_to_druid.html b/_modules/hive_to_druid.html
index d85bfb3..35dc4ff 100644
--- a/_modules/hive_to_druid.html
+++ b/_modules/hive_to_druid.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>hive_to_druid</li>
+
+ <li>hive_to_druid</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -150,13 +180,13 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveCliHook</span><span class="p">,</span> <span class="n">HiveMetastoreHook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.druid_hook</span> <span class="k">import</span> <span class="n">DruidHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span><span class="p">,</span> <span class="n">HiveMetastoreHook</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.druid_hook</span> <span class="kn">import</span> <span class="n">DruidHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="n">LOAD_CHECK_INTERVAL</span> <span class="o">=</span> <span class="mi">5</span>
+<span class="n">DEFAULT_TARGET_PARTITION_SIZE</span> <span class="o">=</span> <span class="mi">5000000</span>
<div class="viewcode-block" id="HiveToDruidTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.HiveToDruidTransfer">[docs]</a><span class="k">class</span> <span class="nc">HiveToDruidTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -189,7 +219,6 @@
<span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'intervals'</span><span class="p">)</span>
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
- <span class="c1">#ui_color = '#a0e08c'</span>
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
@@ -197,22 +226,26 @@
<span class="n">sql</span><span class="p">,</span>
<span class="n">druid_datasource</span><span class="p">,</span>
<span class="n">ts_dim</span><span class="p">,</span>
- <span class="n">metric_spec</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">metric_spec</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
<span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="s1">'druid_ingest_default'</span><span class="p">,</span>
<span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">,</span>
- <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">intervals</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">num_shards</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span>
<span class="n">target_partition_size</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span>
+ <span class="n">query_granularity</span><span class="o">=</span><span class="s2">"NONE"</span><span class="p">,</span>
+ <span class="n">segment_granularity</span><span class="o">=</span><span class="s2">"DAY"</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HiveToDruidTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HiveToDruidTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">druid_datasource</span> <span class="o">=</span> <span class="n">druid_datasource</span>
<span class="bp">self</span><span class="o">.</span><span class="n">ts_dim</span> <span class="o">=</span> <span class="n">ts_dim</span>
<span class="bp">self</span><span class="o">.</span><span class="n">intervals</span> <span class="o">=</span> <span class="n">intervals</span> <span class="ow">or</span> <span class="p">[</span><span class="s1">'{{ ds }}/{{ tomorrow_ds }}'</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">num_shards</span> <span class="o">=</span> <span class="n">num_shards</span>
<span class="bp">self</span><span class="o">.</span><span class="n">target_partition_size</span> <span class="o">=</span> <span class="n">target_partition_size</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">query_granularity</span> <span class="o">=</span> <span class="n">query_granularity</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">segment_granularity</span> <span class="o">=</span> <span class="n">segment_granularity</span>
<span class="bp">self</span><span class="o">.</span><span class="n">metric_spec</span> <span class="o">=</span> <span class="n">metric_spec</span> <span class="ow">or</span> <span class="p">[{</span>
<span class="s2">"name"</span><span class="p">:</span> <span class="s2">"count"</span><span class="p">,</span>
<span class="s2">"type"</span><span class="p">:</span> <span class="s2">"count"</span><span class="p">}]</span>
@@ -223,28 +256,30 @@
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from Hive"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from Hive"</span><span class="p">)</span>
<span class="n">hive_table</span> <span class="o">=</span> <span class="s1">'druid.'</span> <span class="o">+</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task_instance_key_str'</span><span class="p">]</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="s1">'_'</span><span class="p">)</span>
<span class="n">sql</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">';'</span><span class="p">)</span>
<span class="n">hql</span> <span class="o">=</span> <span class="s2">"""</span><span class="se">\</span>
-<span class="s2"> set mapred.output.compress=false;</span>
-<span class="s2"> set hive.exec.compress.output=false;</span>
-<span class="s2"> DROP TABLE IF EXISTS {hive_table};</span>
-<span class="s2"> CREATE TABLE {hive_table}</span>
+<span class="s2"> SET mapred.output.compress=false;</span>
+<span class="s2"> SET hive.exec.compress.output=false;</span>
+<span class="s2"> DROP TABLE IF EXISTS </span><span class="si">{hive_table}</span><span class="s2">;</span>
+<span class="s2"> CREATE TABLE </span><span class="si">{hive_table}</span><span class="s2"></span>
<span class="s2"> ROW FORMAT DELIMITED FIELDS TERMINATED BY '</span><span class="se">\t</span><span class="s2">'</span>
<span class="s2"> STORED AS TEXTFILE</span>
<span class="s2"> TBLPROPERTIES ('serialization.null.format' = '')</span>
<span class="s2"> AS</span>
-<span class="s2"> {sql}</span>
+<span class="s2"> </span><span class="si">{sql}</span><span class="s2"></span>
<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command:</span><span class="se">\n</span><span class="s2"> {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hql</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command:</span><span class="se">\n</span><span class="s2"> </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">hql</span><span class="p">)</span>
<span class="n">hive</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
- <span class="n">t</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span>
+ <span class="c1"># Get the Hive table and extract the columns</span>
+ <span class="n">t</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span>
<span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="n">col</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">t</span><span class="o">.</span><span class="n">sd</span><span class="o">.</span><span class="n">cols</span><span class="p">]</span>
+ <span class="c1"># Get the path on hdfs</span>
<span class="n">hdfs_uri</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span><span class="o">.</span><span class="n">sd</span><span class="o">.</span><span class="n">location</span>
<span class="n">pos</span> <span class="o">=</span> <span class="n">hdfs_uri</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">)</span>
<span class="n">static_path</span> <span class="o">=</span> <span class="n">hdfs_uri</span><span class="p">[</span><span class="n">pos</span><span class="p">:]</span>
@@ -252,26 +287,113 @@
<span class="n">schema</span><span class="p">,</span> <span class="n">table</span> <span class="o">=</span> <span class="n">hive_table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
<span class="n">druid</span> <span class="o">=</span> <span class="n">DruidHook</span><span class="p">(</span><span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into Druid"</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"HDFS path: "</span> <span class="o">+</span> <span class="n">static_path</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
- <span class="n">druid</span><span class="o">.</span><span class="n">load_from_hdfs</span><span class="p">(</span>
- <span class="n">datasource</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_datasource</span><span class="p">,</span>
- <span class="n">intervals</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">intervals</span><span class="p">,</span>
- <span class="n">static_path</span><span class="o">=</span><span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">ts_dim</span><span class="p">,</span>
- <span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> <span class="n">num_shards</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">target_partition_size</span><span class="p">,</span>
- <span class="n">metric_spec</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metric_spec</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hadoop_dependency_coordinates</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Load seems to have succeeded!"</span><span class="p">)</span>
+ <span class="n">index_spec</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">construct_ingest_query</span><span class="p">(</span>
+ <span class="n">static_path</span><span class="o">=</span><span class="n">static_path</span><span class="p">,</span>
+ <span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span>
+ <span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into Druid, hdfs path: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">static_path</span><span class="p">)</span>
+
+ <span class="n">druid</span><span class="o">.</span><span class="n">submit_indexing_job</span><span class="p">(</span><span class="n">index_spec</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Load seems to have succeeded!"</span><span class="p">)</span>
<span class="k">finally</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s2">"Cleaning up by dropping the temp "</span>
- <span class="s2">"Hive table {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hive_table</span><span class="p">))</span>
- <span class="n">hql</span> <span class="o">=</span> <span class="s2">"DROP TABLE IF EXISTS {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span>
- <span class="n">hive</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Cleaning up by dropping the temp Hive table </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
+ <span class="n">hive_table</span>
+ <span class="p">)</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"DROP TABLE IF EXISTS </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+
+<div class="viewcode-block" id="HiveToDruidTransfer.construct_ingest_query"><a class="viewcode-back" href="../code.html#airflow.operators.HiveToDruidTransfer.construct_ingest_query">[docs]</a> <span class="k">def</span> <span class="nf">construct_ingest_query</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">columns</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Builds an ingest query for an HDFS TSV load.</span>
+
+<span class="sd"> :param static_path: The path on hdfs where the data is</span>
+<span class="sd"> :type static_path: str</span>
+<span class="sd"> :param columns: List of all the columns that are available</span>
+<span class="sd"> :type columns: list</span>
+<span class="sd"> """</span>
+
+ <span class="c1"># backward compatibilty for num_shards, but target_partition_size is the default setting</span>
+ <span class="c1"># and overwrites the num_shards</span>
+ <span class="n">num_shards</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_shards</span>
+ <span class="n">target_partition_size</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">target_partition_size</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">target_partition_size</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">num_shards</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
+ <span class="n">target_partition_size</span> <span class="o">=</span> <span class="n">DEFAULT_TARGET_PARTITION_SIZE</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">num_shards</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
+
+ <span class="n">metric_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">m</span><span class="p">[</span><span class="s1">'fieldName'</span><span class="p">]</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">metric_spec</span> <span class="k">if</span> <span class="n">m</span><span class="p">[</span><span class="s1">'type'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'count'</span><span class="p">]</span>
+
+ <span class="c1"># Take all the columns, which are not the time dimension or a metric, as the dimension columns</span>
+ <span class="n">dimensions</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">columns</span> <span class="k">if</span> <span class="n">c</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">metric_names</span> <span class="ow">and</span> <span class="n">c</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ts_dim</span><span class="p">]</span>
+
+ <span class="n">ingest_query_dict</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"index_hadoop"</span><span class="p">,</span>
+ <span class="s2">"spec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"dataSchema"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"metricsSpec"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">metric_spec</span><span class="p">,</span>
+ <span class="s2">"granularitySpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"queryGranularity"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">query_granularity</span><span class="p">,</span>
+ <span class="s2">"intervals"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">intervals</span><span class="p">,</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"uniform"</span><span class="p">,</span>
+ <span class="s2">"segmentGranularity"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">segment_granularity</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="s2">"parser"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"string"</span><span class="p">,</span>
+ <span class="s2">"parseSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"columns"</span><span class="p">:</span> <span class="n">columns</span><span class="p">,</span>
+ <span class="s2">"dimensionsSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"dimensionExclusions"</span><span class="p">:</span> <span class="p">[],</span>
+ <span class="s2">"dimensions"</span><span class="p">:</span> <span class="n">dimensions</span><span class="p">,</span> <span class="c1"># list of names</span>
+ <span class="s2">"spatialDimensions"</span><span class="p">:</span> <span class="p">[]</span>
+ <span class="p">},</span>
+ <span class="s2">"timestampSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"column"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">ts_dim</span><span class="p">,</span>
+ <span class="s2">"format"</span><span class="p">:</span> <span class="s2">"auto"</span>
+ <span class="p">},</span>
+ <span class="s2">"format"</span><span class="p">:</span> <span class="s2">"tsv"</span>
+ <span class="p">}</span>
+ <span class="p">},</span>
+ <span class="s2">"dataSource"</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">druid_datasource</span>
+ <span class="p">},</span>
+ <span class="s2">"tuningConfig"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hadoop"</span><span class="p">,</span>
+ <span class="s2">"jobProperties"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"mapreduce.job.user.classpath.first"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
+ <span class="s2">"mapreduce.map.output.compress"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
+ <span class="s2">"mapreduce.output.fileoutputformat.compress"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="s2">"partitionsSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hashed"</span><span class="p">,</span>
+ <span class="s2">"targetPartitionSize"</span><span class="p">:</span> <span class="n">target_partition_size</span><span class="p">,</span>
+ <span class="s2">"numShards"</span><span class="p">:</span> <span class="n">num_shards</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="p">},</span>
+ <span class="s2">"ioConfig"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"inputSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"paths"</span><span class="p">:</span> <span class="n">static_path</span><span class="p">,</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"static"</span>
+ <span class="p">},</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hadoop"</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">hadoop_dependency_coordinates</span><span class="p">:</span>
+ <span class="n">ingest_query_dict</span><span class="p">[</span><span class="s1">'hadoopDependencyCoordinates'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hadoop_dependency_coordinates</span>
+
+ <span class="k">return</span> <span class="n">ingest_query_dict</span></div></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -304,7 +426,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/hive_to_mysql.html
----------------------------------------------------------------------
diff --git a/_modules/hive_to_mysql.html b/_modules/hive_to_mysql.html
index fd8b342..6262205 100644
--- a/_modules/hive_to_mysql.html
+++ b/_modules/hive_to_mysql.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>hive_to_mysql</li>
+
+ <li>hive_to_mysql</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -150,15 +180,12 @@
<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="k">import</span> <span class="n">HiveServer2Hook</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.mysql_hook</span> <span class="k">import</span> <span class="n">MySqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
-<span class="kn">from</span> <span class="nn">airflow.hooks.hive_hooks</span> <span class="kn">import</span> <span class="n">HiveServer2Hook</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.mysql_hook</span> <span class="kn">import</span> <span class="n">MySqlHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
-
-<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">NamedTemporaryFile</span>
<div class="viewcode-block" id="HiveToMySqlTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.HiveToMySqlTransfer">[docs]</a><span class="k">class</span> <span class="nc">HiveToMySqlTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -204,11 +231,11 @@
<span class="n">mysql_table</span><span class="p">,</span>
<span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="s1">'hiveserver2_default'</span><span class="p">,</span>
<span class="n">mysql_conn_id</span><span class="o">=</span><span class="s1">'mysql_default'</span><span class="p">,</span>
- <span class="n">mysql_preoperator</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">mysql_postoperator</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">bulk_load</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">mysql_preoperator</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mysql_postoperator</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">bulk_load</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HiveToMySqlTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HiveToMySqlTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mysql_table</span> <span class="o">=</span> <span class="n">mysql_table</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
@@ -219,22 +246,21 @@
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">hive</span> <span class="o">=</span> <span class="n">HiveServer2Hook</span><span class="p">(</span><span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from Hive"</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from Hive: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">bulk_load</span><span class="p">:</span>
<span class="n">tmpfile</span> <span class="o">=</span> <span class="n">NamedTemporaryFile</span><span class="p">()</span>
<span class="n">hive</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span> <span class="n">tmpfile</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span>
- <span class="n">lineterminator</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="n">output_header</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+ <span class="n">lineterminator</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="n">output_header</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">hive</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
<span class="n">mysql</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mysql_preoperator</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running MySQL preoperator"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running MySQL preoperator"</span><span class="p">)</span>
<span class="n">mysql</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_preoperator</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into MySQL"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into MySQL"</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">bulk_load</span><span class="p">:</span>
<span class="n">mysql</span><span class="o">.</span><span class="n">bulk_load</span><span class="p">(</span><span class="n">table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_table</span><span class="p">,</span> <span class="n">tmp_file</span><span class="o">=</span><span class="n">tmpfile</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
@@ -243,13 +269,16 @@
<span class="n">mysql</span><span class="o">.</span><span class="n">insert_rows</span><span class="p">(</span><span class="n">table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_table</span><span class="p">,</span> <span class="n">rows</span><span class="o">=</span><span class="n">results</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mysql_postoperator</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running MySQL postoperator"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running MySQL postoperator"</span><span class="p">)</span>
<span class="n">mysql</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_postoperator</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -282,7 +311,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/http_hook.html
----------------------------------------------------------------------
diff --git a/_modules/http_hook.html b/_modules/http_hook.html
index 5458ec1..ec71e54 100644
--- a/_modules/http_hook.html
+++ b/_modules/http_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -177,7 +182,6 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">requests</span>
@@ -201,9 +205,13 @@
<span class="sd"> """</span>
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span><span class="p">)</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span>
- <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'http'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="s1">'http://'</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span>
+
+ <span class="k">if</span> <span class="s2">"://"</span> <span class="ow">in</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="c1"># schema defaults to HTTP</span>
+ <span class="n">schema</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span> <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span> <span class="k">else</span> <span class="s2">"http"</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">schema</span> <span class="o">+</span> <span class="s2">"://"</span> <span class="o">+</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span>
<span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"/"</span>
@@ -230,6 +238,11 @@
<span class="n">url</span><span class="p">,</span>
<span class="n">params</span><span class="o">=</span><span class="n">data</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'HEAD'</span><span class="p">:</span>
+ <span class="c1"># HEAD doesn't use params</span>
+ <span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
+ <span class="n">url</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># Others use data</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
@@ -238,7 +251,7 @@
<span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
<span class="n">prepped_request</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">prepare_request</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Sending '"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">+</span> <span class="s2">"' to url: "</span> <span class="o">+</span> <span class="n">url</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Sending '</span><span class="si">%s</span><span class="s2">' to url: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="n">url</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_and_check</span><span class="p">(</span><span class="n">session</span><span class="p">,</span> <span class="n">prepped_request</span><span class="p">,</span> <span class="n">extra_options</span><span class="p">)</span></div>
<div class="viewcode-block" id="HttpHook.run_and_check"><a class="viewcode-back" href="../code.html#airflow.hooks.HttpHook.run_and_check">[docs]</a> <span class="k">def</span> <span class="nf">run_and_check</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">session</span><span class="p">,</span> <span class="n">prepped_request</span><span class="p">,</span> <span class="n">extra_options</span><span class="p">):</span>
@@ -263,13 +276,13 @@
<span class="c1"># Tried rewrapping, but not supported. This way, it's possible</span>
<span class="c1"># to get reason and code for failure by checking first 3 chars</span>
<span class="c1"># for the code, or do a split on ':'</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"HTTP error: "</span> <span class="o">+</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">!=</span> <span class="s1">'GET'</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"HTTP error: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'GET'</span><span class="p">,</span> <span class="s1">'HEAD'</span><span class="p">):</span>
<span class="c1"># The sensor uses GET, so this prevents filling up the log</span>
<span class="c1"># with the body every time the GET 'misses'.</span>
<span class="c1"># That's ok to do, because GETs should be repeatable and</span>
<span class="c1"># all data should be visible in the log (no post data)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">)</span><span class="o">+</span><span class="s2">":"</span><span class="o">+</span><span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
<span class="k">return</span> <span class="n">response</span></div></div>
</pre></div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/http_operator.html
----------------------------------------------------------------------
diff --git a/_modules/http_operator.html b/_modules/http_operator.html
index a22b710..8a42e96 100644
--- a/_modules/http_operator.html
+++ b/_modules/http_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -176,8 +181,6 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-
<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
<span class="kn">from</span> <span class="nn">airflow.hooks.http_hook</span> <span class="k">import</span> <span class="n">HttpHook</span>
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
@@ -209,7 +212,7 @@
<span class="sd"> depends on the option that's being modified.</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span><span class="s1">'data'</span><span class="p">,)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">'data'</span><span class="p">,)</span>
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
<span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f4a460'</span>
@@ -227,7 +230,7 @@
<span class="sd"> If xcom_push is True, response of an HTTP request will also</span>
<span class="sd"> be pushed to an XCom.</span>
<span class="sd"> """</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SimpleHttpOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SimpleHttpOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="n">method</span>
<span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
@@ -239,7 +242,9 @@
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="n">http</span> <span class="o">=</span> <span class="n">HttpHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Calling HTTP method"</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Calling HTTP method"</span><span class="p">)</span>
+
<span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">,</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/index.html
----------------------------------------------------------------------
diff --git a/_modules/index.html b/_modules/index.html
index b1faeb5..6bce09a 100644
--- a/_modules/index.html
+++ b/_modules/index.html
@@ -13,6 +13,8 @@
+
+
@@ -79,7 +81,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -159,32 +164,72 @@
<div itemprop="articleBody">
<h1>All modules for which code is available</h1>
-<ul><li><a href="airflow/contrib/executors/mesos_executor.html">airflow.contrib.executors.mesos_executor</a></li>
+<ul><li><a href="S3_hook.html">S3_hook</a></li>
+<li><a href="airflow/contrib/executors/mesos_executor.html">airflow.contrib.executors.mesos_executor</a></li>
+<li><a href="airflow/contrib/hooks/bigquery_hook.html">airflow.contrib.hooks.bigquery_hook</a></li>
+<li><a href="airflow/contrib/hooks/datastore_hook.html">airflow.contrib.hooks.datastore_hook</a></li>
+<li><a href="airflow/contrib/hooks/emr_hook.html">airflow.contrib.hooks.emr_hook</a></li>
+<li><a href="airflow/contrib/hooks/gcp_dataflow_hook.html">airflow.contrib.hooks.gcp_dataflow_hook</a></li>
+<li><a href="airflow/contrib/hooks/gcp_mlengine_hook.html">airflow.contrib.hooks.gcp_mlengine_hook</a></li>
+<li><a href="airflow/contrib/hooks/gcs_hook.html">airflow.contrib.hooks.gcs_hook</a></li>
+<li><a href="airflow/contrib/hooks/wasb_hook.html">airflow.contrib.hooks.wasb_hook</a></li>
+<li><a href="airflow/contrib/operators/bigquery_check_operator.html">airflow.contrib.operators.bigquery_check_operator</a></li>
+<li><a href="airflow/contrib/operators/bigquery_operator.html">airflow.contrib.operators.bigquery_operator</a></li>
+<li><a href="airflow/contrib/operators/bigquery_to_bigquery.html">airflow.contrib.operators.bigquery_to_bigquery</a></li>
+<li><a href="airflow/contrib/operators/bigquery_to_gcs.html">airflow.contrib.operators.bigquery_to_gcs</a></li>
+<li><a href="airflow/contrib/operators/databricks_operator.html">airflow.contrib.operators.databricks_operator</a></li>
+<li><a href="airflow/contrib/operators/dataflow_operator.html">airflow.contrib.operators.dataflow_operator</a></li>
+<li><a href="airflow/contrib/operators/dataproc_operator.html">airflow.contrib.operators.dataproc_operator</a></li>
+<li><a href="airflow/contrib/operators/ecs_operator.html">airflow.contrib.operators.ecs_operator</a></li>
+<li><a href="airflow/contrib/operators/emr_add_steps_operator.html">airflow.contrib.operators.emr_add_steps_operator</a></li>
+<li><a href="airflow/contrib/operators/emr_create_job_flow_operator.html">airflow.contrib.operators.emr_create_job_flow_operator</a></li>
+<li><a href="airflow/contrib/operators/emr_terminate_job_flow_operator.html">airflow.contrib.operators.emr_terminate_job_flow_operator</a></li>
+<li><a href="airflow/contrib/operators/file_to_wasb.html">airflow.contrib.operators.file_to_wasb</a></li>
+<li><a href="airflow/contrib/operators/gcs_download_operator.html">airflow.contrib.operators.gcs_download_operator</a></li>
+<li><a href="airflow/contrib/operators/gcs_to_bq.html">airflow.contrib.operators.gcs_to_bq</a></li>
<li><a href="airflow/contrib/operators/hipchat_operator.html">airflow.contrib.operators.hipchat_operator</a></li>
+<li><a href="airflow/contrib/operators/mlengine_operator.html">airflow.contrib.operators.mlengine_operator</a></li>
+<li><a href="airflow/contrib/sensors/wasb_sensor.html">airflow.contrib.sensors.wasb_sensor</a></li>
<li><a href="airflow/executors/local_executor.html">airflow.executors.local_executor</a></li>
<li><a href="airflow/executors/sequential_executor.html">airflow.executors.sequential_executor</a></li>
+<li><a href="airflow/hooks/S3_hook.html">airflow.hooks.S3_hook</a></li>
<li><a href="airflow/macros.html">airflow.macros</a></li>
<ul><li><a href="airflow/macros/hive.html">airflow.macros.hive</a></li>
</ul><li><a href="airflow/models.html">airflow.models</a></li>
+<li><a href="airflow/operators/docker_operator.html">airflow.operators.docker_operator</a></li>
+<li><a href="airflow/operators/redshift_to_s3_operator.html">airflow.operators.redshift_to_s3_operator</a></li>
+<li><a href="airflow/operators/s3_file_transform_operator.html">airflow.operators.s3_file_transform_operator</a></li>
+<li><a href="airflow/operators/s3_to_hive_operator.html">airflow.operators.s3_to_hive_operator</a></li>
<li><a href="airflow/operators/sensors.html">airflow.operators.sensors</a></li>
<li><a href="bash_operator.html">bash_operator</a></li>
+<li><a href="bigquery_hook.html">bigquery_hook</a></li>
<li><a href="dagrun_operator.html">dagrun_operator</a></li>
<li><a href="dbapi_hook.html">dbapi_hook</a></li>
+<li><a href="druid_hook.html">druid_hook</a></li>
<li><a href="dummy_operator.html">dummy_operator</a></li>
<li><a href="email_operator.html">email_operator</a></li>
<li><a href="ftp_hook.html">ftp_hook</a></li>
+<li><a href="gcs_hook.html">gcs_hook</a></li>
<li><a href="generic_transfer.html">generic_transfer</a></li>
+<li><a href="hive_hooks.html">hive_hooks</a></li>
+<li><a href="hive_operator.html">hive_operator</a></li>
+<li><a href="hive_to_druid.html">hive_to_druid</a></li>
+<li><a href="hive_to_mysql.html">hive_to_mysql</a></li>
<li><a href="http_hook.html">http_hook</a></li>
<li><a href="http_operator.html">http_operator</a></li>
<li><a href="mysql_hook.html">mysql_hook</a></li>
<li><a href="mysql_operator.html">mysql_operator</a></li>
+<li><a href="mysql_to_hive.html">mysql_to_hive</a></li>
+<li><a href="postgres_hook.html">postgres_hook</a></li>
+<li><a href="postgres_operator.html">postgres_operator</a></li>
<li><a href="presto_check_operator.html">presto_check_operator</a></li>
<li><a href="presto_hook.html">presto_hook</a></li>
<li><a href="python_operator.html">python_operator</a></li>
+<li><a href="qubole_operator.html">qubole_operator</a></li>
+<li><a href="s3_to_hive_operator.html">s3_to_hive_operator</a></li>
<li><a href="sensors.html">sensors</a></li>
+<li><a href="slack_operator.html">slack_operator</a></li>
<li><a href="sqlite_hook.html">sqlite_hook</a></li>
-<li><a href="ssh_execute_operator.html">ssh_execute_operator</a></li>
-<li><a href="ssh_hook.html">ssh_hook</a></li>
</ul>
</div>
[26/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/gcs_download_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/gcs_download_operator.html b/_modules/airflow/contrib/operators/gcs_download_operator.html
index 39186e2..de261e7 100644
--- a/_modules/airflow/contrib/operators/gcs_download_operator.html
+++ b/_modules/airflow/contrib/operators/gcs_download_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
<link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
<link rel="up" title="Module code" href="../../../index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../../../../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../../../../index.html">Docs</a> »</li>
-
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
<li><a href="../../../index.html">Module code</a> »</li>
-
- <li>airflow.contrib.operators.gcs_download_operator</li>
+
+ <li>airflow.contrib.operators.gcs_download_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,58 +181,52 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">sys</span>
-<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcs_hook</span> <span class="kn">import</span> <span class="n">GoogleCloudStorageHook</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcs_hook</span> <span class="k">import</span> <span class="n">GoogleCloudStorageHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
-<div class="viewcode-block" id="GoogleCloudStorageDownloadOperator"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageDownloadOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+<div class="viewcode-block" id="GoogleCloudStorageDownloadOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageDownloadOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Downloads a file from Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The Google cloud storage bucket where the object is.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The name of the object to download in the Google cloud</span>
+<span class="sd"> storage bucket.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param filename: The file path on the local file system (where the</span>
+<span class="sd"> operator is being executed) that the file should be downloaded to.</span>
+<span class="sd"> If false, the downloaded data will not be stored on the local file</span>
+<span class="sd"> system.</span>
+<span class="sd"> :type filename: string</span>
+<span class="sd"> :param store_to_xcom_key: If this param is set, the operator will push</span>
+<span class="sd"> the contents of the downloaded file to XCom with the key set in this</span>
+<span class="sd"> parameter. If false, the downloaded data will not be pushed to XCom.</span>
+<span class="sd"> :type store_to_xcom_key: string</span>
+<span class="sd"> :param google_cloud_storage_conn_id: The connection ID to use when</span>
+<span class="sd"> connecting to Google cloud storage.</span>
+<span class="sd"> :type google_cloud_storage_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
+<span class="sd"> For this to work, the service account making the request must have domain-wide delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket'</span><span class="p">,</span><span class="s1">'object'</span><span class="p">,</span><span class="s1">'filename'</span><span class="p">,</span><span class="s1">'store_to_xcom_key'</span><span class="p">,)</span>
- <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket'</span><span class="p">,</span> <span class="s1">'object'</span><span class="p">,</span> <span class="s1">'filename'</span><span class="p">,</span> <span class="s1">'store_to_xcom_key'</span><span class="p">,)</span>
<span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f0eee4'</span>
<span class="nd">@apply_defaults</span>
- <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span>
- <span class="n">bucket</span><span class="p">,</span>
- <span class="nb">object</span><span class="p">,</span>
- <span class="n">filename</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">store_to_xcom_key</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
- <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_storage_default'</span><span class="p">,</span>
- <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="o">*</span><span class="n">args</span><span class="p">,</span>
- <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Create a new GoogleCloudStorageDownloadOperator.</span>
-
-<span class="sd"> :param bucket: The Google cloud storage bucket where the object is.</span>
-<span class="sd"> :type bucket: string</span>
-<span class="sd"> :param object: The name of the object to download in the Google cloud</span>
-<span class="sd"> storage bucket.</span>
-<span class="sd"> :type object: string</span>
-<span class="sd"> :param filename: The file path on the local file system (where the</span>
-<span class="sd"> operator is being executed) that the file should be downloaded to.</span>
-<span class="sd"> If false, the downloaded data will not be stored on the local file</span>
-<span class="sd"> system.</span>
-<span class="sd"> :type filename: string</span>
-<span class="sd"> :param store_to_xcom_key: If this param is set, the operator will push</span>
-<span class="sd"> the contents of the downloaded file to XCom with the key set in this</span>
-<span class="sd"> parameter. If false, the downloaded data will not be pushed to XCom.</span>
-<span class="sd"> :type store_to_xcom_key: string</span>
-<span class="sd"> :param google_cloud_storage_conn_id: The connection ID to use when</span>
-<span class="sd"> connecting to Google cloud storage.</span>
-<span class="sd"> :type google_cloud_storage_conn_id: string</span>
-<span class="sd"> :param delegate_to: The account to impersonate, if any.</span>
-<span class="sd"> For this to work, the service account making the request must have domain-wide delegation enabled.</span>
-<span class="sd"> :type delegate_to: string</span>
-<span class="sd"> """</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageDownloadOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">bucket</span><span class="p">,</span>
+ <span class="nb">object</span><span class="p">,</span>
+ <span class="n">filename</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">store_to_xcom_key</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_storage_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageDownloadOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bucket</span> <span class="o">=</span> <span class="n">bucket</span>
<span class="bp">self</span><span class="o">.</span><span class="n">object</span> <span class="o">=</span> <span class="nb">object</span>
<span class="bp">self</span><span class="o">.</span><span class="n">filename</span> <span class="o">=</span> <span class="n">filename</span>
@@ -211,7 +235,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing download: </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">object</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing download: </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">object</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
<span class="n">hook</span> <span class="o">=</span> <span class="n">GoogleCloudStorageHook</span><span class="p">(</span><span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span>
<span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
<span class="n">file_bytes</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">download</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">object</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
@@ -220,10 +244,13 @@
<span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">xcom_push</span><span class="p">(</span><span class="n">key</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">store_to_xcom_key</span><span class="p">,</span> <span class="n">value</span><span class="o">=</span><span class="n">file_bytes</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s1">'The size of the downloaded file is too large to push to XCom!'</span><span class="p">)</span>
- <span class="k">print</span><span class="p">(</span><span class="n">file_bytes</span><span class="p">)</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">file_bytes</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -256,7 +283,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../../../../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/gcs_to_bq.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/gcs_to_bq.html b/_modules/airflow/contrib/operators/gcs_to_bq.html
new file mode 100644
index 0000000..1d61268
--- /dev/null
+++ b/_modules/airflow/contrib/operators/gcs_to_bq.html
@@ -0,0 +1,429 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.gcs_to_bq — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.gcs_to_bq</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.gcs_to_bq</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">import</span> <span class="nn">json</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gcs_hook</span> <span class="k">import</span> <span class="n">GoogleCloudStorageHook</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.bigquery_hook</span> <span class="k">import</span> <span class="n">BigQueryHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="GoogleCloudStorageToBigQueryOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageToBigQueryOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads files from Google cloud storage into BigQuery.</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket'</span><span class="p">,</span> <span class="s1">'source_objects'</span><span class="p">,</span>
+ <span class="s1">'schema_object'</span><span class="p">,</span> <span class="s1">'destination_project_dataset_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f0eee4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">bucket</span><span class="p">,</span>
+ <span class="n">source_objects</span><span class="p">,</span>
+ <span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">schema_fields</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">schema_object</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">source_format</span><span class="o">=</span><span class="s1">'CSV'</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="s1">'CREATE_IF_NEEDED'</span><span class="p">,</span>
+ <span class="n">skip_leading_rows</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
+ <span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_EMPTY'</span><span class="p">,</span>
+ <span class="n">field_delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">max_bad_records</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
+ <span class="n">quote_character</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">allow_quoted_newlines</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">allow_jagged_rows</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">max_id_key</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">bigquery_conn_id</span><span class="o">=</span><span class="s1">'bigquery_default'</span><span class="p">,</span>
+ <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_storage_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">schema_update_options</span><span class="o">=</span><span class="p">(),</span>
+ <span class="n">src_fmt_configs</span><span class="o">=</span><span class="p">{},</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> The schema to be used for the BigQuery table may be specified in one of</span>
+<span class="sd"> two ways. You may either directly pass the schema fields in, or you may</span>
+<span class="sd"> point the operator to a Google cloud storage object name. The object in</span>
+<span class="sd"> Google cloud storage must be a JSON file with the schema fields in it.</span>
+
+<span class="sd"> :param bucket: The bucket to load from.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param source_objects: List of Google cloud storage URIs to load from.</span>
+<span class="sd"> If source_format is 'DATASTORE_BACKUP', the list must only contain a single URI.</span>
+<span class="sd"> :type object: list</span>
+<span class="sd"> :param destination_project_dataset_table: The dotted (<project>.)<dataset>.<table></span>
+<span class="sd"> BigQuery table to load data into. If <project> is not included, project will</span>
+<span class="sd"> be the project defined in the connection json.</span>
+<span class="sd"> :type destination_project_dataset_table: string</span>
+<span class="sd"> :param schema_fields: If set, the schema field list as defined here:</span>
+<span class="sd"> https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.load</span>
+<span class="sd"> Should not be set when source_format is 'DATASTORE_BACKUP'.</span>
+<span class="sd"> :type schema_fields: list</span>
+<span class="sd"> :param schema_object: If set, a GCS object path pointing to a .json file that</span>
+<span class="sd"> contains the schema for the table.</span>
+<span class="sd"> :param schema_object: string</span>
+<span class="sd"> :param source_format: File format to export.</span>
+<span class="sd"> :type source_format: string</span>
+<span class="sd"> :param create_disposition: The create disposition if the table doesn't exist.</span>
+<span class="sd"> :type create_disposition: string</span>
+<span class="sd"> :param skip_leading_rows: Number of rows to skip when loading from a CSV.</span>
+<span class="sd"> :type skip_leading_rows: int</span>
+<span class="sd"> :param write_disposition: The write disposition if the table already exists.</span>
+<span class="sd"> :type write_disposition: string</span>
+<span class="sd"> :param field_delimiter: The delimiter to use when loading from a CSV.</span>
+<span class="sd"> :type field_delimiter: string</span>
+<span class="sd"> :param max_bad_records: The maximum number of bad records that BigQuery can</span>
+<span class="sd"> ignore when running the job.</span>
+<span class="sd"> :type max_bad_records: int</span>
+<span class="sd"> :param quote_character: The value that is used to quote data sections in a CSV file.</span>
+<span class="sd"> :type quote_character: string</span>
+<span class="sd"> :param allow_quoted_newlines: Whether to allow quoted newlines (true) or not (false).</span>
+<span class="sd"> :type allow_quoted_newlines: boolean</span>
+<span class="sd"> :param allow_jagged_rows: Accept rows that are missing trailing optional columns.</span>
+<span class="sd"> The missing values are treated as nulls. If false, records with missing trailing columns</span>
+<span class="sd"> are treated as bad records, and if there are too many bad records, an invalid error is</span>
+<span class="sd"> returned in the job result. Only applicable to CSV, ignored for other formats.</span>
+<span class="sd"> :type allow_jagged_rows: bool</span>
+<span class="sd"> :param max_id_key: If set, the name of a column in the BigQuery table</span>
+<span class="sd"> that's to be loaded. Thsi will be used to select the MAX value from</span>
+<span class="sd"> BigQuery after the load occurs. The results will be returned by the</span>
+<span class="sd"> execute() command, which in turn gets stored in XCom for future</span>
+<span class="sd"> operators to use. This can be helpful with incremental loads--during</span>
+<span class="sd"> future executions, you can pick up from the max ID.</span>
+<span class="sd"> :type max_id_key: string</span>
+<span class="sd"> :param bigquery_conn_id: Reference to a specific BigQuery hook.</span>
+<span class="sd"> :type bigquery_conn_id: string</span>
+<span class="sd"> :param google_cloud_storage_conn_id: Reference to a specific Google</span>
+<span class="sd"> cloud storage hook.</span>
+<span class="sd"> :type google_cloud_storage_conn_id: string</span>
+<span class="sd"> :param delegate_to: The account to impersonate, if any. For this to</span>
+<span class="sd"> work, the service account making the request must have domain-wide</span>
+<span class="sd"> delegation enabled.</span>
+<span class="sd"> :type delegate_to: string</span>
+<span class="sd"> :param schema_update_options: Allows the schema of the desitination</span>
+<span class="sd"> table to be updated as a side effect of the load job.</span>
+<span class="sd"> :type schema_update_options: list</span>
+<span class="sd"> :param src_fmt_configs: configure optional fields specific to the source format</span>
+<span class="sd"> :type src_fmt_configs: dict</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageToBigQueryOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="c1"># GCS config</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket</span> <span class="o">=</span> <span class="n">bucket</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_objects</span> <span class="o">=</span> <span class="n">source_objects</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema_object</span> <span class="o">=</span> <span class="n">schema_object</span>
+
+ <span class="c1"># BQ config</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span> <span class="o">=</span> <span class="n">destination_project_dataset_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema_fields</span> <span class="o">=</span> <span class="n">schema_fields</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_format</span> <span class="o">=</span> <span class="n">source_format</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create_disposition</span> <span class="o">=</span> <span class="n">create_disposition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">skip_leading_rows</span> <span class="o">=</span> <span class="n">skip_leading_rows</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span> <span class="o">=</span> <span class="n">write_disposition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">field_delimiter</span> <span class="o">=</span> <span class="n">field_delimiter</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">max_bad_records</span> <span class="o">=</span> <span class="n">max_bad_records</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">quote_character</span> <span class="o">=</span> <span class="n">quote_character</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">allow_quoted_newlines</span> <span class="o">=</span> <span class="n">allow_quoted_newlines</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">allow_jagged_rows</span> <span class="o">=</span> <span class="n">allow_jagged_rows</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">max_id_key</span> <span class="o">=</span> <span class="n">max_id_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span> <span class="o">=</span> <span class="n">bigquery_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">google_cloud_storage_conn_id</span> <span class="o">=</span> <span class="n">google_cloud_storage_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span> <span class="o">=</span> <span class="n">delegate_to</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema_update_options</span> <span class="o">=</span> <span class="n">schema_update_options</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">src_fmt_configs</span> <span class="o">=</span> <span class="n">src_fmt_configs</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">bq_hook</span> <span class="o">=</span> <span class="n">BigQueryHook</span><span class="p">(</span><span class="n">bigquery_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bigquery_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema_fields</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema_object</span> \
+ <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">source_format</span> <span class="o">!=</span> <span class="s1">'DATASTORE_BACKUP'</span><span class="p">:</span>
+ <span class="n">gcs_hook</span> <span class="o">=</span> <span class="n">GoogleCloudStorageHook</span><span class="p">(</span>
+ <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delegate_to</span><span class="p">)</span>
+ <span class="n">schema_fields</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">gcs_hook</span><span class="o">.</span><span class="n">download</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema_object</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">"utf-8"</span><span class="p">))</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">schema_fields</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">schema_fields</span>
+
+ <span class="n">source_uris</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'gs://</span><span class="si">{}</span><span class="s1">/</span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket</span><span class="p">,</span> <span class="n">source_object</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">source_object</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">source_objects</span><span class="p">]</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">bq_hook</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">run_load</span><span class="p">(</span>
+ <span class="n">destination_project_dataset_table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span><span class="p">,</span>
+ <span class="n">schema_fields</span><span class="o">=</span><span class="n">schema_fields</span><span class="p">,</span>
+ <span class="n">source_uris</span><span class="o">=</span><span class="n">source_uris</span><span class="p">,</span>
+ <span class="n">source_format</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">source_format</span><span class="p">,</span>
+ <span class="n">create_disposition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create_disposition</span><span class="p">,</span>
+ <span class="n">skip_leading_rows</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">skip_leading_rows</span><span class="p">,</span>
+ <span class="n">write_disposition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">write_disposition</span><span class="p">,</span>
+ <span class="n">field_delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_delimiter</span><span class="p">,</span>
+ <span class="n">max_bad_records</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">max_bad_records</span><span class="p">,</span>
+ <span class="n">quote_character</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">quote_character</span><span class="p">,</span>
+ <span class="n">allow_quoted_newlines</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">allow_quoted_newlines</span><span class="p">,</span>
+ <span class="n">allow_jagged_rows</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">allow_jagged_rows</span><span class="p">,</span>
+ <span class="n">schema_update_options</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">schema_update_options</span><span class="p">,</span>
+ <span class="n">src_fmt_configs</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">src_fmt_configs</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_id_key</span><span class="p">:</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s1">'SELECT MAX(</span><span class="si">{}</span><span class="s1">) FROM </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">max_id_key</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span><span class="p">))</span>
+ <span class="n">row</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
+ <span class="n">max_id</span> <span class="o">=</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="n">row</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">else</span> <span class="mi">0</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Loaded BQ data with max </span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1">=</span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_project_dataset_table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_id_key</span><span class="p">,</span> <span class="n">max_id</span>
+ <span class="p">)</span>
+ <span class="k">return</span> <span class="n">max_id</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/hipchat_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/hipchat_operator.html b/_modules/airflow/contrib/operators/hipchat_operator.html
index 194afad..1e04bf3 100644
--- a/_modules/airflow/contrib/operators/hipchat_operator.html
+++ b/_modules/airflow/contrib/operators/hipchat_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
@@ -181,7 +186,6 @@
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">json</span>
@@ -193,6 +197,7 @@
<span class="sd"> at https://www.hipchat.com/docs/apiv2. Before using any HipChat API operators you need</span>
<span class="sd"> to get an authentication token at https://www.hipchat.com/docs/apiv2/auth.</span>
<span class="sd"> In the future additional HipChat operators will be derived from this class as well.</span>
+
<span class="sd"> :param token: HipChat REST API authentication token</span>
<span class="sd"> :type token: str</span>
<span class="sd"> :param base_url: HipChat REST API base url.</span>
@@ -204,7 +209,7 @@
<span class="n">base_url</span><span class="o">=</span><span class="s1">'https://api.hipchat.com/v2'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HipChatAPIOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HipChatAPIOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
<span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">base_url</span>
<span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="kc">None</span>
@@ -230,8 +235,8 @@
<span class="s1">'Authorization'</span><span class="p">:</span> <span class="s1">'Bearer </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">token</span><span class="p">},</span>
<span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
<span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">>=</span> <span class="mi">400</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'HipChat API call failed: </span><span class="si">%s</span><span class="s1"> </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
- <span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'HipChat API call failed: </span><span class="si">%s</span><span class="s1"> </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'HipChat API call failed: </span><span class="si">%s</span><span class="s1"> </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span>
<span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">))</span></div>
@@ -240,6 +245,7 @@
<span class="sd">"""</span>
<span class="sd"> Send notification to a specific HipChat room.</span>
<span class="sd"> More info: https://www.hipchat.com/docs/apiv2/method/send_room_notification</span>
+
<span class="sd"> :param room_id: Room in which to send notification on HipChat</span>
<span class="sd"> :type room_id: str</span>
<span class="sd"> :param message: The message body</span>
@@ -262,7 +268,7 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">room_id</span><span class="p">,</span> <span class="n">message</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">HipChatAPISendRoomNotificationOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HipChatAPISendRoomNotificationOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">room_id</span> <span class="o">=</span> <span class="n">room_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span> <span class="o">=</span> <span class="n">message</span>
<span class="n">default_options</span> <span class="o">=</span> <span class="p">{</span>
[15/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/hive_hooks.html
----------------------------------------------------------------------
diff --git a/_modules/hive_hooks.html b/_modules/hive_hooks.html
index 926b54c..5063de3 100644
--- a/_modules/hive_hooks.html
+++ b/_modules/hive_hooks.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>hive_hooks</li>
+
+ <li>hive_hooks</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -152,32 +182,29 @@
<span class="c1"># limitations under the License.</span>
<span class="c1">#</span>
-<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
-<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">zip</span>
-<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="kn">import</span> <span class="nb">basestring</span>
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">print_function</span>
+<span class="kn">from</span> <span class="nn">six.moves</span> <span class="k">import</span> <span class="nb">zip</span>
+<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
-<span class="kn">import</span> <span class="nn">collections</span>
-<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="kn">as</span> <span class="nn">csv</span>
+<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="k">as</span> <span class="nn">csv</span>
<span class="kn">import</span> <span class="nn">itertools</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">import</span> <span class="nn">time</span>
-<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">NamedTemporaryFile</span>
<span class="kn">import</span> <span class="nn">hive_metastore</span>
-<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.helpers</span> <span class="kn">import</span> <span class="n">as_flattened_list</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="kn">import</span> <span class="n">TemporaryDirectory</span>
-<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">configuration</span>
-<span class="kn">import</span> <span class="nn">airflow.security.utils</span> <span class="kn">as</span> <span class="nn">utils</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.helpers</span> <span class="k">import</span> <span class="n">as_flattened_list</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="k">import</span> <span class="n">TemporaryDirectory</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">configuration</span>
+<span class="kn">import</span> <span class="nn">airflow.security.utils</span> <span class="k">as</span> <span class="nn">utils</span>
<span class="n">HIVE_QUEUE_PRIORITIES</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'VERY_HIGH'</span><span class="p">,</span> <span class="s1">'HIGH'</span><span class="p">,</span> <span class="s1">'NORMAL'</span><span class="p">,</span> <span class="s1">'LOW'</span><span class="p">,</span> <span class="s1">'VERY_LOW'</span><span class="p">]</span>
<div class="viewcode-block" id="HiveCliHook"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook">[docs]</a><span class="k">class</span> <span class="nc">HiveCliHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
-
<span class="sd">"""Simple wrapper around the hive CLI.</span>
<span class="sd"> It also supports the ``beeline``</span>
@@ -206,13 +233,13 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s2">"hive_cli_default"</span><span class="p">,</span>
- <span class="n">run_as</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">mapred_queue</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">mapred_queue_priority</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">mapred_job_name</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="n">run_as</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mapred_queue</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mapred_queue_priority</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">mapred_job_name</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_params</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'hive_cli_params'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">use_beeline</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'use_beeline'</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">use_beeline</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'use_beeline'</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">auth</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'auth'</span><span class="p">,</span> <span class="s1">'noSasl'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">conn</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="n">run_as</span>
@@ -222,7 +249,7 @@
<span class="k">if</span> <span class="n">mapred_queue_priority</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">HIVE_QUEUE_PRIORITIES</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
<span class="s2">"Invalid Mapred Queue Priority. Valid values are: "</span>
- <span class="s2">"{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">HIVE_QUEUE_PRIORITIES</span><span class="p">)))</span>
+ <span class="s2">"</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">HIVE_QUEUE_PRIORITIES</span><span class="p">)))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mapred_queue</span> <span class="o">=</span> <span class="n">mapred_queue</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mapred_queue_priority</span> <span class="o">=</span> <span class="n">mapred_queue_priority</span>
@@ -238,7 +265,7 @@
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">use_beeline</span><span class="p">:</span>
<span class="n">hive_bin</span> <span class="o">=</span> <span class="s1">'beeline'</span>
- <span class="n">jdbc_url</span> <span class="o">=</span> <span class="s2">"jdbc:hive2://{conn.host}:{conn.port}/{conn.schema}"</span>
+ <span class="n">jdbc_url</span> <span class="o">=</span> <span class="s2">"jdbc:hive2://</span><span class="si">{conn.host}</span><span class="s2">:</span><span class="si">{conn.port}</span><span class="s2">/</span><span class="si">{conn.schema}</span><span class="s2">"</span>
<span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span><span class="p">:</span>
<span class="n">template</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s1">'principal'</span><span class="p">,</span> <span class="s2">"hive/_HOST@EXAMPLE.COM"</span><span class="p">)</span>
@@ -248,11 +275,11 @@
<span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">""</span> <span class="c1"># noqa</span>
<span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'proxy_user'</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"login"</span> <span class="ow">and</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">:</span>
- <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">"hive.server2.proxy.user={0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">)</span>
+ <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">"hive.server2.proxy.user=</span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'proxy_user'</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"owner"</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span><span class="p">:</span>
- <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">"hive.server2.proxy.user={0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_as</span><span class="p">)</span>
+ <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">"hive.server2.proxy.user=</span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_as</span><span class="p">)</span>
- <span class="n">jdbc_url</span> <span class="o">+=</span> <span class="s2">";principal={template};{proxy_user}"</span>
+ <span class="n">jdbc_url</span> <span class="o">+=</span> <span class="s2">";principal=</span><span class="si">{template}</span><span class="s2">;</span><span class="si">{proxy_user}</span><span class="s2">"</span>
<span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">auth</span><span class="p">:</span>
<span class="n">jdbc_url</span> <span class="o">+=</span> <span class="s2">";auth="</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">auth</span>
@@ -286,13 +313,11 @@
<span class="k">if</span> <span class="ow">not</span> <span class="n">d</span><span class="p">:</span>
<span class="k">return</span> <span class="p">[]</span>
<span class="k">return</span> <span class="n">as_flattened_list</span><span class="p">(</span>
- <span class="n">itertools</span><span class="o">.</span><span class="n">izip</span><span class="p">(</span>
- <span class="p">[</span><span class="s2">"-hiveconf"</span><span class="p">]</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">d</span><span class="p">),</span>
- <span class="p">[</span><span class="s2">"{}={}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">d</span><span class="o">.</span><span class="n">items</span><span class="p">()]</span>
- <span class="p">)</span>
- <span class="p">)</span>
+ <span class="nb">zip</span><span class="p">([</span><span class="s2">"-hiveconf"</span><span class="p">]</span> <span class="o">*</span> <span class="nb">len</span><span class="p">(</span><span class="n">d</span><span class="p">),</span>
+ <span class="p">[</span><span class="s2">"</span><span class="si">{}</span><span class="s2">=</span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">d</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
+ <span class="p">)</span>
-<div class="viewcode-block" id="HiveCliHook.run_cli"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.run_cli">[docs]</a> <span class="k">def</span> <span class="nf">run_cli</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">hive_conf</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+<div class="viewcode-block" id="HiveCliHook.run_cli"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.run_cli">[docs]</a> <span class="k">def</span> <span class="nf">run_cli</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">hive_conf</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Run an hql statement using the hive cli. If hive_conf is specified</span>
<span class="sd"> it should be a dict and the entries will be set as key/value pairs</span>
@@ -313,7 +338,7 @@
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span> <span class="ow">or</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span>
<span class="k">if</span> <span class="n">schema</span><span class="p">:</span>
- <span class="n">hql</span> <span class="o">=</span> <span class="s2">"USE {schema};</span><span class="se">\n</span><span class="s2">{hql}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"USE </span><span class="si">{schema}</span><span class="s2">;</span><span class="se">\n</span><span class="si">{hql}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
<span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflow_hiveop_'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">:</span>
<span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
@@ -324,26 +349,26 @@
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mapred_queue</span><span class="p">:</span>
<span class="n">hive_conf_params</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span>
<span class="p">[</span><span class="s1">'-hiveconf'</span><span class="p">,</span>
- <span class="s1">'mapreduce.job.queuename={}'</span>
+ <span class="s1">'mapreduce.job.queuename=</span><span class="si">{}</span><span class="s1">'</span>
<span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mapred_queue</span><span class="p">)])</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mapred_queue_priority</span><span class="p">:</span>
<span class="n">hive_conf_params</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span>
<span class="p">[</span><span class="s1">'-hiveconf'</span><span class="p">,</span>
- <span class="s1">'mapreduce.job.priority={}'</span>
+ <span class="s1">'mapreduce.job.priority=</span><span class="si">{}</span><span class="s1">'</span>
<span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mapred_queue_priority</span><span class="p">)])</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mapred_job_name</span><span class="p">:</span>
<span class="n">hive_conf_params</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span>
<span class="p">[</span><span class="s1">'-hiveconf'</span><span class="p">,</span>
- <span class="s1">'mapred.job.name={}'</span>
+ <span class="s1">'mapred.job.name=</span><span class="si">{}</span><span class="s1">'</span>
<span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mapred_job_name</span><span class="p">)])</span>
<span class="n">hive_cmd</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">hive_conf_params</span><span class="p">)</span>
<span class="n">hive_cmd</span><span class="o">.</span><span class="n">extend</span><span class="p">([</span><span class="s1">'-f'</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">])</span>
<span class="k">if</span> <span class="n">verbose</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">" "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">hive_cmd</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">" "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">hive_cmd</span><span class="p">))</span>
<span class="n">sp</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
<span class="n">hive_cmd</span><span class="p">,</span>
<span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
@@ -351,13 +376,13 @@
<span class="n">cwd</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sp</span> <span class="o">=</span> <span class="n">sp</span>
<span class="n">stdout</span> <span class="o">=</span> <span class="s1">''</span>
- <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">sp</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">stdout</span> <span class="o">+=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'UTF-8'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">verbose</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'UTF-8'</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'UTF-8'</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="n">sp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">if</span> <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
@@ -388,36 +413,102 @@
<span class="k">for</span> <span class="n">query</span> <span class="ow">in</span> <span class="n">query_set</span><span class="p">:</span>
<span class="n">query_preview</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">split</span><span class="p">())[:</span><span class="mi">50</span><span class="p">]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Testing HQL [{0} (...)]"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">query_preview</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Testing HQL [</span><span class="si">%s</span><span class="s2"> (...)]"</span><span class="p">,</span> <span class="n">query_preview</span><span class="p">)</span>
<span class="k">if</span> <span class="n">query_set</span> <span class="o">==</span> <span class="n">insert</span><span class="p">:</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">other</span> <span class="o">+</span> <span class="s1">'; explain '</span> <span class="o">+</span> <span class="n">query</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">query</span> <span class="o">=</span> <span class="s1">'explain '</span> <span class="o">+</span> <span class="n">query</span>
<span class="k">try</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">except</span> <span class="n">AirflowException</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)[</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="n">error_loc</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="s1">'(\d+):(\d+)'</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
<span class="k">if</span> <span class="n">error_loc</span> <span class="ow">and</span> <span class="n">error_loc</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">isdigit</span><span class="p">():</span>
<span class="n">l</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">error_loc</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="n">begin</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">l</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">end</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">l</span><span class="o">+</span><span class="mi">3</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)))</span>
<span class="n">context</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)[</span><span class="n">begin</span><span class="p">:</span><span class="n">end</span><span class="p">])</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Context :</span><span class="se">\n</span><span class="s2"> {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">context</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Context :</span><span class="se">\n</span><span class="s2"> </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"SUCCESS"</span><span class="p">)</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"SUCCESS"</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="HiveCliHook.load_df"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.load_df">[docs]</a> <span class="k">def</span> <span class="nf">load_df</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">df</span><span class="p">,</span>
+ <span class="n">table</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf8'</span><span class="p">,</span>
+ <span class="n">pandas_kwargs</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a pandas DataFrame into hive.</span>
+
+<span class="sd"> Hive data types will be inferred if not passed but column names will</span>
+<span class="sd"> not be sanitized.</span>
+
+<span class="sd"> :param table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
+<span class="sd"> execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param field_dict: mapping from column name to hive data type</span>
+<span class="sd"> :type field_dict: dict</span>
+<span class="sd"> :param encoding: string encoding to use when writing DataFrame to file</span>
+<span class="sd"> :type encoding: str</span>
+<span class="sd"> :param pandas_kwargs: passed to DataFrame.to_csv</span>
+<span class="sd"> :type pandas_kwargs: dict</span>
+<span class="sd"> :param kwargs: passed to self.load_file</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">_infer_field_types_from_df</span><span class="p">(</span><span class="n">df</span><span class="p">):</span>
+ <span class="n">DTYPE_KIND_HIVE_TYPE</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'b'</span><span class="p">:</span> <span class="s1">'BOOLEAN'</span><span class="p">,</span> <span class="c1"># boolean</span>
+ <span class="s1">'i'</span><span class="p">:</span> <span class="s1">'BIGINT'</span><span class="p">,</span> <span class="c1"># signed integer</span>
+ <span class="s1">'u'</span><span class="p">:</span> <span class="s1">'BIGINT'</span><span class="p">,</span> <span class="c1"># unsigned integer</span>
+ <span class="s1">'f'</span><span class="p">:</span> <span class="s1">'DOUBLE'</span><span class="p">,</span> <span class="c1"># floating-point</span>
+ <span class="s1">'c'</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="c1"># complex floating-point</span>
+ <span class="s1">'O'</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="c1"># object</span>
+ <span class="s1">'S'</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="c1"># (byte-)string</span>
+ <span class="s1">'U'</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="c1"># Unicode</span>
+ <span class="s1">'V'</span><span class="p">:</span> <span class="s1">'STRING'</span> <span class="c1"># void</span>
+ <span class="p">}</span>
+
+ <span class="k">return</span> <span class="nb">dict</span><span class="p">((</span><span class="n">col</span><span class="p">,</span> <span class="n">DTYPE_KIND_HIVE_TYPE</span><span class="p">[</span><span class="n">dtype</span><span class="o">.</span><span class="n">kind</span><span class="p">])</span> <span class="k">for</span> <span class="n">col</span><span class="p">,</span> <span class="n">dtype</span> <span class="ow">in</span> <span class="n">df</span><span class="o">.</span><span class="n">dtypes</span><span class="o">.</span><span class="n">iteritems</span><span class="p">())</span>
+
+ <span class="k">if</span> <span class="n">pandas_kwargs</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">pandas_kwargs</span> <span class="o">=</span> <span class="p">{}</span>
+
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflow_hiveop_'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+
+ <span class="k">if</span> <span class="n">field_dict</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="p">(</span><span class="n">create</span> <span class="ow">or</span> <span class="n">recreate</span><span class="p">):</span>
+ <span class="n">field_dict</span> <span class="o">=</span> <span class="n">_infer_field_types_from_df</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
+
+ <span class="n">df</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="n">delimiter</span><span class="p">,</span> <span class="o">**</span><span class="n">pandas_kwargs</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span><span class="n">filepath</span><span class="o">=</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="n">table</span><span class="o">=</span><span class="n">table</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div>
<div class="viewcode-block" id="HiveCliHook.load_file"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.load_file">[docs]</a> <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">filepath</span><span class="p">,</span>
<span class="n">table</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="s2">","</span><span class="p">,</span>
- <span class="n">field_dict</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
- <span class="n">overwrite</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
- <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
- <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">overwrite</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
+ <span class="n">tblproperties</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Loads a local file into Hive</span>
@@ -428,53 +519,69 @@
<span class="sd"> stage the data into a temporary table before loading it into its</span>
<span class="sd"> final destination using a ``HiveOperator``.</span>
+<span class="sd"> :param filepath: local filepath of the file to load</span>
+<span class="sd"> :type filepath: str</span>
<span class="sd"> :param table: target Hive table, use dot notation to target a</span>
<span class="sd"> specific database</span>
<span class="sd"> :type table: str</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param field_dict: A dictionary of the fields name in the file</span>
+<span class="sd"> as keys and their Hive types as values</span>
+<span class="sd"> :type field_dict: dict</span>
<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
<span class="sd"> :type create: bool</span>
-<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
-<span class="sd"> execution</span>
-<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param overwrite: whether to overwrite the data in table or partition</span>
+<span class="sd"> :type overwrite: bool</span>
<span class="sd"> :param partition: target partition as a dict of partition columns</span>
<span class="sd"> and values</span>
<span class="sd"> :type partition: dict</span>
-<span class="sd"> :param delimiter: field delimiter in the file</span>
-<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
+<span class="sd"> execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param tblproperties: TBLPROPERTIES of the hive table being created</span>
+<span class="sd"> :type tblproperties: dict</span>
<span class="sd"> """</span>
<span class="n">hql</span> <span class="o">=</span> <span class="s1">''</span>
<span class="k">if</span> <span class="n">recreate</span><span class="p">:</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"DROP TABLE IF EXISTS {table};</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"DROP TABLE IF EXISTS </span><span class="si">{table}</span><span class="s2">;</span><span class="se">\n</span><span class="s2">"</span>
<span class="k">if</span> <span class="n">create</span> <span class="ow">or</span> <span class="n">recreate</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">field_dict</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"Must provide a field dict when creating a table"</span><span class="p">)</span>
<span class="n">fields</span> <span class="o">=</span> <span class="s2">",</span><span class="se">\n</span><span class="s2"> "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="p">[</span><span class="n">k</span> <span class="o">+</span> <span class="s1">' '</span> <span class="o">+</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">field_dict</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"CREATE TABLE IF NOT EXISTS {table} (</span><span class="se">\n</span><span class="s2">{fields})</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"CREATE TABLE IF NOT EXISTS </span><span class="si">{table}</span><span class="s2"> (</span><span class="se">\n</span><span class="si">{fields}</span><span class="s2">)</span><span class="se">\n</span><span class="s2">"</span>
<span class="k">if</span> <span class="n">partition</span><span class="p">:</span>
<span class="n">pfields</span> <span class="o">=</span> <span class="s2">",</span><span class="se">\n</span><span class="s2"> "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
<span class="p">[</span><span class="n">p</span> <span class="o">+</span> <span class="s2">" STRING"</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">partition</span><span class="p">])</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"PARTITIONED BY ({pfields})</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"PARTITIONED BY (</span><span class="si">{pfields}</span><span class="s2">)</span><span class="se">\n</span><span class="s2">"</span>
<span class="n">hql</span> <span class="o">+=</span> <span class="s2">"ROW FORMAT DELIMITED</span><span class="se">\n</span><span class="s2">"</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"FIELDS TERMINATED BY '{delimiter}'</span><span class="se">\n</span><span class="s2">"</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"STORED AS textfile;"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"FIELDS TERMINATED BY '</span><span class="si">{delimiter}</span><span class="s2">'</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"STORED AS textfile</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="k">if</span> <span class="n">tblproperties</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">tprops</span> <span class="o">=</span> <span class="s2">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="p">[</span><span class="s2">"'</span><span class="si">{0}</span><span class="s2">'='</span><span class="si">{1}</span><span class="s2">'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">tblproperties</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"TBLPROPERTIES(</span><span class="si">{tprops}</span><span class="s2">)</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">";"</span>
<span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
- <span class="n">hql</span> <span class="o">=</span> <span class="s2">"LOAD DATA LOCAL INPATH '{filepath}' "</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"LOAD DATA LOCAL INPATH '</span><span class="si">{filepath}</span><span class="s2">' "</span>
<span class="k">if</span> <span class="n">overwrite</span><span class="p">:</span>
<span class="n">hql</span> <span class="o">+=</span> <span class="s2">"OVERWRITE "</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"INTO TABLE {table} "</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"INTO TABLE </span><span class="si">{table}</span><span class="s2"> "</span>
<span class="k">if</span> <span class="n">partition</span><span class="p">:</span>
<span class="n">pvals</span> <span class="o">=</span> <span class="s2">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
- <span class="p">[</span><span class="s2">"{0}='{1}'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">partition</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
- <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"PARTITION ({pvals});"</span>
+ <span class="p">[</span><span class="s2">"</span><span class="si">{0}</span><span class="s2">='</span><span class="si">{1}</span><span class="s2">'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">partition</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"PARTITION (</span><span class="si">{pvals}</span><span class="s2">);"</span>
<span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span></div>
<span class="k">def</span> <span class="nf">kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'sp'</span><span class="p">):</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">poll</span><span class="p">()</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
- <span class="k">print</span><span class="p">(</span><span class="s2">"Killing the Hive job"</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">poll</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="nb">print</span><span class="p">(</span><span class="s2">"Killing the Hive job"</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">60</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">kill</span><span class="p">()</span></div>
@@ -491,21 +598,21 @@
<span class="k">def</span> <span class="nf">__getstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># This is for pickling to work despite the thirft hive client not</span>
<span class="c1"># being pickable</span>
- <span class="n">d</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">)</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="vm">__dict__</span><span class="p">)</span>
<span class="k">del</span> <span class="n">d</span><span class="p">[</span><span class="s1">'metastore'</span><span class="p">]</span>
<span class="k">return</span> <span class="n">d</span>
<span class="k">def</span> <span class="nf">__setstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">d</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">[</span><span class="s1">'metastore'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_metastore_client</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="vm">__dict__</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="vm">__dict__</span><span class="p">[</span><span class="s1">'metastore'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_metastore_client</span><span class="p">()</span>
<div class="viewcode-block" id="HiveMetastoreHook.get_metastore_client"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_metastore_client">[docs]</a> <span class="k">def</span> <span class="nf">get_metastore_client</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a Hive thrift client.</span>
<span class="sd"> """</span>
- <span class="kn">from</span> <span class="nn">thrift.transport</span> <span class="kn">import</span> <span class="n">TSocket</span><span class="p">,</span> <span class="n">TTransport</span>
- <span class="kn">from</span> <span class="nn">thrift.protocol</span> <span class="kn">import</span> <span class="n">TBinaryProtocol</span>
- <span class="kn">from</span> <span class="nn">hive_service</span> <span class="kn">import</span> <span class="n">ThriftHive</span>
+ <span class="kn">from</span> <span class="nn">thrift.transport</span> <span class="k">import</span> <span class="n">TSocket</span><span class="p">,</span> <span class="n">TTransport</span>
+ <span class="kn">from</span> <span class="nn">thrift.protocol</span> <span class="k">import</span> <span class="n">TBinaryProtocol</span>
+ <span class="kn">from</span> <span class="nn">hive_service</span> <span class="k">import</span> <span class="n">ThriftHive</span>
<span class="n">ms</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn</span>
<span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">ms</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'NOSASL'</span><span class="p">)</span>
<span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span><span class="p">:</span>
@@ -515,7 +622,7 @@
<span class="n">socket</span> <span class="o">=</span> <span class="n">TSocket</span><span class="o">.</span><span class="n">TSocket</span><span class="p">(</span><span class="n">ms</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">ms</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
<span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span> <span class="ow">and</span> <span class="n">auth_mechanism</span> <span class="o">==</span> <span class="s1">'GSSAPI'</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
- <span class="kn">import</span> <span class="nn">saslwrapper</span> <span class="kn">as</span> <span class="nn">sasl</span>
+ <span class="kn">import</span> <span class="nn">saslwrapper</span> <span class="k">as</span> <span class="nn">sasl</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">sasl</span>
@@ -526,7 +633,7 @@
<span class="n">sasl_client</span><span class="o">.</span><span class="n">init</span><span class="p">()</span>
<span class="k">return</span> <span class="n">sasl_client</span>
- <span class="kn">from</span> <span class="nn">thrift_sasl</span> <span class="kn">import</span> <span class="n">TSaslClientTransport</span>
+ <span class="kn">from</span> <span class="nn">thrift_sasl</span> <span class="k">import</span> <span class="n">TSaslClientTransport</span>
<span class="n">transport</span> <span class="o">=</span> <span class="n">TSaslClientTransport</span><span class="p">(</span><span class="n">sasl_factory</span><span class="p">,</span> <span class="s2">"GSSAPI"</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">transport</span> <span class="o">=</span> <span class="n">TTransport</span><span class="o">.</span><span class="n">TBufferedTransport</span><span class="p">(</span><span class="n">socket</span><span class="p">)</span>
@@ -561,9 +668,9 @@
<span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">if</span> <span class="n">partitions</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">return</span> <span class="kc">True</span>
<span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">False</span></div>
+ <span class="k">return</span> <span class="kc">False</span></div>
<div class="viewcode-block" id="HiveMetastoreHook.check_for_named_partition"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.check_for_named_partition">[docs]</a> <span class="k">def</span> <span class="nf">check_for_named_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition_name</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -588,9 +695,9 @@
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_partition_by_name</span><span class="p">(</span>
<span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition_name</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">return</span> <span class="kc">True</span>
<span class="k">except</span> <span class="n">hive_metastore</span><span class="o">.</span><span class="n">ttypes</span><span class="o">.</span><span class="n">NoSuchObjectException</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">return</span> <span class="kc">False</span>
<span class="k">finally</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
@@ -631,7 +738,7 @@
<span class="k">return</span> <span class="n">dbs</span></div>
<div class="viewcode-block" id="HiveMetastoreHook.get_partitions"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_partitions">[docs]</a> <span class="k">def</span> <span class="nf">get_partitions</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a list of all partitions in a table. Works only</span>
<span class="sd"> for tables with less than 32767 (java short max val).</span>
@@ -662,7 +769,7 @@
<span class="n">pnames</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">table</span><span class="o">.</span><span class="n">partitionKeys</span><span class="p">]</span>
<span class="k">return</span> <span class="p">[</span><span class="nb">dict</span><span class="p">(</span><span class="nb">zip</span><span class="p">(</span><span class="n">pnames</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">values</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">parts</span><span class="p">]</span></div>
-<div class="viewcode-block" id="HiveMetastoreHook.max_partition"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.max_partition">[docs]</a> <span class="k">def</span> <span class="nf">max_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="n">field</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+<div class="viewcode-block" id="HiveMetastoreHook.max_partition"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.max_partition">[docs]</a> <span class="k">def</span> <span class="nf">max_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="n">field</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns the maximum value for all partitions in a table. Works only</span>
<span class="sd"> for tables that have a single partition key. For subpartitioned</span>
@@ -675,7 +782,7 @@
<span class="sd"> """</span>
<span class="n">parts</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_partitions</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="nb">filter</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">parts</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">None</span>
+ <span class="k">return</span> <span class="kc">None</span>
<span class="k">elif</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">field</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">elif</span> <span class="ow">not</span> <span class="n">field</span><span class="p">:</span>
@@ -697,9 +804,9 @@
<span class="sd"> """</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">table_name</span><span class="p">,</span> <span class="n">db</span><span class="p">)</span>
- <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">return</span> <span class="kc">True</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
- <span class="k">return</span> <span class="bp">False</span></div></div>
+ <span class="k">return</span> <span class="kc">False</span></div></div>
<div class="viewcode-block" id="HiveServer2Hook"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveServer2Hook">[docs]</a><span class="k">class</span> <span class="nc">HiveServer2Hook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
@@ -712,33 +819,35 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="s1">'hiveserver2_default'</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span> <span class="o">=</span> <span class="n">hiveserver2_conn_id</span>
- <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">db</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
<span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'PLAIN'</span><span class="p">)</span>
- <span class="n">kerberos_service_name</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="n">kerberos_service_name</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span><span class="p">:</span>
<span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'GSSAPI'</span><span class="p">)</span>
<span class="n">kerberos_service_name</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'kerberos_service_name'</span><span class="p">,</span> <span class="s1">'hive'</span><span class="p">)</span>
<span class="c1"># impyla uses GSSAPI instead of KERBEROS as a auth_mechanism identifier</span>
<span class="k">if</span> <span class="n">auth_mechanism</span> <span class="o">==</span> <span class="s1">'KERBEROS'</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Detected deprecated 'KERBEROS' for authMechanism for </span><span class="si">%s</span><span class="s2">. Please use 'GSSAPI' instead"</span><span class="p">,</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
+ <span class="s2">"Detected deprecated 'KERBEROS' for authMechanism for </span><span class="si">%s</span><span class="s2">. Please use 'GSSAPI' instead"</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span>
+ <span class="p">)</span>
<span class="n">auth_mechanism</span> <span class="o">=</span> <span class="s1">'GSSAPI'</span>
- <span class="kn">from</span> <span class="nn">impala.dbapi</span> <span class="kn">import</span> <span class="n">connect</span>
+ <span class="kn">from</span> <span class="nn">impala.dbapi</span> <span class="k">import</span> <span class="n">connect</span>
<span class="k">return</span> <span class="n">connect</span><span class="p">(</span>
<span class="n">host</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
<span class="n">port</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">port</span><span class="p">,</span>
<span class="n">auth_mechanism</span><span class="o">=</span><span class="n">auth_mechanism</span><span class="p">,</span>
<span class="n">kerberos_service_name</span><span class="o">=</span><span class="n">kerberos_service_name</span><span class="p">,</span>
<span class="n">user</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
- <span class="n">database</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="s1">'default'</span><span class="p">)</span>
+ <span class="n">database</span><span class="o">=</span><span class="n">schema</span> <span class="ow">or</span> <span class="n">db</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="s1">'default'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_results</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">arraysize</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
- <span class="kn">from</span> <span class="nn">impala.error</span> <span class="kn">import</span> <span class="n">ProgrammingError</span>
- <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hql</span><span class="p">,</span> <span class="nb">basestring</span><span class="p">):</span>
+ <span class="kn">from</span> <span class="nn">impala.error</span> <span class="k">import</span> <span class="n">ProgrammingError</span>
+ <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hql</span><span class="p">,</span> <span class="n">basestring</span><span class="p">):</span>
<span class="n">hql</span> <span class="o">=</span> <span class="p">[</span><span class="n">hql</span><span class="p">]</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'data'</span><span class="p">:</span> <span class="p">[],</span>
@@ -754,7 +863,7 @@
<span class="c1"># may be `SET` or DDL</span>
<span class="n">records</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
<span class="k">except</span> <span class="n">ProgrammingError</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"get_results returned no records"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"get_results returned no records"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">records</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'data'</span><span class="p">:</span> <span class="n">records</span><span class="p">,</span>
@@ -769,12 +878,12 @@
<span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
<span class="n">delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
<span class="n">lineterminator</span><span class="o">=</span><span class="s1">'</span><span class="se">\r\n</span><span class="s1">'</span><span class="p">,</span>
- <span class="n">output_header</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">output_header</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">fetch_size</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span> <span class="ow">or</span> <span class="s1">'default'</span>
- <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">(</span><span class="n">schema</span><span class="p">)</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
<span class="k">with</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running query: "</span> <span class="o">+</span> <span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running query: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">hql</span><span class="p">)</span>
<span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">description</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">csv_filepath</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
@@ -785,15 +894,15 @@
<span class="k">if</span> <span class="n">output_header</span><span class="p">:</span>
<span class="n">writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">([</span><span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">cur</span><span class="o">.</span><span class="n">description</span><span class="p">])</span>
<span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
- <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">rows</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchmany</span><span class="p">(</span><span class="n">fetch_size</span><span class="p">)</span> <span class="k">if</span> <span class="n">row</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">rows</span><span class="p">:</span>
<span class="k">br
<TRUNCATED>
[19/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/bash_operator.html
----------------------------------------------------------------------
diff --git a/_modules/bash_operator.html b/_modules/bash_operator.html
index 045c47e..3b6f9ef 100644
--- a/_modules/bash_operator.html
+++ b/_modules/bash_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -180,7 +185,6 @@
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">bytes</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">signal</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">subprocess</span> <span class="k">import</span> <span class="n">Popen</span><span class="p">,</span> <span class="n">STDOUT</span><span class="p">,</span> <span class="n">PIPE</span>
<span class="kn">from</span> <span class="nn">tempfile</span> <span class="k">import</span> <span class="n">gettempdir</span><span class="p">,</span> <span class="n">NamedTemporaryFile</span>
@@ -220,7 +224,7 @@
<span class="n">output_encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">BashOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BashOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span> <span class="o">=</span> <span class="n">bash_command</span>
<span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span>
<span class="bp">self</span><span class="o">.</span><span class="n">xcom_push_flag</span> <span class="o">=</span> <span class="n">xcom_push</span>
@@ -232,7 +236,7 @@
<span class="sd"> which will be cleaned afterwards</span>
<span class="sd"> """</span>
<span class="n">bash_command</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"tmp dir root location: </span><span class="se">\n</span><span class="s2">"</span> <span class="o">+</span> <span class="n">gettempdir</span><span class="p">())</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Tmp dir root location: </span><span class="se">\n</span><span class="s2"> </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">gettempdir</span><span class="p">())</span>
<span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflowtmp'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">:</span>
<span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
@@ -240,25 +244,35 @@
<span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="n">fname</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span>
<span class="n">script_location</span> <span class="o">=</span> <span class="n">tmp_dir</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="n">fname</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Temporary script "</span>
- <span class="s2">"location :</span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">script_location</span><span class="p">))</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command: "</span> <span class="o">+</span> <span class="n">bash_command</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Temporary script location: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
+ <span class="n">script_location</span>
+ <span class="p">)</span>
+ <span class="k">def</span> <span class="nf">pre_exec</span><span class="p">():</span>
+ <span class="c1"># Restore default signal disposition and invoke setsid</span>
+ <span class="k">for</span> <span class="n">sig</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'SIGPIPE'</span><span class="p">,</span> <span class="s1">'SIGXFZ'</span><span class="p">,</span> <span class="s1">'SIGXFSZ'</span><span class="p">):</span>
+ <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">signal</span><span class="p">,</span> <span class="n">sig</span><span class="p">):</span>
+ <span class="n">signal</span><span class="o">.</span><span class="n">signal</span><span class="p">(</span><span class="nb">getattr</span><span class="p">(</span><span class="n">signal</span><span class="p">,</span> <span class="n">sig</span><span class="p">),</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIG_DFL</span><span class="p">)</span>
+ <span class="n">os</span><span class="o">.</span><span class="n">setsid</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command: </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">bash_command</span><span class="p">)</span>
<span class="n">sp</span> <span class="o">=</span> <span class="n">Popen</span><span class="p">(</span>
<span class="p">[</span><span class="s1">'bash'</span><span class="p">,</span> <span class="n">fname</span><span class="p">],</span>
<span class="n">stdout</span><span class="o">=</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">STDOUT</span><span class="p">,</span>
<span class="n">cwd</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">env</span><span class="p">,</span>
- <span class="n">preexec_fn</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">setsid</span><span class="p">)</span>
+ <span class="n">preexec_fn</span><span class="o">=</span><span class="n">pre_exec</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sp</span> <span class="o">=</span> <span class="n">sp</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Output:"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Output:"</span><span class="p">)</span>
<span class="n">line</span> <span class="o">=</span> <span class="s1">''</span>
- <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">iter</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">,</span> <span class="n">b</span><span class="s1">''</span><span class="p">):</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">iter</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">,</span> <span class="sa">b</span><span class="s1">''</span><span class="p">):</span>
<span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output_encoding</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
<span class="n">sp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Command exited with "</span>
- <span class="s2">"return code </span><span class="si">{0}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Command exited with return code </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span>
+ <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span>
+ <span class="p">)</span>
<span class="k">if</span> <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Bash command failed"</span><span class="p">)</span>
@@ -267,7 +281,7 @@
<span class="k">return</span> <span class="n">line</span></div>
<span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sending SIGTERM signal to bash process group'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sending SIGTERM signal to bash process group'</span><span class="p">)</span>
<span class="n">os</span><span class="o">.</span><span class="n">killpg</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getpgid</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">pid</span><span class="p">),</span> <span class="n">signal</span><span class="o">.</span><span class="n">SIGTERM</span><span class="p">)</span></div>
</pre></div>
[09/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/slack_operator.html
----------------------------------------------------------------------
diff --git a/_modules/slack_operator.html b/_modules/slack_operator.html
index 152902b..65a1a4d 100644
--- a/_modules/slack_operator.html
+++ b/_modules/slack_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>slack_operator</li>
+
+ <li>slack_operator</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,12 +181,12 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">from</span> <span class="nn">slackclient</span> <span class="kn">import</span> <span class="n">SlackClient</span>
-<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
-<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
-<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
<span class="kn">import</span> <span class="nn">json</span>
-<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">slackclient</span> <span class="k">import</span> <span class="n">SlackClient</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
<div class="viewcode-block" id="SlackAPIOperator"><a class="viewcode-back" href="../code.html#airflow.operators.SlackAPIOperator">[docs]</a><span class="k">class</span> <span class="nc">SlackAPIOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
@@ -177,9 +207,9 @@
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
<span class="n">token</span><span class="o">=</span><span class="s1">'unset'</span><span class="p">,</span>
<span class="n">method</span><span class="o">=</span><span class="s1">'unset'</span><span class="p">,</span>
- <span class="n">api_params</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">api_params</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SlackAPIOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SlackAPIOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
<span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="n">method</span>
<span class="bp">self</span><span class="o">.</span><span class="n">api_params</span> <span class="o">=</span> <span class="n">api_params</span>
@@ -205,8 +235,9 @@
<span class="n">sc</span> <span class="o">=</span> <span class="n">SlackClient</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">token</span><span class="p">)</span>
<span class="n">rc</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">api_call</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">api_params</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">rc</span><span class="p">[</span><span class="s1">'ok'</span><span class="p">]:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Slack API call failed ({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rc</span><span class="p">[</span><span class="s1">'error'</span><span class="p">]))</span>
- <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Slack API call failed: ({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rc</span><span class="p">[</span><span class="s1">'error'</span><span class="p">]))</span></div></div>
+ <span class="n">msg</span> <span class="o">=</span> <span class="s2">"Slack API call failed (</span><span class="si">%s</span><span class="s2">)"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rc</span><span class="p">[</span><span class="s1">'error'</span><span class="p">])</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span></div></div>
<div class="viewcode-block" id="SlackAPIPostOperator"><a class="viewcode-back" href="../code.html#airflow.operators.SlackAPIPostOperator">[docs]</a><span class="k">class</span> <span class="nc">SlackAPIPostOperator</span><span class="p">(</span><span class="n">SlackAPIOperator</span><span class="p">):</span>
@@ -225,7 +256,7 @@
<span class="sd"> :type attachments: array of hashes</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'username'</span><span class="p">,</span> <span class="s1">'text'</span><span class="p">,</span> <span class="s1">'attachments'</span><span class="p">)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'username'</span><span class="p">,</span> <span class="s1">'text'</span><span class="p">,</span> <span class="s1">'attachments'</span><span class="p">,</span> <span class="s1">'channel'</span><span class="p">)</span>
<span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#FFBA40'</span>
<span class="nd">@apply_defaults</span>
@@ -236,7 +267,7 @@
<span class="s1">'Here is a cat video instead</span><span class="se">\n</span><span class="s1">'</span>
<span class="s1">'https://www.youtube.com/watch?v=J---aiyznGQ'</span><span class="p">,</span>
<span class="n">icon_url</span><span class="o">=</span><span class="s1">'https://raw.githubusercontent.com/airbnb/airflow/master/airflow/www/static/pin_100.png'</span><span class="p">,</span>
- <span class="n">attachments</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">attachments</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="s1">'chat.postMessage'</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span> <span class="o">=</span> <span class="n">channel</span>
@@ -244,7 +275,7 @@
<span class="bp">self</span><span class="o">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">text</span>
<span class="bp">self</span><span class="o">.</span><span class="n">icon_url</span> <span class="o">=</span> <span class="n">icon_url</span>
<span class="bp">self</span><span class="o">.</span><span class="n">attachments</span> <span class="o">=</span> <span class="n">attachments</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">SlackAPIPostOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SlackAPIPostOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">construct_api_call_params</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
@@ -258,6 +289,9 @@
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -290,7 +324,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/sqlite_hook.html
----------------------------------------------------------------------
diff --git a/_modules/sqlite_hook.html b/_modules/sqlite_hook.html
index dc33f36..d64dd1c 100644
--- a/_modules/sqlite_hook.html
+++ b/_modules/sqlite_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/api.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/api.rst.txt b/_sources/api.rst.txt
index eef671c..856ec9e 100644
--- a/_sources/api.rst.txt
+++ b/_sources/api.rst.txt
@@ -28,16 +28,26 @@ configure as follows:
Authentication
--------------
-Only Kerberos authentication is currently supported for the API. To enable this set the following
-in the configuration:
+Authentication for the API is handled separately to the Web Authentication. The default is to not
+require any authentication on the API -- i.e. wide open by default. This is not recommended if your
+Airflow webserver is publicly accessible, and you should probably use the deny all backend:
-.. code-block:: bash
+.. code-block:: ini
+
+ [api]
+ auth_backend = airflow.api.auth.backend.deny_all
+
+
+Kerberos is the only "real" authentication mechanism currently supported for the API. To enable
+this set the following in the configuration:
+
+.. code-block:: ini
[api]
- auth_backend = airflow.api.auth.backend.default
+ auth_backend = airflow.api.auth.backend.kerberos_auth
[kerberos]
keytab = <KEYTAB>
-The Kerberos service is configured as `airflow/fully.qualified.domainname@REALM`. Make sure this
+The Kerberos service is configured as ``airflow/fully.qualified.domainname@REALM``. Make sure this
principal exists in the keytab file.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/code.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/code.rst.txt b/_sources/code.rst.txt
index fabe6db..1369b32 100644
--- a/_sources/code.rst.txt
+++ b/_sources/code.rst.txt
@@ -72,6 +72,7 @@ Operator API
PrestoIntervalCheckOperator,
PrestoValueCheckOperator,
PythonOperator,
+ PythonVirtualenvOperator,
S3KeySensor,
S3ToHiveTransfer,
ShortCircuitOperator,
@@ -91,13 +92,15 @@ Community-contributed Operators
.. automodule:: airflow.contrib.operators
:show-inheritance:
:members:
- SSHExecuteOperator,
+ SSHOperator,
VerticaOperator,
VerticaToHiveTransfer
.. autoclass:: airflow.contrib.operators.bigquery_operator.BigQueryOperator
.. autoclass:: airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator
+.. autoclass:: airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator
.. autoclass:: airflow.contrib.operators.ecs_operator.ECSOperator
+.. autoclass:: airflow.contrib.operators.file_to_wasb.FileToWasbOperator
.. autoclass:: airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator
.. autoclass:: airflow.contrib.operators.QuboleOperator
.. autoclass:: airflow.contrib.operators.hipchat_operator.HipChatAPIOperator
@@ -143,13 +146,13 @@ Variable Description
key within the JSON object
``{{ task_instance_key_str }}`` a unique, human-readable key to the task instance
formatted ``{dag_id}_{task_id}_{ds}``
-``conf`` the full configuration object located at
+``{{ conf }}`` the full configuration object located at
``airflow.configuration.conf`` which
represents the content of your
``airflow.cfg``
-``run_id`` the ``run_id`` of the current DAG run
-``dag_run`` a reference to the DagRun object
-``test_mode`` whether the task instance was called using
+``{{ run_id }}`` the ``run_id`` of the current DAG run
+``{{ dag_run }}`` a reference to the DagRun object
+``{{ test_mode }}`` whether the task instance was called using
the CLI's test subcommand
================================= ====================================
@@ -213,6 +216,7 @@ Hooks
:show-inheritance:
:members:
DbApiHook,
+ DockerHook,
HiveCliHook,
HiveMetastoreHook,
HiveServer2Hook,
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/concepts.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/concepts.rst.txt b/_sources/concepts.rst.txt
index 9f65256..56a9bdb 100644
--- a/_sources/concepts.rst.txt
+++ b/_sources/concepts.rst.txt
@@ -110,7 +110,7 @@ Airflow provides operators for many common tasks, including:
- ``PythonOperator`` - calls an arbitrary Python function
- ``EmailOperator`` - sends an email
- ``HTTPOperator`` - sends an HTTP request
-- ``SqlOperator`` - executes a SQL command
+- ``MySqlOperator``, ``SqliteOperator``, ``PostgresOperator``, ``MsSqlOperator``, ``OracleOperator``, ``JdbcOperator``, etc. - executes a SQL command
- ``Sensor`` - waits for a certain time, file, database row, S3 key, etc...
@@ -207,8 +207,7 @@ We can put this all together to build a simple pipeline:
with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag:
(
- dag
- >> DummyOperator(task_id='dummy_1')
+ DummyOperator(task_id='dummy_1')
>> BashOperator(
task_id='bash_1',
bash_command='echo "HELLO!"')
@@ -756,6 +755,8 @@ to the related tasks in Airflow.
This content will get rendered as markdown respectively in the "Graph View" and
"Task Details" pages.
+.. _jinja-templating:
+
Jinja Templating
================
@@ -781,7 +782,8 @@ Here, ``{{ ds }}`` is a macro, and because the ``env`` parameter of the
as an environment variable named ``EXECUTION_DATE`` in your Bash script.
You can use Jinja templating with every parameter that is marked as "templated"
-in the documentation.
+in the documentation. Template substitution occurs just before the pre_execute
+function of your operator is called.
Packaged dags
'''''''''''''
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/configuration.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/configuration.rst.txt b/_sources/configuration.rst.txt
index c4a3442..e68a341 100644
--- a/_sources/configuration.rst.txt
+++ b/_sources/configuration.rst.txt
@@ -83,6 +83,31 @@ within the metadata database. The ``crypto`` package is highly recommended
during installation. The ``crypto`` package does require that your operating
system have libffi-dev installed.
+If ``crypto`` package was not installed initially, you can still enable encryption for
+connections by following steps below:
+
+1. Install crypto package ``pip install apache-airflow[crypto]``
+2. Generate fernet_key, using this code snippet below. fernet_key must be a base64-encoded 32-byte key.
+
+.. code:: python
+
+ from cryptography.fernet import Fernet
+ fernet_key= Fernet.generate_key()
+ print(fernet_key) # your fernet_key, keep it in secured place!
+
+3. Replace ``airflow.cfg`` fernet_key value with the one from step 2.
+Alternatively, you can store your fernet_key in OS environment variable. You
+do not need to change ``airflow.cfg`` in this case as AirFlow will use environment
+variable over the value in ``airflow.cfg``:
+
+.. code-block:: bash
+
+ # Note the double underscores
+ EXPORT AIRFLOW__CORE__FERNET_KEY = your_fernet_key
+
+4. Restart AirFlow webserver.
+5. For existing connections (the ones that you had defined before installing ``airflow[crypto]`` and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it.
+
Connections in Airflow pipelines can be created using environment variables.
The environment variable needs to have a prefix of ``AIRFLOW_CONN_`` for
Airflow with the value in a URI format to use the connection properly. Please
@@ -131,6 +156,41 @@ to monitor your workers. You can use the shortcut command ``airflow flower``
to start a Flower web server.
+Scaling Out with Dask
+'''''''''''''''''''''
+
+``DaskExecutor`` allows you to run Airflow tasks in a Dask Distributed cluster.
+
+Dask clusters can be run on a single machine or on remote networks. For complete
+details, consult the `Distributed documentation <https://distributed.readthedocs.io/>`_.
+
+To create a cluster, first start a Scheduler:
+
+.. code-block:: bash
+
+ # default settings for a local cluster
+ DASK_HOST=127.0.0.1
+ DASK_PORT=8786
+
+ dask-scheduler --host $DASK_HOST --port $DASK_PORT
+
+Next start at least one Worker on any machine that can connect to the host:
+
+.. code-block:: bash
+
+ dask-worker $DASK_HOST:$DASK_PORT
+
+Edit your ``airflow.cfg`` to set your executor to ``DaskExecutor`` and provide
+the Dask Scheduler address in the ``[dask]`` section.
+
+Please note:
+
+- Each Dask worker must be able to import Airflow and any dependencies you
+ require.
+- Dask does not support queues. If an Airflow task was created with a queue, a
+ warning will be raised but the task will be submitted to the cluster.
+
+
Logs
''''
Users can specify a logs folder in ``airflow.cfg``. By default, it is in
@@ -159,7 +219,8 @@ try to use ``S3Hook('MyS3Conn')``.
In the Airflow Web UI, local logs take precedance over remote logs. If local logs
can not be found or accessed, the remote logs will be displayed. Note that logs
are only sent to remote storage once a task completes (including failure). In other
-words, remote logs for running tasks are unavailable.
+words, remote logs for running tasks are unavailable. Logs are stored in the log
+folder as ``{dag_id}/{task_id}/{execution_date}/{try_number}.log``.
Scaling Out on Mesos (community contributed)
''''''''''''''''''''''''''''''''''''''''''''
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/faq.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/faq.rst.txt b/_sources/faq.rst.txt
index 1e4c038..04cf346 100644
--- a/_sources/faq.rst.txt
+++ b/_sources/faq.rst.txt
@@ -24,7 +24,7 @@ Here are some of the common causes:
do not override their parent DAG's ``schedule_interval``.
- Is your ``start_date`` beyond where you can see it in the UI? If you
- set your it to some time say 3 months ago, you won't be able to see
+ set your ``start_date`` to some time say 3 months ago, you won't be able to see
it in the main view in the UI, but you should be able to see it in the
``Menu -> Browse ->Task Instances``.
@@ -66,8 +66,8 @@ documentation
Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?
------------------------------------------------------------------------------------------------------
-- Verify that the ``fernet_key`` defined in ``$AIRFLOW_HOME/airflow.cfg`` is a valid Fernet key. It must be a base64-encoded 32-byte key. You need to restart the webserver after you update the key
-- For existing connections (the ones that you had defined before installing ``airflow[crypto]`` and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it
+Check out the ``Connections`` section in the Configuration section of the
+documentation
What's the deal with ``start_date``?
------------------------------------
@@ -80,7 +80,7 @@ task. From that point on, the scheduler creates new DagRuns based on
your ``schedule_interval`` and the corresponding task instances run as your
dependencies are met. When introducing new tasks to your DAG, you need to
pay special attention to ``start_date``, and may want to reactivate
-inactive DagRuns to get the new task to get onboarded properly.
+inactive DagRuns to get the new task onboarded properly.
We recommend against using dynamic values as ``start_date``, especially
``datetime.now()`` as it can be quite confusing. The task is triggered
@@ -103,7 +103,7 @@ it enforces this idea of rounded schedules.
When using ``depends_on_past=True`` it's important to pay special attention
to ``start_date`` as the past dependency is not enforced only on the specific
-schedule of the ``start_date`` specified for the task. It' also
+schedule of the ``start_date`` specified for the task. It's also
important to watch DagRun activity status in time when introducing
new ``depends_on_past=True``, unless you are planning on running a backfill
for the new task(s).
@@ -111,7 +111,7 @@ for the new task(s).
Also important to note is that the tasks ``start_date``, in the context of a
backfill CLI command, get overridden by the backfill's command ``start_date``.
This allows for a backfill on tasks that have ``depends_on_past=True`` to
-actually start, if it wasn't the case, the backfill just wouldn't start.
+actually start, if that wasn't the case, the backfill just wouldn't start.
How can I create DAGs dynamically?
----------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/installation.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/installation.rst.txt b/_sources/installation.rst.txt
index 289f64f..b4fb126 100644
--- a/_sources/installation.rst.txt
+++ b/_sources/installation.rst.txt
@@ -8,18 +8,18 @@ The easiest way to install the latest stable version of Airflow is with ``pip``:
.. code-block:: bash
- pip install airflow
+ pip install apache-airflow
You can also install Airflow with support for extra features like ``s3`` or ``postgres``:
.. code-block:: bash
- pip install "airflow[s3, postgres]"
+ pip install "apache-airflow[s3, postgres]"
Extra Packages
''''''''''''''
-The ``airflow`` PyPI basic package only installs what's needed to get started.
+The ``apache-airflow`` PyPI basic package only installs what's needed to get started.
Subpackages can be installed depending on what will be useful in your
environment. For instance, if you don't need connectivity with Postgres,
you won't have to go through the trouble of installing the ``postgres-devel``
@@ -30,61 +30,63 @@ these extra dependencies.
Here's the list of the subpackages and what they enable:
-+---------------+-------------------------------------+-------------------------------------------------+
-| subpackage | install command | enables |
-+===============+=====================================+=================================================+
-| all | ``pip install airflow[all]`` | All Airflow features known to man |
-+---------------+-------------------------------------+-------------------------------------------------+
-| all_dbs | ``pip install airflow[all_dbs]`` | All databases integrations |
-+---------------+-------------------------------------+-------------------------------------------------+
-| async | ``pip install airflow[async]`` | Async worker classes for gunicorn |
-+---------------+-------------------------------------+-------------------------------------------------+
-| devel | ``pip install airflow[devel]`` | Minimum dev tools requirements |
-+---------------+-------------------------------------+-------------------------------------------------+
-| devel_hadoop |``pip install airflow[devel_hadoop]``| Airflow + dependencies on the Hadoop stack |
-+---------------+-------------------------------------+-------------------------------------------------+
-| celery | ``pip install airflow[celery]`` | CeleryExecutor |
-+---------------+-------------------------------------+-------------------------------------------------+
-| crypto | ``pip install airflow[crypto]`` | Encrypt connection passwords in metadata db |
-+---------------+-------------------------------------+-------------------------------------------------+
-| druid | ``pip install airflow[druid]`` | Druid.io related operators & hooks |
-+---------------+-------------------------------------+-------------------------------------------------+
-| gcp_api | ``pip install airflow[gcp_api]`` | Google Cloud Platform hooks and operators |
-| | | (using ``google-api-python-client``) |
-+---------------+-------------------------------------+-------------------------------------------------+
-| jdbc | ``pip install airflow[jdbc]`` | JDBC hooks and operators |
-+---------------+-------------------------------------+-------------------------------------------------+
-| hdfs | ``pip install airflow[hdfs]`` | HDFS hooks and operators |
-+---------------+-------------------------------------+-------------------------------------------------+
-| hive | ``pip install airflow[hive]`` | All Hive related operators |
-+---------------+-------------------------------------+-------------------------------------------------+
-| kerberos | ``pip install airflow[kerberos]`` | kerberos integration for kerberized hadoop |
-+---------------+-------------------------------------+-------------------------------------------------+
-| ldap | ``pip install airflow[ldap]`` | ldap authentication for users |
-+---------------+-------------------------------------+-------------------------------------------------+
-| mssql | ``pip install airflow[mssql]`` | Microsoft SQL operators and hook, |
-| | | support as an Airflow backend |
-+---------------+-------------------------------------+-------------------------------------------------+
-| mysql | ``pip install airflow[mysql]`` | MySQL operators and hook, support as |
-| | | an Airflow backend |
-+---------------+-------------------------------------+-------------------------------------------------+
-| password | ``pip install airflow[password]`` | Password Authentication for users |
-+---------------+-------------------------------------+-------------------------------------------------+
-| postgres | ``pip install airflow[postgres]`` | Postgres operators and hook, support |
-| | | as an Airflow backend |
-+---------------+-------------------------------------+-------------------------------------------------+
-| qds | ``pip install airflow[qds]`` | Enable QDS (qubole data services) support |
-+---------------+-------------------------------------+-------------------------------------------------+
-| rabbitmq | ``pip install airflow[rabbitmq]`` | Rabbitmq support as a Celery backend |
-+---------------+-------------------------------------+-------------------------------------------------+
-| s3 | ``pip install airflow[s3]`` | ``S3KeySensor``, ``S3PrefixSensor`` |
-+---------------+-------------------------------------+-------------------------------------------------+
-| samba | ``pip install airflow[samba]`` | ``Hive2SambaOperator`` |
-+---------------+-------------------------------------+-------------------------------------------------+
-| slack | ``pip install airflow[slack]`` | ``SlackAPIPostOperator`` |
-+---------------+-------------------------------------+-------------------------------------------------+
-| vertica | ``pip install airflow[vertica]`` | Vertica hook |
-| | | support as an Airflow backend |
-+---------------+-------------------------------------+-------------------------------------------------+
-| cloudant | ``pip install airflow[cloudant]`` | Cloudant hook |
-+---------------+-------------------------------------+-------------------------------------------------+
++---------------+----------------------------------------------+-------------------------------------------------+
+| subpackage | install command | enables |
++===============+==============================================+=================================================+
+| all | ``pip install apache-airflow[all]`` | All Airflow features known to man |
++---------------+----------------------------------------------+-------------------------------------------------+
+| all_dbs | ``pip install apache-airflow[all_dbs]`` | All databases integrations |
++---------------+----------------------------------------------+-------------------------------------------------+
+| async | ``pip install apache-airflow[async]`` | Async worker classes for gunicorn |
++---------------+----------------------------------------------+-------------------------------------------------+
+| devel | ``pip install apache-airflow[devel]`` | Minimum dev tools requirements |
++---------------+----------------------------------------------+-------------------------------------------------+
+| devel_hadoop | ``pip install apache-airflow[devel_hadoop]`` | Airflow + dependencies on the Hadoop stack |
++---------------+----------------------------------------------+-------------------------------------------------+
+| celery | ``pip install apache-airflow[celery]`` | CeleryExecutor |
++---------------+----------------------------------------------+-------------------------------------------------+
+| crypto | ``pip install apache-airflow[crypto]`` | Encrypt connection passwords in metadata db |
++---------------+----------------------------------------------+-------------------------------------------------+
+| druid | ``pip install apache-airflow[druid]`` | Druid.io related operators & hooks |
++---------------+----------------------------------------------+-------------------------------------------------+
+| gcp_api | ``pip install apache-airflow[gcp_api]`` | Google Cloud Platform hooks and operators |
+| | | (using ``google-api-python-client``) |
++---------------+----------------------------------------------+-------------------------------------------------+
+| jdbc | ``pip install apache-airflow[jdbc]`` | JDBC hooks and operators |
++---------------+----------------------------------------------+-------------------------------------------------+
+| hdfs | ``pip install apache-airflow[hdfs]`` | HDFS hooks and operators |
++---------------+----------------------------------------------+-------------------------------------------------+
+| hive | ``pip install apache-airflow[hive]`` | All Hive related operators |
++---------------+----------------------------------------------+-------------------------------------------------+
+| kerberos | ``pip install apache-airflow[kerberos]`` | kerberos integration for kerberized hadoop |
++---------------+----------------------------------------------+-------------------------------------------------+
+| ldap | ``pip install apache-airflow[ldap]`` | ldap authentication for users |
++---------------+----------------------------------------------+-------------------------------------------------+
+| mssql | ``pip install apache-airflow[mssql]`` | Microsoft SQL operators and hook, |
+| | | support as an Airflow backend |
++---------------+----------------------------------------------+-------------------------------------------------+
+| mysql | ``pip install apache-airflow[mysql]`` | MySQL operators and hook, support as |
+| | | an Airflow backend |
++---------------+----------------------------------------------+-------------------------------------------------+
+| password | ``pip install apache-airflow[password]`` | Password Authentication for users |
++---------------+----------------------------------------------+-------------------------------------------------+
+| postgres | ``pip install apache-airflow[postgres]`` | Postgres operators and hook, support |
+| | | as an Airflow backend |
++---------------+----------------------------------------------+-------------------------------------------------+
+| qds | ``pip install apache-airflow[qds]`` | Enable QDS (qubole data services) support |
++---------------+----------------------------------------------+-------------------------------------------------+
+| rabbitmq | ``pip install apache-airflow[rabbitmq]`` | Rabbitmq support as a Celery backend |
++---------------+----------------------------------------------+-------------------------------------------------+
+| s3 | ``pip install apache-airflow[s3]`` | ``S3KeySensor``, ``S3PrefixSensor`` |
++---------------+----------------------------------------------+-------------------------------------------------+
+| samba | ``pip install apache-airflow[samba]`` | ``Hive2SambaOperator`` |
++---------------+----------------------------------------------+-------------------------------------------------+
+| slack | ``pip install apache-airflow[slack]`` | ``SlackAPIPostOperator`` |
++---------------+----------------------------------------------+-------------------------------------------------+
+| vertica | ``pip install apache-airflow[vertica]`` | Vertica hook |
+| | | support as an Airflow backend |
++---------------+----------------------------------------------+-------------------------------------------------+
+| cloudant | ``pip install apache-airflow[cloudant]`` | Cloudant hook |
++---------------+----------------------------------------------+-------------------------------------------------+
+| redis | ``pip install apache-airflow[redis]`` | Redis hooks and sensors |
++---------------+----------------------------------------------+-------------------------------------------------+
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/integration.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/integration.rst.txt b/_sources/integration.rst.txt
index 10bc038..4887486 100644
--- a/_sources/integration.rst.txt
+++ b/_sources/integration.rst.txt
@@ -1,15 +1,179 @@
Integration
===========
+- :ref:`Azure`
- :ref:`AWS`
+- :ref:`Databricks`
- :ref:`GCP`
+
+.. _Azure:
+
+Azure: Microsoft Azure
+----------------------
+
+Airflow has limited support for Microsoft Azure: interfaces exist only for Azure Blob
+Storage. Note that the Hook, Sensor and Operator are in the contrib section.
+
+Azure Blob Storage
+''''''''''''''''''
+
+All classes communicate via the Window Azure Storage Blob protocol. Make sure that a
+Airflow connection of type `wasb` exists. Authorization can be done by supplying a
+login (=Storage account name) and password (=KEY), or login and SAS token in the extra
+field (see connection `wasb_default` for an example).
+
+- :ref:`WasbBlobSensor`: Checks if a blob is present on Azure Blob storage.
+- :ref:`WasbPrefixSensor`: Checks if blobs matching a prefix are present on Azure Blob storage.
+- :ref:`FileToWasbOperator`: Uploads a local file to a container as a blob.
+- :ref:`WasbHook`: Interface with Azure Blob Storage.
+
+.. _WasbBlobSensor:
+
+WasbBlobSensor
+"""""""""""""""
+
+.. autoclass:: airflow.contrib.sensors.wasb_sensor.WasbBlobSensor
+
+.. _WasbPrefixSensor:
+
+WasbPrefixSensor
+"""""""""""""""""
+
+.. autoclass:: airflow.contrib.sensors.wasb_sensor.WasbPrefixSensor
+
+.. _FileToWasbOperator:
+
+FileToWasbOperator
+"""""""""""""""""""
+
+.. autoclass:: airflow.contrib.operators.file_to_wasb.FileToWasbOperator
+
+.. _WasbHook:
+
+WasbHook
+"""""""""
+
+.. autoclass:: airflow.contrib.hooks.wasb_hook.WasbHook
+
+
+
.. _AWS:
-AWS: Amazon Webservices
+AWS: Amazon Web Services
-----------------------
----
+Airflow has extensive support for Amazon Web Services. But note that the Hooks, Sensors and
+Operators are in the contrib section.
+
+AWS EMR
+''''''''
+
+- :ref:`EmrAddStepsOperator` : Adds steps to an existing EMR JobFlow.
+- :ref:`EmrCreateJobFlowOperator` : Creates an EMR JobFlow, reading the config from the EMR connection.
+- :ref:`EmrTerminateJobFlowOperator` : Terminates an EMR JobFlow.
+- :ref:`EmrHook` : Interact with AWS EMR.
+
+.. _EmrAddStepsOperator:
+
+EmrAddStepsOperator
+""""""""
+
+.. autoclass:: airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator
+
+.. _EmrCreateJobFlowOperator:
+
+EmrCreateJobFlowOperator
+""""""""
+
+.. autoclass:: airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator
+
+.. _EmrTerminateJobFlowOperator:
+
+EmrTerminateJobFlowOperator
+""""""""
+
+.. autoclass:: airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator
+
+.. _EmrHook:
+
+EmrHook
+""""""""
+
+.. autoclass:: airflow.contrib.hooks.emr_hook.EmrHook
+
+
+AWS S3
+'''''''
+
+- :ref:`S3FileTransformOperator` : Copies data from a source S3 location to a temporary location on the local filesystem.
+- :ref:`S3ToHiveTransfer` : Moves data from S3 to Hive. The operator downloads a file from S3, stores the file locally before loading it into a Hive table.
+- :ref:`S3Hook` : Interact with AWS S3.
+
+.. _S3FileTransformOperator:
+
+S3FileTransformOperator
+""""""""""""""""""""""""
+
+.. autoclass:: airflow.operators.s3_file_transform_operator.S3FileTransformOperator
+
+.. _S3ToHiveTransfer:
+
+S3ToHiveTransfer
+"""""""""""""""""
+
+.. autoclass:: airflow.operators.s3_to_hive_operator.S3ToHiveTransfer
+
+.. _S3Hook:
+
+S3Hook
+"""""""
+
+.. autoclass:: airflow.hooks.S3_hook.S3Hook
+
+
+AWS EC2 Container Service
+''''''''''''''''''''''''''
+
+- :ref:`ECSOperator` : Execute a task on AWS EC2 Container Service.
+
+.. _ECSOperator:
+
+ECSOperator
+""""""""""""
+
+.. autoclass:: airflow.contrib.operators.ecs_operator.ECSOperator
+
+
+AWS RedShift
+'''''''''''''
+
+- :ref:`RedshiftToS3Transfer` : Executes an unload command to S3 as a CSV with headers.
+
+.. _RedshiftToS3Transfer:
+
+RedshiftToS3Transfer
+"""""""""""""""""""""
+
+.. autoclass:: airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer
+
+
+
+.. _Databricks:
+
+Databricks
+----------
+
+`Databricks <https://databricks.com/>`_ has contributed an Airflow operator which enables
+submitting runs to the Databricks platform. Internally the operator talks to the
+``api/2.0/jobs/runs/submit`` `endpoint <https://docs.databricks.com/api/latest/jobs.html#runs-submit>`_.
+
+DatabricksSubmitRunOperator
+'''''''''''''''''''''''''''
+
+.. autoclass:: airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator
+
+
.. _GCP:
@@ -20,11 +184,81 @@ Airflow has extensive support for the Google Cloud Platform. But note that most
Operators are in the contrib section. Meaning that they have a *beta* status, meaning that
they can have breaking changes between minor releases.
+Logging
+'''''''
+
+Airflow can be configured to read and write task logs in Google cloud storage.
+Follow the steps below to enable Google cloud storage logging.
+
+#. Airflow's logging system requires a custom .py file to be located in the ``PYTHONPATH``, so that it's importable from Airflow. Start by creating a directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
+#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and ``$AIRFLOW_HOME/config/__init__.py``.
+#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py`` into the ``log_config.py`` file that was just created in the step above.
+#. Customize the following portions of the template:
+
+ .. code-block:: bash
+
+ # Add this variable to the top of the file. Note the trailing slash.
+ GCS_LOG_FOLDER = 'gs://<bucket where logs should be persisted>/'
+
+ # Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
+ LOGGING_CONFIG = ...
+
+ # Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG variable
+ 'gcs.task': {
+ 'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
+ 'formatter': 'airflow.task',
+ 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ 'gcs_log_folder': GCS_LOG_FOLDER,
+ 'filename_template': FILENAME_TEMPLATE,
+ },
+
+ # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of 'file.task'.
+ 'loggers': {
+ 'airflow.task': {
+ 'handlers': ['gcs.task'],
+ ...
+ },
+ 'airflow.task_runner': {
+ 'handlers': ['gcs.task'],
+ ...
+ },
+ 'airflow': {
+ 'handlers': ['console'],
+ ...
+ },
+ }
+
+#. Make sure a Google cloud platform connection hook has been defined in Airflow. The hook should have read and write access to the Google cloud storage bucket defined above in ``GCS_LOG_FOLDER``.
+
+#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
+
+ .. code-block:: bash
+
+ task_log_reader = gcs.task
+ logging_config_class = log_config.LOGGING_CONFIG
+ remote_log_conn_id = <name of the Google cloud platform hook>
+
+#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
+#. Verify that logs are showing up for newly executed tasks in the bucket you've defined.
+#. Verify that the Google cloud storage viewer is working in the UI. Pull up a newly executed task, and verify that you see something like:
+
+ .. code-block:: bash
+
+ *** Reading remote log from gs://<bucket where logs should be persisted>/example_bash_operator/run_this_last/2017-10-03T00:00:00/16.log.
+ [2017-10-03 21:57:50,056] {cli.py:377} INFO - Running on host chrisr-00532
+ [2017-10-03 21:57:50,093] {base_task_runner.py:115} INFO - Running: ['bash', '-c', u'airflow run example_bash_operator run_this_last 2017-10-03T00:00:00 --job_id 47 --raw -sd DAGS_FOLDER/example_dags/example_bash_operator.py']
+ [2017-10-03 21:57:51,264] {base_task_runner.py:98} INFO - Subtask: [2017-10-03 21:57:51,263] {__init__.py:45} INFO - Using executor SequentialExecutor
+ [2017-10-03 21:57:51,306] {base_task_runner.py:98} INFO - Subtask: [2017-10-03 21:57:51,306] {models.py:186} INFO - Filling up the DagBag from /airflow/dags/example_dags/example_bash_operator.py
+
+Note the top line that says it's reading from the remote log file.
+
+Please be aware that if you were persisting logs to Google cloud storage using the old-style airflow.cfg configuration method, the old logs will no longer be visible in the Airflow UI, though they'll still exist in Google cloud storage. This is a backwards incompatbile change. If you are unhappy with it, you can change the ``FILENAME_TEMPLATE`` to reflect the old-style log filename format.
+
BigQuery
''''''''
BigQuery Operators
-^^^^^^^^^^^^^^^^^^
+""""""""""""""""""
- :ref:`BigQueryCheckOperator` : Performs checks against a SQL query that will return a single row with different values.
- :ref:`BigQueryValueCheckOperator` : Performs a simple value check using SQL code.
@@ -37,48 +271,48 @@ BigQuery Operators
.. _BigQueryCheckOperator:
BigQueryCheckOperator
-"""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator
.. _BigQueryValueCheckOperator:
BigQueryValueCheckOperator
-""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator
.. _BigQueryIntervalCheckOperator:
BigQueryIntervalCheckOperator
-"""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator
.. _BigQueryOperator:
BigQueryOperator
-""""""""""""""""
+^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.bigquery_operator.BigQueryOperator
.. _BigQueryToBigQueryOperator:
BigQueryToBigQueryOperator
-""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator
.. _BigQueryToCloudStorageOperator:
BigQueryToCloudStorageOperator
-""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator
BigQueryHook
-^^^^^^^^^^^^
+""""""""""""
.. autoclass:: airflow.contrib.hooks.bigquery_hook.BigQueryHook
:members:
@@ -88,14 +322,15 @@ Cloud DataFlow
''''''''''''''
DataFlow Operators
-^^^^^^^^^^^^^^^^^^
+""""""""""""""""""
-- :ref:`DataFlowJavaOperator` :
+- :ref:`DataFlowJavaOperator` : launching Cloud Dataflow jobs written in Java.
+- :ref:`DataFlowPythonOperator` : launching Cloud Dataflow jobs written in python.
.. _DataFlowJavaOperator:
DataFlowJavaOperator
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator
@@ -133,8 +368,16 @@ DataFlowJavaOperator
},
dag=dag)
+.. _DataFlowPythonOperator:
+
+DataFlowPythonOperator
+^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator
+
+
DataFlowHook
-^^^^^^^^^^^^
+""""""""""""
.. autoclass:: airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook
:members:
@@ -145,7 +388,7 @@ Cloud DataProc
''''''''''''''
DataProc Operators
-^^^^^^^^^^^^^^^^^^
+""""""""""""""""""
- :ref:`DataProcPigOperator` : Start a Pig query Job on a Cloud DataProc cluster.
- :ref:`DataProcHiveOperator` : Start a Hive query Job on a Cloud DataProc cluster.
@@ -157,35 +400,35 @@ DataProc Operators
.. _DataProcPigOperator:
DataProcPigOperator
-"""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.dataproc_operator.DataProcPigOperator
.. _DataProcHiveOperator:
DataProcHiveOperator
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.dataproc_operator.DataProcHiveOperator
.. _DataProcSparkSqlOperator:
DataProcSparkSqlOperator
-""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator
.. _DataProcSparkOperator:
DataProcSparkOperator
-"""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.dataproc_operator.DataProcSparkOperator
.. _DataProcHadoopOperator:
DataProcHadoopOperator
-""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.dataproc_operator.DataProcHadoopOperator
@@ -201,22 +444,73 @@ DataProcPySparkOperator
Cloud Datastore
'''''''''''''''
-Datastore Operators
-^^^^^^^^^^^^^^^^^^^
-
DatastoreHook
-~~~~~~~~~~~~~
+"""""""""""""
.. autoclass:: airflow.contrib.hooks.datastore_hook.DatastoreHook
:members:
+Cloud ML Engine
+'''''''''''''''
+
+Cloud ML Engine Operators
+"""""""""""""""""""""""""
+
+- :ref:`MLEngineBatchPredictionOperator` : Start a Cloud ML Engine batch prediction job.
+- :ref:`MLEngineModelOperator` : Manages a Cloud ML Engine model.
+- :ref:`MLEngineTrainingOperator` : Start a Cloud ML Engine training job.
+- :ref:`MLEngineVersionOperator` : Manages a Cloud ML Engine model version.
+
+.. _MLEngineBatchPredictionOperator:
+
+MLEngineBatchPredictionOperator
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator
+ :members:
+
+.. _MLEngineModelOperator:
+
+MLEngineModelOperator
+^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: airflow.contrib.operators.mlengine_operator.MLEngineModelOperator
+ :members:
+
+.. _MLEngineTrainingOperator:
+
+MLEngineTrainingOperator
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator
+ :members:
+
+.. _MLEngineVersionOperator:
+
+MLEngineVersionOperator
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. autoclass:: airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator
+ :members:
+
+Cloud ML Engine Hook
+""""""""""""""""""""
+
+.. _MLEngineHook:
+
+MLEngineHook
+^^^^^^^^^^^
+
+.. autoclass:: airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook
+ :members:
+
Cloud Storage
'''''''''''''
Storage Operators
-^^^^^^^^^^^^^^^^^
+"""""""""""""""""
- :ref:`GoogleCloudStorageDownloadOperator` : Downloads a file from Google Cloud Storage.
- :ref:`GoogleCloudStorageToBigQueryOperator` : Loads files from Google cloud storage into BigQuery.
@@ -224,7 +518,7 @@ Storage Operators
.. _GoogleCloudStorageDownloadOperator:
GoogleCloudStorageDownloadOperator
-""""""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator
:members:
@@ -232,15 +526,14 @@ GoogleCloudStorageDownloadOperator
.. _GoogleCloudStorageToBigQueryOperator:
GoogleCloudStorageToBigQueryOperator
-""""""""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autoclass:: airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator
:members:
GoogleCloudStorageHook
-^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""""
.. autoclass:: airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook
:members:
-
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/scheduler.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/scheduler.rst.txt b/_sources/scheduler.rst.txt
index 749d58a..8029eb0 100644
--- a/_sources/scheduler.rst.txt
+++ b/_sources/scheduler.rst.txt
@@ -91,6 +91,7 @@ will do, is to instruct the scheduler to only create a DAG Run for the most curr
interval series.
.. code:: python
+
"""
Code that goes along with the Airflow tutorial located at:
https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py
@@ -104,7 +105,7 @@ interval series.
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2015, 12, 1),
- 'email': ['airflow@airflow.com'],
+ 'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
@@ -146,8 +147,19 @@ To Keep in Mind
Here are some of the ways you can **unblock tasks**:
-* From the UI, you can **clear** (as in delete the status of) individual task instances from the task instances dialog, while defining whether you want to includes the past/future and the upstream/downstream dependencies. Note that a confirmation window comes next and allows you to see the set you are about to clear.
-* The CLI command ``airflow clear -h`` has lots of options when it comes to clearing task instance states, including specifying date ranges, targeting task_ids by specifying a regular expression, flags for including upstream and downstream relatives, and targeting task instances in specific states (``failed``, or ``success``)
-* Marking task instances as successful can be done through the UI. This is mostly to fix false negatives, or for instance when the fix has been applied outside of Airflow.
-* The ``airflow backfill`` CLI subcommand has a flag to ``--mark_success`` and allows selecting subsections of the DAG as well as specifying date ranges.
+* From the UI, you can **clear** (as in delete the status of) individual task instances
+ from the task instances dialog, while defining whether you want to includes the past/future
+ and the upstream/downstream dependencies. Note that a confirmation window comes next and
+ allows you to see the set you are about to clear. You can also clear all task instances
+ associated with the dag.
+* The CLI command ``airflow clear -h`` has lots of options when it comes to clearing task instance
+ states, including specifying date ranges, targeting task_ids by specifying a regular expression,
+ flags for including upstream and downstream relatives, and targeting task instances in specific
+ states (``failed``, or ``success``)
+* Clearing a task instance will no longer delete the task instance record. Instead it updates
+ max_tries and set the current task instance state to be None.
+* Marking task instances as successful can be done through the UI. This is mostly to fix false negatives,
+ or for instance when the fix has been applied outside of Airflow.
+* The ``airflow backfill`` CLI subcommand has a flag to ``--mark_success`` and allows selecting
+ subsections of the DAG as well as specifying date ranges.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/security.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/security.rst.txt b/_sources/security.rst.txt
index 70db606..f33ff04 100644
--- a/_sources/security.rst.txt
+++ b/_sources/security.rst.txt
@@ -6,7 +6,9 @@ to the web application is to do it at the network level, or by using
SSH tunnels.
It is however possible to switch on authentication by either using one of the supplied
-backends or create your own.
+backends or creating your own.
+
+Be sure to checkout :doc:`api` for securing the API.
Web Authentication
------------------
@@ -72,6 +74,11 @@ Valid search_scope options can be found in the `ldap3 Documentation <http://ldap
user_filter = objectClass=*
# in case of Active Directory you would use: user_name_attr = sAMAccountName
user_name_attr = uid
+ # group_member_attr should be set accordingly with *_filter
+ # eg :
+ # group_member_attr = groupMembership
+ # superuser_filter = groupMembership=CN=airflow-super-users...
+ group_member_attr = memberOf
superuser_filter = memberOf=CN=airflow-super-users,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
data_profiler_filter = memberOf=CN=airflow-data-profilers,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
bind_user = cn=Manager,dc=example,dc=com
@@ -89,7 +96,7 @@ Roll your own
Airflow uses ``flask_login`` and
exposes a set of hooks in the ``airflow.default_login`` module. You can
-alter the content and make it part of the ``PYTHONPATH`` and configure it as a backend in ``airflow.cfg```.
+alter the content and make it part of the ``PYTHONPATH`` and configure it as a backend in ``airflow.cfg``.
.. code-block:: bash
@@ -100,12 +107,14 @@ alter the content and make it part of the ``PYTHONPATH`` and configure it as a b
Multi-tenancy
-------------
-You can filter the list of dags in webserver by owner name, when authentication
-is turned on, by setting webserver.filter_by_owner as true in your ``airflow.cfg``
-With this, when a user authenticates and logs into webserver, it will see only the dags
-which it is owner of. A super_user, will be able to see all the dags although.
-This makes the web UI a multi-tenant UI, where a user will only be able to see dags
-created by itself.
+You can filter the list of dags in webserver by owner name when authentication
+is turned on by setting ``webserver:filter_by_owner`` in your config. With this, a user will see
+only the dags which it is owner of, unless it is a superuser.
+
+.. code-block:: bash
+
+ [webserver]
+ filter_by_owner = True
Kerberos
@@ -118,17 +127,18 @@ to authenticate against kerberized services.
Limitations
'''''''''''
-Please note that at this time not all hooks have been adjusted to make use of this functionality yet.
+Please note that at this time, not all hooks have been adjusted to make use of this functionality.
Also it does not integrate kerberos into the web interface and you will have to rely on network
level security for now to make sure your service remains secure.
-Celery integration has not been tried and tested yet. However if you generate a key tab for every host
-and launch a ticket renewer next to every worker it will most likely work.
+Celery integration has not been tried and tested yet. However, if you generate a key tab for every
+host and launch a ticket renewer next to every worker it will most likely work.
Enabling kerberos
'''''''''''''''''
-#### Airflow
+Airflow
+^^^^^^^
To enable kerberos you will need to generate a (service) key tab.
@@ -160,7 +170,8 @@ Launch the ticket renewer by
# run ticket renewer
airflow kerberos
-#### Hadoop
+Hadoop
+^^^^^^
If want to use impersonation this needs to be enabled in ``core-site.xml`` of your hadoop config.
@@ -186,8 +197,8 @@ Of course if you need to tighten your security replace the asterisk with somethi
Using kerberos authentication
'''''''''''''''''''''''''''''
-The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to use it simply
-update the connection details with, for example:
+The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to
+use it, simply update the connection details with, for example:
.. code-block:: bash
@@ -197,7 +208,7 @@ Adjust the principal to your settings. The _HOST part will be replaced by the fu
the server.
You can specify if you would like to use the dag owner as the user for the connection or the user specified in the login
-section of the connection. For the login user specify the following as extra:
+section of the connection. For the login user, specify the following as extra:
.. code-block:: bash
@@ -209,7 +220,7 @@ For the DAG owner use:
{ "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "owner"}
-and in your DAG, when initializing the HiveOperator, specify
+and in your DAG, when initializing the HiveOperator, specify:
.. code-block:: bash
@@ -226,9 +237,6 @@ against an installation of GitHub Enterprise using OAuth2. You can optionally
specify a team whitelist (composed of slug cased team names) to restrict login
to only members of those teams.
-*NOTE* If you do not specify a team whitelist, anyone with a valid account on
-your GHE installation will be able to login to Airflow.
-
.. code-block:: bash
[webserver]
@@ -242,6 +250,9 @@ your GHE installation will be able to login to Airflow.
oauth_callback_route = /example/ghe_oauth/callback
allowed_teams = 1, 345, 23
+.. note:: If you do not specify a team whitelist, anyone with a valid account on
+ your GHE installation will be able to login to Airflow.
+
Setting up GHE Authentication
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -283,6 +294,7 @@ backend. In order to setup an application:
1. Navigate to https://console.developers.google.com/apis/
2. Select 'Credentials' from the left hand nav
+2. Select 'Credentials' from the left hand nav
3. Click 'Create credentials' and choose 'OAuth client ID'
4. Choose 'Web application'
5. Fill in the required information (the 'Authorized redirect URIs' must be fully qualifed e.g. http://airflow.example.com/oauth2callback)
@@ -311,24 +323,44 @@ standard port 443, you'll need to configure that too. Be aware that super user p
web_server_port = 443
base_url = http://<hostname or IP>:443
+Enable CeleryExecutor with SSL. Ensure you properly generate client and server
+certs and keys.
+
+.. code-block:: bash
+
+ [celery]
+ CELERY_SSL_ACTIVE = True
+ CELERY_SSL_KEY = <path to key>
+ CELERY_SSL_CERT = <path to cert>
+ CELERY_SSL_CACERT = <path to cacert>
+
Impersonation
-'''''''''''''
+-------------
Airflow has the ability to impersonate a unix user while running task
instances based on the task's ``run_as_user`` parameter, which takes a user's name.
-*NOTE* For impersonations to work, Airflow must be run with `sudo` as subtasks are run
+**NOTE:** For impersonations to work, Airflow must be run with `sudo` as subtasks are run
with `sudo -u` and permissions of files are changed. Furthermore, the unix user needs to
exist on the worker. Here is what a simple sudoers file entry could look like to achieve
this, assuming as airflow is running as the `airflow` user. Note that this means that
the airflow user must be trusted and treated the same way as the root user.
.. code-block:: none
+
airflow ALL=(ALL) NOPASSWD: ALL
+
Subtasks with impersonation will still log to the same folder, except that the files they
log to will have permissions changed such that only the unix user can write to it.
-*Default impersonation* To prevent tasks that don't use impersonation to be run with
-`sudo` privileges, you can set the `default_impersonation` config in `core` which sets a
-default user impersonate if `run_as_user` is not set.
+Default Impersonation
+'''''''''''''''''''''
+To prevent tasks that don't use impersonation to be run with `sudo` privileges, you can set the
+``core:default_impersonation`` config which sets a default user impersonate if `run_as_user` is
+not set.
+
+.. code-block:: bash
+
+ [core]
+ default_impersonation = airflow
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/start.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/start.rst.txt b/_sources/start.rst.txt
index cc41d4b..36128f5 100644
--- a/_sources/start.rst.txt
+++ b/_sources/start.rst.txt
@@ -11,7 +11,7 @@ The installation is quick and straightforward.
export AIRFLOW_HOME=~/airflow
# install from pypi using pip
- pip install airflow
+ pip install apache-airflow
# initialize the database
airflow initdb
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_sources/tutorial.rst.txt
----------------------------------------------------------------------
diff --git a/_sources/tutorial.rst.txt b/_sources/tutorial.rst.txt
index 97bbe11..8d2203c 100644
--- a/_sources/tutorial.rst.txt
+++ b/_sources/tutorial.rst.txt
@@ -26,7 +26,7 @@ complicated, a line by line explanation follows below.
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2015, 6, 1),
- 'email': ['airflow@airflow.com'],
+ 'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
@@ -117,7 +117,7 @@ of default parameters that we can use when creating tasks.
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2015, 6, 1),
- 'email': ['airflow@airflow.com'],
+ 'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
@@ -195,7 +195,8 @@ templates.
This tutorial barely scratches the surface of what you can do with
templating in Airflow, but the goal of this section is to let you know
this feature exists, get you familiar with double curly brackets, and
-point to the most common template variable: ``{{ ds }}``.
+point to the most common template variable: ``{{ ds }}`` (today's "date
+stamp").
.. code:: python
@@ -231,6 +232,16 @@ different languages, and general flexibility in structuring pipelines. It is
also possible to define your ``template_searchpath`` as pointing to any folder
locations in the DAG constructor call.
+Using that same DAG constructor call, it is possible to define
+``user_defined_macros`` which allow you to specify your own variables.
+For example, passing ``dict(foo='bar')`` to this argument allows you
+to use ``{{ foo }}`` in your templates. Moreover, specifying
+``user_defined_filters`` allow you to register you own filters. For example,
+passing ``dict(hello=lambda name: 'Hello %s' % name)`` to this argument allows
+you to use ``{{ 'world' | hello }}`` in your templates. For more information
+regarding custom filters have a look at the
+`Jinja Documentation <http://jinja.pocoo.org/docs/dev/api/#writing-filters>`_
+
For more information on the variables and macros that can be referenced
in templates, make sure to read through the :ref:`macros` section
@@ -278,7 +289,7 @@ something like this:
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2015, 6, 1),
- 'email': ['airflow@airflow.com'],
+ 'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_static/basic.css
----------------------------------------------------------------------
diff --git a/_static/basic.css b/_static/basic.css
index 7ed0e58..3c7223b 100644
--- a/_static/basic.css
+++ b/_static/basic.css
@@ -4,7 +4,7 @@
*
* Sphinx stylesheet -- basic theme.
*
- * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
* :license: BSD, see LICENSE for details.
*
*/
@@ -398,6 +398,13 @@ table.field-list td, table.field-list th {
margin: 0;
}
+.field-name {
+ -moz-hyphens: manual;
+ -ms-hyphens: manual;
+ -webkit-hyphens: manual;
+ hyphens: manual;
+}
+
/* -- other body styles ----------------------------------------------------- */
ol.arabic {
@@ -438,10 +445,14 @@ dd {
margin-left: 30px;
}
-dt:target, .highlighted {
+dt:target, span.highlighted {
background-color: #fbe54e;
}
+rect.highlighted {
+ fill: #fbe54e;
+}
+
dl.glossary dt {
font-weight: bold;
font-size: 1.1em;
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_static/doctools.js
----------------------------------------------------------------------
diff --git a/_static/doctools.js b/_static/doctools.js
index 8163495..24992e6 100644
--- a/_static/doctools.js
+++ b/_static/doctools.js
@@ -4,7 +4,7 @@
*
* Sphinx JavaScript utilities for all documentation.
*
- * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
* :license: BSD, see LICENSE for details.
*
*/
@@ -45,7 +45,7 @@ jQuery.urlencode = encodeURIComponent;
* it will always return arrays of strings for the value parts.
*/
jQuery.getQueryParameters = function(s) {
- if (typeof s == 'undefined')
+ if (typeof s === 'undefined')
s = document.location.search;
var parts = s.substr(s.indexOf('?') + 1).split('&');
var result = {};
@@ -66,29 +66,53 @@ jQuery.getQueryParameters = function(s) {
* span elements with the given class name.
*/
jQuery.fn.highlightText = function(text, className) {
- function highlight(node) {
- if (node.nodeType == 3) {
+ function highlight(node, addItems) {
+ if (node.nodeType === 3) {
var val = node.nodeValue;
var pos = val.toLowerCase().indexOf(text);
if (pos >= 0 && !jQuery(node.parentNode).hasClass(className)) {
- var span = document.createElement("span");
- span.className = className;
+ var span;
+ var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg");
+ if (isInSVG) {
+ span = document.createElementNS("http://www.w3.org/2000/svg", "tspan");
+ } else {
+ span = document.createElement("span");
+ span.className = className;
+ }
span.appendChild(document.createTextNode(val.substr(pos, text.length)));
node.parentNode.insertBefore(span, node.parentNode.insertBefore(
document.createTextNode(val.substr(pos + text.length)),
node.nextSibling));
node.nodeValue = val.substr(0, pos);
+ if (isInSVG) {
+ var bbox = span.getBBox();
+ var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect");
+ rect.x.baseVal.value = bbox.x;
+ rect.y.baseVal.value = bbox.y;
+ rect.width.baseVal.value = bbox.width;
+ rect.height.baseVal.value = bbox.height;
+ rect.setAttribute('class', className);
+ var parentOfText = node.parentNode.parentNode;
+ addItems.push({
+ "parent": node.parentNode,
+ "target": rect});
+ }
}
}
else if (!jQuery(node).is("button, select, textarea")) {
jQuery.each(node.childNodes, function() {
- highlight(this);
+ highlight(this, addItems);
});
}
}
- return this.each(function() {
- highlight(this);
+ var addItems = [];
+ var result = this.each(function() {
+ highlight(this, addItems);
});
+ for (var i = 0; i < addItems.length; ++i) {
+ jQuery(addItems[i].parent).before(addItems[i].target);
+ }
+ return result;
};
/*
@@ -131,21 +155,21 @@ var Documentation = {
* i18n support
*/
TRANSLATIONS : {},
- PLURAL_EXPR : function(n) { return n == 1 ? 0 : 1; },
+ PLURAL_EXPR : function(n) { return n === 1 ? 0 : 1; },
LOCALE : 'unknown',
// gettext and ngettext don't access this so that the functions
// can safely bound to a different name (_ = Documentation.gettext)
gettext : function(string) {
var translated = Documentation.TRANSLATIONS[string];
- if (typeof translated == 'undefined')
+ if (typeof translated === 'undefined')
return string;
- return (typeof translated == 'string') ? translated : translated[0];
+ return (typeof translated === 'string') ? translated : translated[0];
},
ngettext : function(singular, plural, n) {
var translated = Documentation.TRANSLATIONS[singular];
- if (typeof translated == 'undefined')
+ if (typeof translated === 'undefined')
return (n == 1) ? singular : plural;
return translated[Documentation.PLURALEXPR(n)];
},
@@ -216,7 +240,7 @@ var Documentation = {
var src = $(this).attr('src');
var idnum = $(this).attr('id').substr(7);
$('tr.cg-' + idnum).toggle();
- if (src.substr(-9) == 'minus.png')
+ if (src.substr(-9) === 'minus.png')
$(this).attr('src', src.substr(0, src.length-9) + 'plus.png');
else
$(this).attr('src', src.substr(0, src.length-8) + 'minus.png');
@@ -248,7 +272,7 @@ var Documentation = {
var path = document.location.pathname;
var parts = path.split(/\//);
$.each(DOCUMENTATION_OPTIONS.URL_ROOT.split(/\//), function() {
- if (this == '..')
+ if (this === '..')
parts.pop();
});
var url = parts.join('/');
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_static/pygments.css
----------------------------------------------------------------------
diff --git a/_static/pygments.css b/_static/pygments.css
index 8213e90..20c4814 100644
--- a/_static/pygments.css
+++ b/_static/pygments.css
@@ -47,8 +47,10 @@
.highlight .mh { color: #208050 } /* Literal.Number.Hex */
.highlight .mi { color: #208050 } /* Literal.Number.Integer */
.highlight .mo { color: #208050 } /* Literal.Number.Oct */
+.highlight .sa { color: #4070a0 } /* Literal.String.Affix */
.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
.highlight .sc { color: #4070a0 } /* Literal.String.Char */
+.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */
.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
.highlight .s2 { color: #4070a0 } /* Literal.String.Double */
.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
@@ -59,7 +61,9 @@
.highlight .s1 { color: #4070a0 } /* Literal.String.Single */
.highlight .ss { color: #517918 } /* Literal.String.Symbol */
.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
+.highlight .fm { color: #06287e } /* Name.Function.Magic */
.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
+.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */
.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_static/searchtools.js
----------------------------------------------------------------------
diff --git a/_static/searchtools.js b/_static/searchtools.js
index bbfb3ac..33fedf4 100644
--- a/_static/searchtools.js
+++ b/_static/searchtools.js
@@ -4,7 +4,7 @@
*
* Sphinx JavaScript utilities for the full-text search.
*
- * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
* :license: BSD, see LICENSE for details.
*
*/
@@ -540,6 +540,9 @@ var Search = {
});
} else if (DOCUMENTATION_OPTIONS.HAS_SOURCE) {
var suffix = DOCUMENTATION_OPTIONS.SOURCELINK_SUFFIX;
+ if (suffix === undefined) {
+ suffix = '.txt';
+ }
$.ajax({url: DOCUMENTATION_OPTIONS.URL_ROOT + '_sources/' + item[5] + (item[5].slice(-suffix.length) === suffix ? '' : suffix),
dataType: "text",
complete: function(jqxhr, textstatus) {
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_static/websupport.js
----------------------------------------------------------------------
diff --git a/_static/websupport.js b/_static/websupport.js
index 98e7f40..53f6a45 100644
--- a/_static/websupport.js
+++ b/_static/websupport.js
@@ -4,7 +4,7 @@
*
* sphinx.websupport utilities for all documentation.
*
- * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
* :license: BSD, see LICENSE for details.
*
*/
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/api.html
----------------------------------------------------------------------
diff --git a/api.html b/api.html
index 7aca3bb..e9244a0 100644
--- a/api.html
+++ b/api.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -191,16 +196,23 @@ configure as follows:</p>
</div>
<div class="section" id="authentication">
<h2>Authentication<a class="headerlink" href="#authentication" title="Permalink to this headline">¶</a></h2>
-<p>Only Kerberos authentication is currently supported for the API. To enable this set the following
-in the configuration:</p>
-<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>api<span class="o">]</span>
-<span class="nv">auth_backend</span> <span class="o">=</span> airflow.api.auth.backend.default
+<p>Authentication for the API is handled separately to the Web Authentication. The default is to not
+require any authentication on the API – i.e. wide open by default. This is not recommended if your
+Airflow webserver is publicly accessible, and you should probably use the deny all backend:</p>
+<div class="highlight-ini"><div class="highlight"><pre><span></span><span class="k">[api]</span>
+<span class="na">auth_backend</span> <span class="o">=</span> <span class="s">airflow.api.auth.backend.deny_all</span>
+</pre></div>
+</div>
+<p>Kerberos is the only “real” authentication mechanism currently supported for the API. To enable
+this set the following in the configuration:</p>
+<div class="highlight-ini"><div class="highlight"><pre><span></span><span class="k">[api]</span>
+<span class="na">auth_backend</span> <span class="o">=</span> <span class="s">airflow.api.auth.backend.kerberos_auth</span>
-<span class="o">[</span>kerberos<span class="o">]</span>
-<span class="nv">keytab</span> <span class="o">=</span> <KEYTAB>
+<span class="k">[kerberos]</span>
+<span class="na">keytab</span> <span class="o">=</span> <span class="s"><KEYTAB></span>
</pre></div>
</div>
-<p>The Kerberos service is configured as <cite>airflow/fully.qualified.domainname@REALM</cite>. Make sure this
+<p>The Kerberos service is configured as <code class="docutils literal"><span class="pre">airflow/fully.qualified.domainname@REALM</span></code>. Make sure this
principal exists in the keytab file.</p>
</div>
</div>
[17/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/dagrun_operator.html
----------------------------------------------------------------------
diff --git a/_modules/dagrun_operator.html b/_modules/dagrun_operator.html
index ec09c2c..8dd70d8 100644
--- a/_modules/dagrun_operator.html
+++ b/_modules/dagrun_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -177,13 +182,11 @@
<span class="c1"># limitations under the License.</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span>
-<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">DagBag</span>
<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="k">import</span> <span class="n">State</span>
<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">settings</span>
-<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">configuration</span> <span class="k">as</span> <span class="n">conf</span>
<span class="k">class</span> <span class="nc">DagRunOrder</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
@@ -219,12 +222,12 @@
<span class="n">trigger_dag_id</span><span class="p">,</span>
<span class="n">python_callable</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">TriggerDagRunOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TriggerDagRunOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span> <span class="o">=</span> <span class="n">python_callable</span>
<span class="bp">self</span><span class="o">.</span><span class="n">trigger_dag_id</span> <span class="o">=</span> <span class="n">trigger_dag_id</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
- <span class="n">dro</span> <span class="o">=</span> <span class="n">DagRunOrder</span><span class="p">(</span><span class="n">run_id</span><span class="o">=</span><span class="s1">'trig__'</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">isoformat</span><span class="p">())</span>
+ <span class="n">dro</span> <span class="o">=</span> <span class="n">DagRunOrder</span><span class="p">(</span><span class="n">run_id</span><span class="o">=</span><span class="s1">'trig__'</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">isoformat</span><span class="p">())</span>
<span class="n">dro</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">dro</span><span class="p">)</span>
<span class="k">if</span> <span class="n">dro</span><span class="p">:</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
@@ -235,12 +238,12 @@
<span class="n">state</span><span class="o">=</span><span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span><span class="p">,</span>
<span class="n">conf</span><span class="o">=</span><span class="n">dro</span><span class="o">.</span><span class="n">payload</span><span class="p">,</span>
<span class="n">external_trigger</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Creating DagRun </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">dr</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Creating DagRun </span><span class="si">%s</span><span class="s2">"</span><span class="p">,</span> <span class="n">dr</span><span class="p">)</span>
<span class="n">session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">dr</span><span class="p">)</span>
<span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Criteria not met, moving on"</span><span class="p">)</span></div>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Criteria not met, moving on"</span><span class="p">)</span></div>
</pre></div>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/dbapi_hook.html
----------------------------------------------------------------------
diff --git a/_modules/dbapi_hook.html b/_modules/dbapi_hook.html
index ae37fd3..34111f9 100644
--- a/_modules/dbapi_hook.html
+++ b/_modules/dbapi_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -179,8 +184,7 @@
<span class="kn">from</span> <span class="nn">builtins</span> <span class="k">import</span> <span class="nb">str</span>
<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="k">import</span> <span class="n">basestring</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span>
-<span class="kn">import</span> <span class="nn">numpy</span>
-<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">contextlib</span> <span class="k">import</span> <span class="n">closing</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="k">import</span> <span class="n">create_engine</span>
@@ -251,10 +255,9 @@
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">pandas.io.sql</span> <span class="k">as</span> <span class="nn">psql</span>
- <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
- <span class="n">df</span> <span class="o">=</span> <span class="n">psql</span><span class="o">.</span><span class="n">read_sql</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">con</span><span class="o">=</span><span class="n">conn</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">parameters</span><span class="p">)</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">df</span></div>
+
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">())</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">psql</span><span class="o">.</span><span class="n">read_sql</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">con</span><span class="o">=</span><span class="n">conn</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">parameters</span><span class="p">)</span></div>
<div class="viewcode-block" id="DbApiHook.get_records"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_records">[docs]</a> <span class="k">def</span> <span class="nf">get_records</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -268,16 +271,14 @@
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
- <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
- <span class="n">cur</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_cursor</span><span class="p">()</span>
- <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
- <span class="n">rows</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">rows</span></div>
+
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">())</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">())</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span></div>
<div class="viewcode-block" id="DbApiHook.get_first"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_first">[docs]</a> <span class="k">def</span> <span class="nf">get_first</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -291,16 +292,14 @@
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
- <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
- <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
- <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
- <span class="n">rows</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="k">return</span> <span class="n">rows</span></div>
+
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">())</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">())</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span></div>
<div class="viewcode-block" id="DbApiHook.run"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.run">[docs]</a> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">autocommit</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="sd">"""</span>
@@ -317,25 +316,24 @@
<span class="sd"> :param parameters: The parameters to render the SQL query with.</span>
<span class="sd"> :type parameters: mapping or iterable</span>
<span class="sd"> """</span>
- <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">basestring</span><span class="p">):</span>
<span class="n">sql</span> <span class="o">=</span> <span class="p">[</span><span class="n">sql</span><span class="p">]</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">set_autocommit</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">autocommit</span><span class="p">)</span>
-
- <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
- <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">sql</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
- <span class="n">s</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">())</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">set_autocommit</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">autocommit</span><span class="p">)</span>
+
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">())</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">sql</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o"><</span> <span class="mi">3</span><span class="p">:</span>
+ <span class="n">s</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
+
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span></div>
<span class="k">def</span> <span class="nf">set_autocommit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn</span><span class="p">,</span> <span class="n">autocommit</span><span class="p">):</span>
<span class="n">conn</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
@@ -349,7 +347,7 @@
<div class="viewcode-block" id="DbApiHook.insert_rows"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.insert_rows">[docs]</a> <span class="k">def</span> <span class="nf">insert_rows</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">target_fields</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">commit_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> A generic way to insert a set of tuples into a table,</span>
-<span class="sd"> the whole set of inserts is treated as one transaction</span>
+<span class="sd"> a new transaction is created every commit_every rows</span>
<span class="sd"> :param table: Name of the target table</span>
<span class="sd"> :type table: str</span>
@@ -366,31 +364,33 @@
<span class="n">target_fields</span> <span class="o">=</span> <span class="s2">"(</span><span class="si">{}</span><span class="s2">)"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">target_fields</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">target_fields</span> <span class="o">=</span> <span class="s1">''</span>
- <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
- <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span><span class="p">:</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">set_autocommit</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
- <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
- <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
- <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
- <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
- <span class="n">l</span> <span class="o">=</span> <span class="p">[]</span>
- <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">:</span>
- <span class="n">l</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">conn</span><span class="p">))</span>
- <span class="n">values</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
- <span class="n">sql</span> <span class="o">=</span> <span class="s2">"INSERT INTO </span><span class="si">{0}</span><span class="s2"> </span><span class="si">{1}</span><span class="s2"> VALUES (</span><span class="si">{2}</span><span class="s2">);"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
- <span class="n">table</span><span class="p">,</span>
- <span class="n">target_fields</span><span class="p">,</span>
- <span class="s2">","</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">values</span><span class="p">))</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">commit_every</span> <span class="ow">and</span> <span class="n">i</span> <span class="o">%</span> <span class="n">commit_every</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
- <span class="s2">"Loaded </span><span class="si">{i}</span><span class="s2"> into </span><span class="si">{table}</span><span class="s2"> rows so far"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
- <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">())</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">set_autocommit</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="kc">False</span><span class="p">)</span>
+
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+
+ <span class="k">with</span> <span class="n">closing</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">())</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">row</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">rows</span><span class="p">,</span> <span class="mi">1</span><span class="p">):</span>
+ <span class="n">l</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">:</span>
+ <span class="n">l</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">conn</span><span class="p">))</span>
+ <span class="n">values</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
+ <span class="n">placeholders</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"</span><span class="si">%s</span><span class="s2">"</span><span class="p">,]</span><span class="o">*</span><span class="nb">len</span><span class="p">(</span><span class="n">values</span><span class="p">)</span>
+ <span class="n">sql</span> <span class="o">=</span> <span class="s2">"INSERT INTO </span><span class="si">{0}</span><span class="s2"> </span><span class="si">{1}</span><span class="s2"> VALUES (</span><span class="si">{2}</span><span class="s2">);"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">table</span><span class="p">,</span>
+ <span class="n">target_fields</span><span class="p">,</span>
+ <span class="s2">","</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">placeholders</span><span class="p">))</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">values</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">commit_every</span> <span class="ow">and</span> <span class="n">i</span> <span class="o">%</span> <span class="n">commit_every</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Loaded </span><span class="si">{i}</span><span class="s2"> into </span><span class="si">{table}</span><span class="s2"> rows so far"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="p">)</span>
+
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s2">"Done loading. Loaded a total of </span><span class="si">{i}</span><span class="s2"> rows"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
<span class="nd">@staticmethod</span>
@@ -406,16 +406,11 @@
<span class="sd"> :rtype: str</span>
<span class="sd"> """</span>
- <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">basestring</span><span class="p">):</span>
- <span class="k">return</span> <span class="s2">"'"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"'"</span><span class="p">,</span> <span class="s2">"''"</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"'"</span>
- <span class="k">elif</span> <span class="n">cell</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
- <span class="k">return</span> <span class="s1">'NULL'</span>
- <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">numpy</span><span class="o">.</span><span class="n">datetime64</span><span class="p">):</span>
- <span class="k">return</span> <span class="s2">"'"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"'"</span>
- <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">datetime</span><span class="p">):</span>
- <span class="k">return</span> <span class="s2">"'"</span> <span class="o">+</span> <span class="n">cell</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"'"</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">cell</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="k">return</span> <span class="kc">None</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">datetime</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">cell</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span>
+ <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span>
<div class="viewcode-block" id="DbApiHook.bulk_dump"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.bulk_dump">[docs]</a> <span class="k">def</span> <span class="nf">bulk_dump</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">tmp_file</span><span class="p">):</span>
<span class="sd">"""</span>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/druid_hook.html
----------------------------------------------------------------------
diff --git a/_modules/druid_hook.html b/_modules/druid_hook.html
index 20ffb1d..5bb5145 100644
--- a/_modules/druid_hook.html
+++ b/_modules/druid_hook.html
@@ -13,6 +13,8 @@
+
+
@@ -30,6 +32,9 @@
+ <link rel="index" title="Index"
+ href="../genindex.html"/>
+ <link rel="search" title="Search" href="../search.html"/>
<link rel="top" title="Airflow Documentation" href="../index.html"/>
<link rel="up" title="Module code" href="index.html"/>
@@ -40,6 +45,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -76,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -90,6 +99,8 @@
<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
</ul>
@@ -104,8 +115,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="../index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+
</nav>
@@ -118,19 +131,36 @@
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="../index.html">Docs</a> »</li>
-
+
+ <li><a href="../index.html">Docs</a> »</li>
+
<li><a href="index.html">Module code</a> »</li>
-
- <li>druid_hook</li>
+
+ <li>druid_hook</li>
+
+
<li class="wy-breadcrumbs-aside">
-
+
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -151,180 +181,90 @@
<span class="c1"># See the License for the specific language governing permissions and</span>
<span class="c1"># limitations under the License.</span>
-<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
-<span class="kn">import</span> <span class="nn">logging</span>
-<span class="kn">import</span> <span class="nn">json</span>
-<span class="kn">import</span> <span class="nn">time</span>
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="k">import</span> <span class="n">print_function</span>
-<span class="kn">from</span> <span class="nn">pydruid.client</span> <span class="kn">import</span> <span class="n">PyDruid</span>
<span class="kn">import</span> <span class="nn">requests</span>
+<span class="kn">import</span> <span class="nn">time</span>
-<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
-<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
-
-<span class="n">LOAD_CHECK_INTERVAL</span> <span class="o">=</span> <span class="mi">5</span>
-<span class="n">DEFAULT_TARGET_PARTITION_SIZE</span> <span class="o">=</span> <span class="mi">5000000</span>
-
-
-<span class="k">class</span> <span class="nc">AirflowDruidLoadException</span><span class="p">(</span><span class="n">AirflowException</span><span class="p">):</span>
- <span class="k">pass</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
<div class="viewcode-block" id="DruidHook"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook">[docs]</a><span class="k">class</span> <span class="nc">DruidHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
- <span class="sd">'''</span>
-<span class="sd"> Interact with druid.</span>
-<span class="sd"> '''</span>
-
+ <span class="sd">"""</span>
+<span class="sd"> Connection to Druid</span>
+
+<span class="sd"> :param druid_ingest_conn_id: The connection id to the Druid overlord machine which accepts index jobs</span>
+<span class="sd"> :type druid_ingest_conn_id: string</span>
+<span class="sd"> :param timeout: The interval between polling the Druid job for the status of the ingestion job</span>
+<span class="sd"> :type timeout: int</span>
+<span class="sd"> :param max_ingestion_time: The maximum ingestion time before assuming the job failed</span>
+<span class="sd"> :type max_ingestion_time: int</span>
+<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
- <span class="n">druid_query_conn_id</span><span class="o">=</span><span class="s1">'druid_query_default'</span><span class="p">,</span>
- <span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="s1">'druid_ingest_default'</span><span class="p">):</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">druid_query_conn_id</span> <span class="o">=</span> <span class="n">druid_query_conn_id</span>
+ <span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="s1">'druid_ingest_default'</span><span class="p">,</span>
+ <span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
+ <span class="n">max_ingestion_time</span><span class="o">=</span><span class="mi">18000</span><span class="p">):</span>
+
<span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span> <span class="o">=</span> <span class="n">druid_ingest_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timeout</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">max_ingestion_time</span> <span class="o">=</span> <span class="n">max_ingestion_time</span>
<span class="bp">self</span><span class="o">.</span><span class="n">header</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'content-type'</span><span class="p">:</span> <span class="s1">'application/json'</span><span class="p">}</span>
-<div class="viewcode-block" id="DruidHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Returns a druid connection object for query</span>
-<span class="sd"> """</span>
- <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_query_conn_id</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">PyDruid</span><span class="p">(</span>
- <span class="s2">"http://{conn.host}:{conn.port}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()),</span>
- <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">''</span><span class="p">))</span></div>
-
- <span class="nd">@property</span>
- <span class="k">def</span> <span class="nf">ingest_post_url</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">get_conn_url</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span><span class="p">)</span>
<span class="n">host</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span>
+ <span class="n">schema</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'schema'</span><span class="p">,</span> <span class="s1">'http'</span><span class="p">)</span>
<span class="n">endpoint</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
- <span class="k">return</span> <span class="s2">"http://{host}:{port}/{endpoint}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
-
- <span class="k">def</span> <span class="nf">get_ingest_status_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_id</span><span class="p">):</span>
- <span class="n">post_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ingest_post_url</span>
- <span class="k">return</span> <span class="s2">"{post_url}/{task_id}/status"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
-
-<div class="viewcode-block" id="DruidHook.construct_ingest_query"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook.construct_ingest_query">[docs]</a> <span class="k">def</span> <span class="nf">construct_ingest_query</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span> <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span> <span class="n">metric_spec</span><span class="p">,</span>
- <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> Builds an ingest query for an HDFS TSV load.</span>
-
-<span class="sd"> :param datasource: target datasource in druid</span>
-<span class="sd"> :param columns: list of all columns in the TSV, in the right order</span>
-<span class="sd"> """</span>
-
- <span class="c1"># backward compatibilty for num_shards, but target_partition_size is the default setting</span>
- <span class="c1"># and overwrites the num_shards</span>
- <span class="k">if</span> <span class="n">target_partition_size</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
- <span class="k">if</span> <span class="n">num_shards</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
- <span class="n">target_partition_size</span> <span class="o">=</span> <span class="n">DEFAULT_TARGET_PARTITION_SIZE</span>
- <span class="k">else</span><span class="p">:</span>
- <span class="n">num_shards</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
-
- <span class="n">metric_names</span> <span class="o">=</span> <span class="p">[</span>
- <span class="n">m</span><span class="p">[</span><span class="s1">'fieldName'</span><span class="p">]</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">metric_spec</span> <span class="k">if</span> <span class="n">m</span><span class="p">[</span><span class="s1">'type'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'count'</span><span class="p">]</span>
- <span class="n">dimensions</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">columns</span> <span class="k">if</span> <span class="n">c</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">metric_names</span> <span class="ow">and</span> <span class="n">c</span> <span class="o">!=</span> <span class="n">ts_dim</span><span class="p">]</span>
- <span class="n">ingest_query_dict</span> <span class="o">=</span> <span class="p">{</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"index_hadoop"</span><span class="p">,</span>
- <span class="s2">"spec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"dataSchema"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"metricsSpec"</span><span class="p">:</span> <span class="n">metric_spec</span><span class="p">,</span>
- <span class="s2">"granularitySpec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"queryGranularity"</span><span class="p">:</span> <span class="s2">"NONE"</span><span class="p">,</span>
- <span class="s2">"intervals"</span><span class="p">:</span> <span class="n">intervals</span><span class="p">,</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"uniform"</span><span class="p">,</span>
- <span class="s2">"segmentGranularity"</span><span class="p">:</span> <span class="s2">"DAY"</span><span class="p">,</span>
- <span class="p">},</span>
- <span class="s2">"parser"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"string"</span><span class="p">,</span>
- <span class="s2">"parseSpec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"columns"</span><span class="p">:</span> <span class="n">columns</span><span class="p">,</span>
- <span class="s2">"dimensionsSpec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"dimensionExclusions"</span><span class="p">:</span> <span class="p">[],</span>
- <span class="s2">"dimensions"</span><span class="p">:</span> <span class="n">dimensions</span><span class="p">,</span> <span class="c1"># list of names</span>
- <span class="s2">"spatialDimensions"</span><span class="p">:</span> <span class="p">[]</span>
- <span class="p">},</span>
- <span class="s2">"timestampSpec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"column"</span><span class="p">:</span> <span class="n">ts_dim</span><span class="p">,</span>
- <span class="s2">"format"</span><span class="p">:</span> <span class="s2">"auto"</span>
- <span class="p">},</span>
- <span class="s2">"format"</span><span class="p">:</span> <span class="s2">"tsv"</span>
- <span class="p">}</span>
- <span class="p">},</span>
- <span class="s2">"dataSource"</span><span class="p">:</span> <span class="n">datasource</span>
- <span class="p">},</span>
- <span class="s2">"tuningConfig"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hadoop"</span><span class="p">,</span>
- <span class="s2">"jobProperties"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"mapreduce.job.user.classpath.first"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
- <span class="s2">"mapreduce.map.output.compress"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
- <span class="s2">"mapreduce.output.fileoutputformat.compress"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
- <span class="p">},</span>
- <span class="s2">"partitionsSpec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hashed"</span><span class="p">,</span>
- <span class="s2">"targetPartitionSize"</span><span class="p">:</span> <span class="n">target_partition_size</span><span class="p">,</span>
- <span class="s2">"numShards"</span><span class="p">:</span> <span class="n">num_shards</span><span class="p">,</span>
- <span class="p">},</span>
- <span class="p">},</span>
- <span class="s2">"ioConfig"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"inputSpec"</span><span class="p">:</span> <span class="p">{</span>
- <span class="s2">"paths"</span><span class="p">:</span> <span class="n">static_path</span><span class="p">,</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"static"</span>
- <span class="p">},</span>
- <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hadoop"</span>
- <span class="p">}</span>
- <span class="p">}</span>
- <span class="p">}</span>
- <span class="k">if</span> <span class="n">hadoop_dependency_coordinates</span><span class="p">:</span>
- <span class="n">ingest_query_dict</span><span class="p">[</span>
- <span class="s1">'hadoopDependencyCoordinates'</span><span class="p">]</span> <span class="o">=</span> <span class="n">hadoop_dependency_coordinates</span>
-
- <span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">ingest_query_dict</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span></div>
-
- <span class="k">def</span> <span class="nf">send_ingest_query</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span> <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span> <span class="n">metric_spec</span><span class="p">,</span>
- <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
- <span class="n">query</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">construct_ingest_query</span><span class="p">(</span>
- <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span>
- <span class="n">metric_spec</span><span class="p">,</span> <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="p">)</span>
- <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
- <span class="bp">self</span><span class="o">.</span><span class="n">ingest_post_url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">header</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ingest_post_url</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
- <span class="n">d</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
- <span class="k">if</span> <span class="s2">"task"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">d</span><span class="p">:</span>
- <span class="k">raise</span> <span class="n">AirflowDruidLoadException</span><span class="p">(</span>
- <span class="s2">"[Error]: Ingesting data to druid failed."</span><span class="p">)</span>
- <span class="k">return</span> <span class="n">d</span><span class="p">[</span><span class="s2">"task"</span><span class="p">]</span>
-
-<div class="viewcode-block" id="DruidHook.load_from_hdfs"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook.load_from_hdfs">[docs]</a> <span class="k">def</span> <span class="nf">load_from_hdfs</span><span class="p">(</span>
- <span class="bp">self</span><span class="p">,</span> <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span>
- <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">metric_spec</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
- <span class="sd">"""</span>
-<span class="sd"> load data to druid from hdfs</span>
-
-<span class="sd"> :param ts_dim: The column name to use as a timestamp</span>
-<span class="sd"> :param metric_spec: A list of dictionaries</span>
-<span class="sd"> """</span>
- <span class="n">task_id</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">send_ingest_query</span><span class="p">(</span>
- <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span> <span class="n">metric_spec</span><span class="p">,</span>
- <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="p">)</span>
- <span class="n">status_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_ingest_status_url</span><span class="p">(</span><span class="n">task_id</span><span class="p">)</span>
- <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
- <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">status_url</span><span class="p">)</span>
- <span class="n">d</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
- <span class="k">if</span> <span class="n">d</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'FAILED'</span><span class="p">:</span>
- <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
- <span class="k">raise</span> <span class="n">AirflowDruidLoadException</span><span class="p">(</span>
- <span class="s2">"[Error]: Ingesting data to druid failed."</span><span class="p">)</span>
- <span class="k">elif</span> <span class="n">d</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'SUCCESS'</span><span class="p">:</span>
- <span class="k">break</span>
- <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">LOAD_CHECK_INTERVAL</span><span class="p">)</span></div></div>
+ <span class="k">return</span> <span class="s2">"http://</span><span class="si">{host}</span><span class="s2">:</span><span class="si">{port}</span><span class="s2">/</span><span class="si">{endpoint}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+
+ <span class="k">def</span> <span class="nf">submit_indexing_job</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">json_index_spec</span><span class="p">):</span>
+ <span class="n">url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn_url</span><span class="p">()</span>
+
+ <span class="n">req_index</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">json_index_spec</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">header</span><span class="p">)</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">req_index</span><span class="o">.</span><span class="n">status_code</span> <span class="o">!=</span> <span class="mi">200</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Did not get 200 when submitting the Druid job to </span><span class="si">{}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">url</span><span class="p">))</span>
+
+ <span class="n">req_json</span> <span class="o">=</span> <span class="n">req_index</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
+ <span class="c1"># Wait until the job is completed</span>
+ <span class="n">druid_task_id</span> <span class="o">=</span> <span class="n">req_json</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span>
+
+ <span class="n">running</span> <span class="o">=</span> <span class="kc">True</span>
+
+ <span class="n">sec</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">while</span> <span class="n">running</span><span class="p">:</span>
+ <span class="n">req_status</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"</span><span class="si">{0}</span><span class="s2">/</span><span class="si">{1}</span><span class="s2">/status"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">druid_task_id</span><span class="p">))</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Job still running for </span><span class="si">%s</span><span class="s2"> seconds..."</span><span class="p">,</span> <span class="n">sec</span><span class="p">)</span>
+
+ <span class="n">sec</span> <span class="o">=</span> <span class="n">sec</span> <span class="o">+</span> <span class="mi">1</span>
+
+ <span class="k">if</span> <span class="n">sec</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">max_ingestion_time</span><span class="p">:</span>
+ <span class="c1"># ensure that the job gets killed if the max ingestion time is exceeded</span>
+ <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="s2">"</span><span class="si">{0}</span><span class="s2">/</span><span class="si">{1}</span><span class="s2">/shutdown"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">druid_task_id</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Druid ingestion took more than </span><span class="si">%s</span><span class="s1"> seconds'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_ingestion_time</span><span class="p">)</span>
+
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">)</span>
+
+ <span class="n">status</span> <span class="o">=</span> <span class="n">req_status</span><span class="o">.</span><span class="n">json</span><span class="p">()[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'status'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">status</span> <span class="o">==</span> <span class="s1">'RUNNING'</span><span class="p">:</span>
+ <span class="n">running</span> <span class="o">=</span> <span class="kc">True</span>
+ <span class="k">elif</span> <span class="n">status</span> <span class="o">==</span> <span class="s1">'SUCCESS'</span><span class="p">:</span>
+ <span class="n">running</span> <span class="o">=</span> <span class="kc">False</span> <span class="c1"># Great success!</span>
+ <span class="k">elif</span> <span class="n">status</span> <span class="o">==</span> <span class="s1">'FAILED'</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Druid indexing job failed, check console for more info'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Could not get status of the job, got </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">status</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Successful index'</span><span class="p">)</span></div>
</pre></div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
@@ -357,7 +297,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/dummy_operator.html
----------------------------------------------------------------------
diff --git a/_modules/dummy_operator.html b/_modules/dummy_operator.html
index a5452a1..e69044b 100644
--- a/_modules/dummy_operator.html
+++ b/_modules/dummy_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -191,7 +196,7 @@
<span class="nd">@apply_defaults</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">DummyOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DummyOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
<span class="k">pass</span></div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/email_operator.html
----------------------------------------------------------------------
diff --git a/_modules/email_operator.html b/_modules/email_operator.html
index ccb6373..45b816a 100644
--- a/_modules/email_operator.html
+++ b/_modules/email_operator.html
@@ -13,6 +13,8 @@
+
+
@@ -80,7 +82,10 @@
- <ul>
+
+
+
+ <ul>
<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
@@ -200,7 +205,7 @@
<span class="sd"> :type bcc: list or string (comma or semicolon delimited)</span>
<span class="sd"> """</span>
- <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'subject'</span><span class="p">,</span> <span class="s1">'html_content'</span><span class="p">)</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'to'</span><span class="p">,</span> <span class="s1">'subject'</span><span class="p">,</span> <span class="s1">'html_content'</span><span class="p">)</span>
<span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.html'</span><span class="p">,)</span>
<span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e6faf9'</span>
@@ -215,7 +220,7 @@
<span class="n">bcc</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">mime_subtype</span><span class="o">=</span><span class="s1">'mixed'</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
- <span class="nb">super</span><span class="p">(</span><span class="n">EmailOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">EmailOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">to</span> <span class="o">=</span> <span class="n">to</span>
<span class="bp">self</span><span class="o">.</span><span class="n">subject</span> <span class="o">=</span> <span class="n">subject</span>
<span class="bp">self</span><span class="o">.</span><span class="n">html_content</span> <span class="o">=</span> <span class="n">html_content</span>
[27/35] incubator-airflow-site git commit: 1.9.0
Posted by cr...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/ecs_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/ecs_operator.html b/_modules/airflow/contrib/operators/ecs_operator.html
new file mode 100644
index 0000000..5c181c5
--- /dev/null
+++ b/_modules/airflow/contrib/operators/ecs_operator.html
@@ -0,0 +1,360 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.ecs_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.ecs_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.ecs_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">import</span> <span class="nn">sys</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.aws_hook</span> <span class="k">import</span> <span class="n">AwsHook</span>
+
+
+<div class="viewcode-block" id="ECSOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.ecs_operator.ECSOperator">[docs]</a><span class="k">class</span> <span class="nc">ECSOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Execute a task on AWS EC2 Container Service</span>
+
+<span class="sd"> :param task_definition: the task definition name on EC2 Container Service</span>
+<span class="sd"> :type task_definition: str</span>
+<span class="sd"> :param cluster: the cluster name on EC2 Container Service</span>
+<span class="sd"> :type cluster: str</span>
+<span class="sd"> :param: overrides: the same parameter that boto3 will receive:</span>
+<span class="sd"> http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task</span>
+<span class="sd"> :type: overrides: dict</span>
+<span class="sd"> :param aws_conn_id: connection id of AWS credentials / region name. If None,</span>
+<span class="sd"> credential boto3 strategy will be used (http://boto3.readthedocs.io/en/latest/guide/configuration.html).</span>
+<span class="sd"> :type aws_conn_id: str</span>
+<span class="sd"> :param region_name: region name to use in AWS Hook. Override the region_name in connection (if provided)</span>
+<span class="sd"> """</span>
+
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f0ede4'</span>
+ <span class="n">client</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="n">arn</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'overrides'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_definition</span><span class="p">,</span> <span class="n">cluster</span><span class="p">,</span> <span class="n">overrides</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">region_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">ECSOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">region_name</span> <span class="o">=</span> <span class="n">region_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_definition</span> <span class="o">=</span> <span class="n">task_definition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cluster</span> <span class="o">=</span> <span class="n">cluster</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">overrides</span> <span class="o">=</span> <span class="n">overrides</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Running ECS Task - Task definition: </span><span class="si">%s</span><span class="s1"> - on cluster </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_definition</span><span class="p">,</span><span class="bp">self</span><span class="o">.</span><span class="n">cluster</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'ECSOperator overrides: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">overrides</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">client</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">get_client_type</span><span class="p">(</span>
+ <span class="s1">'ecs'</span><span class="p">,</span>
+ <span class="n">region_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">region_name</span>
+ <span class="p">)</span>
+
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">run_task</span><span class="p">(</span>
+ <span class="n">cluster</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">cluster</span><span class="p">,</span>
+ <span class="n">taskDefinition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">task_definition</span><span class="p">,</span>
+ <span class="n">overrides</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">overrides</span><span class="p">,</span>
+ <span class="n">startedBy</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">owner</span>
+ <span class="p">)</span>
+
+ <span class="n">failures</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="s1">'failures'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">failures</span><span class="p">)</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'ECS Task started: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">response</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">arn</span> <span class="o">=</span> <span class="n">response</span><span class="p">[</span><span class="s1">'tasks'</span><span class="p">][</span><span class="mi">0</span><span class="p">][</span><span class="s1">'taskArn'</span><span class="p">]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_wait_for_task_ended</span><span class="p">()</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">_check_success_task</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'ECS Task has been successfully executed: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">response</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_wait_for_task_ended</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">waiter</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">get_waiter</span><span class="p">(</span><span class="s1">'tasks_stopped'</span><span class="p">)</span>
+ <span class="n">waiter</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">max_attempts</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">maxsize</span> <span class="c1"># timeout is managed by airflow</span>
+ <span class="n">waiter</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span>
+ <span class="n">cluster</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">cluster</span><span class="p">,</span>
+ <span class="n">tasks</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">arn</span><span class="p">]</span>
+ <span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">_check_success_task</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">describe_tasks</span><span class="p">(</span>
+ <span class="n">cluster</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">cluster</span><span class="p">,</span>
+ <span class="n">tasks</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">arn</span><span class="p">]</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'ECS Task stopped, check status: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="n">response</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'failures'</span><span class="p">,</span> <span class="p">[]))</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
+
+ <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'tasks'</span><span class="p">]:</span>
+ <span class="n">containers</span> <span class="o">=</span> <span class="n">task</span><span class="p">[</span><span class="s1">'containers'</span><span class="p">]</span>
+ <span class="k">for</span> <span class="n">container</span> <span class="ow">in</span> <span class="n">containers</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">container</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'lastStatus'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'STOPPED'</span> <span class="ow">and</span> <span class="n">container</span><span class="p">[</span><span class="s1">'exitCode'</span><span class="p">]</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'This task is not in success state </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">task</span><span class="p">))</span>
+ <span class="k">elif</span> <span class="n">container</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'lastStatus'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'PENDING'</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'This task is still pending </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">task</span><span class="p">))</span>
+ <span class="k">elif</span> <span class="s1">'error'</span> <span class="ow">in</span> <span class="n">container</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'reason'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">():</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'This containers encounter an error during launching : </span><span class="si">{}</span><span class="s1">'</span><span class="o">.</span>
+ <span class="nb">format</span><span class="p">(</span><span class="n">container</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'reason'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span><span class="o">.</span><span class="n">lower</span><span class="p">()))</span>
+
+ <span class="k">def</span> <span class="nf">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">AwsHook</span><span class="p">(</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span>
+ <span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">stop_task</span><span class="p">(</span>
+ <span class="n">cluster</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">cluster</span><span class="p">,</span>
+ <span class="n">task</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">arn</span><span class="p">,</span>
+ <span class="n">reason</span><span class="o">=</span><span class="s1">'Task killed by the user'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">response</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/emr_add_steps_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/emr_add_steps_operator.html b/_modules/airflow/contrib/operators/emr_add_steps_operator.html
new file mode 100644
index 0000000..daac24b
--- /dev/null
+++ b/_modules/airflow/contrib/operators/emr_add_steps_operator.html
@@ -0,0 +1,291 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.emr_add_steps_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.emr_add_steps_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.emr_add_steps_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.emr_hook</span> <span class="k">import</span> <span class="n">EmrHook</span>
+
+
+<div class="viewcode-block" id="EmrAddStepsOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator">[docs]</a><span class="k">class</span> <span class="nc">EmrAddStepsOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> An operator that adds steps to an existing EMR job_flow.</span>
+
+<span class="sd"> :param job_flow_id: id of the JobFlow to add steps to</span>
+<span class="sd"> :type job_flow_name: str</span>
+<span class="sd"> :param aws_conn_id: aws connection to uses</span>
+<span class="sd"> :type aws_conn_id: str</span>
+<span class="sd"> :param steps: boto3 style steps to be added to the jobflow</span>
+<span class="sd"> :type steps: list</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'job_flow_id'</span><span class="p">]</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f9c915'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">job_flow_id</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">steps</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">EmrAddStepsOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">steps</span> <span class="o">=</span> <span class="n">steps</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span> <span class="o">=</span> <span class="n">job_flow_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">steps</span> <span class="o">=</span> <span class="n">steps</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">emr</span> <span class="o">=</span> <span class="n">EmrHook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Adding steps to </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">emr</span><span class="o">.</span><span class="n">add_job_flow_steps</span><span class="p">(</span><span class="n">JobFlowId</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span><span class="p">,</span> <span class="n">Steps</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">steps</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">response</span><span class="p">[</span><span class="s1">'ResponseMetadata'</span><span class="p">][</span><span class="s1">'HTTPStatusCode'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Adding steps failed: </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">response</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Steps </span><span class="si">%s</span><span class="s1"> added to JobFlow'</span><span class="p">,</span> <span class="n">response</span><span class="p">[</span><span class="s1">'StepIds'</span><span class="p">])</span>
+ <span class="k">return</span> <span class="n">response</span><span class="p">[</span><span class="s1">'StepIds'</span><span class="p">]</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/emr_create_job_flow_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/emr_create_job_flow_operator.html b/_modules/airflow/contrib/operators/emr_create_job_flow_operator.html
new file mode 100644
index 0000000..1e477c5
--- /dev/null
+++ b/_modules/airflow/contrib/operators/emr_create_job_flow_operator.html
@@ -0,0 +1,296 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.emr_create_job_flow_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.emr_create_job_flow_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.emr_create_job_flow_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.emr_hook</span> <span class="k">import</span> <span class="n">EmrHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+
+
+<div class="viewcode-block" id="EmrCreateJobFlowOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator">[docs]</a><span class="k">class</span> <span class="nc">EmrCreateJobFlowOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Creates an EMR JobFlow, reading the config from the EMR connection.</span>
+<span class="sd"> A dictionary of JobFlow overrides can be passed that override the config from the connection.</span>
+
+<span class="sd"> :param aws_conn_id: aws connection to uses</span>
+<span class="sd"> :type aws_conn_id: str</span>
+<span class="sd"> :param emr_conn_id: emr connection to use</span>
+<span class="sd"> :type emr_conn_id: str</span>
+<span class="sd"> :param job_flow_overrides: boto3 style arguments to override emr_connection extra</span>
+<span class="sd"> :type steps: dict</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f9c915'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">emr_conn_id</span><span class="o">=</span><span class="s1">'emr_default'</span><span class="p">,</span>
+ <span class="n">job_flow_overrides</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">EmrCreateJobFlowOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">emr_conn_id</span> <span class="o">=</span> <span class="n">emr_conn_id</span>
+ <span class="k">if</span> <span class="n">job_flow_overrides</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">job_flow_overrides</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">job_flow_overrides</span> <span class="o">=</span> <span class="n">job_flow_overrides</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">emr</span> <span class="o">=</span> <span class="n">EmrHook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">,</span> <span class="n">emr_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">emr_conn_id</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Creating JobFlow using aws-conn-id: </span><span class="si">%s</span><span class="s1">, emr-conn-id: </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">emr_conn_id</span>
+ <span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">emr</span><span class="o">.</span><span class="n">create_job_flow</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">job_flow_overrides</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">response</span><span class="p">[</span><span class="s1">'ResponseMetadata'</span><span class="p">][</span><span class="s1">'HTTPStatusCode'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'JobFlow creation failed: </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">response</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'JobFlow with id </span><span class="si">%s</span><span class="s1"> created'</span><span class="p">,</span> <span class="n">response</span><span class="p">[</span><span class="s1">'JobFlowId'</span><span class="p">])</span>
+ <span class="k">return</span> <span class="n">response</span><span class="p">[</span><span class="s1">'JobFlowId'</span><span class="p">]</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/emr_terminate_job_flow_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/emr_terminate_job_flow_operator.html b/_modules/airflow/contrib/operators/emr_terminate_job_flow_operator.html
new file mode 100644
index 0000000..7f2f1ca
--- /dev/null
+++ b/_modules/airflow/contrib/operators/emr_terminate_job_flow_operator.html
@@ -0,0 +1,285 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.emr_terminate_job_flow_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.emr_terminate_job_flow_operator</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.emr_terminate_job_flow_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="k">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.emr_hook</span> <span class="k">import</span> <span class="n">EmrHook</span>
+
+
+<div class="viewcode-block" id="EmrTerminateJobFlowOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator">[docs]</a><span class="k">class</span> <span class="nc">EmrTerminateJobFlowOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Operator to terminate EMR JobFlows.</span>
+
+<span class="sd"> :param job_flow_id: id of the JobFlow to terminate</span>
+<span class="sd"> :type job_flow_name: str</span>
+<span class="sd"> :param aws_conn_id: aws connection to uses</span>
+<span class="sd"> :type aws_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'job_flow_id'</span><span class="p">]</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f9c915'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">job_flow_id</span><span class="p">,</span>
+ <span class="n">aws_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">EmrTerminateJobFlowOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span> <span class="o">=</span> <span class="n">job_flow_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span> <span class="o">=</span> <span class="n">aws_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">emr</span> <span class="o">=</span> <span class="n">EmrHook</span><span class="p">(</span><span class="n">aws_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">aws_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Terminating JobFlow </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">emr</span><span class="o">.</span><span class="n">terminate_job_flows</span><span class="p">(</span><span class="n">JobFlowIds</span><span class="o">=</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span><span class="p">])</span>
+
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">response</span><span class="p">[</span><span class="s1">'ResponseMetadata'</span><span class="p">][</span><span class="s1">'HTTPStatusCode'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">200</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'JobFlow termination failed: </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">response</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'JobFlow with id </span><span class="si">%s</span><span class="s1"> terminated'</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">job_flow_id</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/_modules/airflow/contrib/operators/file_to_wasb.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/file_to_wasb.html b/_modules/airflow/contrib/operators/file_to_wasb.html
new file mode 100644
index 0000000..0c67052
--- /dev/null
+++ b/_modules/airflow/contrib/operators/file_to_wasb.html
@@ -0,0 +1,290 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.file_to_wasb — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="index" title="Index"
+ href="../../../../genindex.html"/>
+ <link rel="search" title="Search" href="../../../../search.html"/>
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../integration.html">Integration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+ <ul class="wy-breadcrumbs">
+
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.file_to_wasb</li>
+
+
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+
+ </ul>
+
+
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.file_to_wasb</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.wasb_hook</span> <span class="k">import</span> <span class="n">WasbHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="k">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="FileToWasbOperator"><a class="viewcode-back" href="../../../../integration.html#airflow.contrib.operators.file_to_wasb.FileToWasbOperator">[docs]</a><span class="k">class</span> <span class="nc">FileToWasbOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Uploads a file to Azure Blob Storage.</span>
+
+<span class="sd"> :param file_path: Path to the file to load.</span>
+<span class="sd"> :type file_path: str</span>
+<span class="sd"> :param container_name: Name of the container.</span>
+<span class="sd"> :type container_name: str</span>
+<span class="sd"> :param blob_name: Name of the blob.</span>
+<span class="sd"> :type blob_name: str</span>
+<span class="sd"> :param wasb_conn_id: Reference to the wasb connection.</span>
+<span class="sd"> :type wasb_conn_id: str</span>
+<span class="sd"> :param load_options: Optional keyword arguments that</span>
+<span class="sd"> `WasbHook.load_file()` takes.</span>
+<span class="sd"> :type load_options: dict</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'file_path'</span><span class="p">,</span> <span class="s1">'container_name'</span><span class="p">,</span> <span class="s1">'blob_name'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file_path</span><span class="p">,</span> <span class="n">container_name</span><span class="p">,</span> <span class="n">blob_name</span><span class="p">,</span>
+ <span class="n">wasb_conn_id</span><span class="o">=</span><span class="s1">'wasb_default'</span><span class="p">,</span> <span class="n">load_options</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">FileToWasbOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">load_options</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
+ <span class="n">load_options</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">file_path</span> <span class="o">=</span> <span class="n">file_path</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">container_name</span> <span class="o">=</span> <span class="n">container_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">blob_name</span> <span class="o">=</span> <span class="n">blob_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wasb_conn_id</span> <span class="o">=</span> <span class="n">wasb_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">load_options</span> <span class="o">=</span> <span class="n">load_options</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="sd">"""Upload a file to Azure Blob Storage."""</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">WasbHook</span><span class="p">(</span><span class="n">wasb_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">wasb_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">log</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Uploading </span><span class="si">{self.file_path}</span><span class="s1"> to wasb://</span><span class="si">{self.container_name}</span><span class="s1"> as </span><span class="si">{self.blob_name}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="p">)</span>
+ <span class="n">hook</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">file_path</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">container_name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">blob_name</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">load_options</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ <div class="articleComments">
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file