You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/08/08 23:33:35 UTC
[beam-site] branch asf-site updated (24a8588 -> 582961e)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from 24a8588 Prepare repository for deployment.
add 488f57c Add section for SQL shell
add f079eb1 This closes #527
new 582961e Prepare repository for deployment.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../dsls/sql/aggregate-functions/index.html | 1 +
.../documentation/dsls/sql/create-table/index.html | 1 +
.../documentation/dsls/sql/data-types/index.html | 1 +
content/documentation/dsls/sql/joins/index.html | 1 +
content/documentation/dsls/sql/lexical/index.html | 1 +
content/documentation/dsls/sql/overview/index.html | 1 +
.../dsls/sql/scalar-functions/index.html | 1 +
content/documentation/dsls/sql/select/index.html | 1 +
content/documentation/dsls/sql/set/index.html | 1 +
.../sql/shell}/index.html | 205 ++++++++++-----------
.../dsls/sql/user-defined-functions/index.html | 1 +
.../documentation/dsls/sql/walkthrough/index.html | 1 +
.../dsls/sql/windowing-and-triggering/index.html | 1 +
.../sdks/feature-comparison/index.html | 1 +
content/documentation/sdks/go/index.html | 1 +
.../documentation/sdks/java-extensions/index.html | 1 +
.../documentation/sdks/java-thirdparty/index.html | 1 +
.../documentation/sdks/java/euphoria/index.html | 1 +
content/documentation/sdks/java/index.html | 1 +
content/documentation/sdks/java/nexmark/index.html | 1 +
.../documentation/sdks/python-custom-io/index.html | 1 +
.../sdks/python-pipeline-dependencies/index.html | 1 +
.../documentation/sdks/python-streaming/index.html | 1 +
.../sdks/python-type-safety/index.html | 1 +
content/documentation/sdks/python/index.html | 1 +
src/_includes/section-menu/sdks.html | 1 +
src/documentation/dsls/sql/shell.md | 153 +++++++++++++++
27 files changed, 279 insertions(+), 104 deletions(-)
copy content/documentation/{sdks/python-pipeline-dependencies => dsls/sql/shell}/index.html (64%)
create mode 100644 src/documentation/dsls/sql/shell.md
[beam-site] 01/01: Prepare repository for deployment.
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 582961ed7829d896ea683f2f2ca3d9ac8e4b656e
Author: Mergebot <me...@apache.org>
AuthorDate: Wed Aug 8 23:33:33 2018 +0000
Prepare repository for deployment.
---
.../dsls/sql/aggregate-functions/index.html | 1 +
.../documentation/dsls/sql/create-table/index.html | 1 +
.../documentation/dsls/sql/data-types/index.html | 1 +
content/documentation/dsls/sql/joins/index.html | 1 +
content/documentation/dsls/sql/lexical/index.html | 1 +
content/documentation/dsls/sql/overview/index.html | 1 +
.../dsls/sql/scalar-functions/index.html | 1 +
content/documentation/dsls/sql/select/index.html | 1 +
content/documentation/dsls/sql/set/index.html | 1 +
.../sql/shell}/index.html | 205 ++++++++++-----------
.../dsls/sql/user-defined-functions/index.html | 1 +
.../documentation/dsls/sql/walkthrough/index.html | 1 +
.../dsls/sql/windowing-and-triggering/index.html | 1 +
.../sdks/feature-comparison/index.html | 1 +
content/documentation/sdks/go/index.html | 1 +
.../documentation/sdks/java-extensions/index.html | 1 +
.../documentation/sdks/java-thirdparty/index.html | 1 +
.../documentation/sdks/java/euphoria/index.html | 1 +
content/documentation/sdks/java/index.html | 1 +
content/documentation/sdks/java/nexmark/index.html | 1 +
.../documentation/sdks/python-custom-io/index.html | 1 +
.../sdks/python-pipeline-dependencies/index.html | 1 +
.../documentation/sdks/python-streaming/index.html | 1 +
.../sdks/python-type-safety/index.html | 1 +
content/documentation/sdks/python/index.html | 1 +
25 files changed, 125 insertions(+), 104 deletions(-)
diff --git a/content/documentation/dsls/sql/aggregate-functions/index.html b/content/documentation/dsls/sql/aggregate-functions/index.html
index 5fa8c55..ce77ea5 100644
--- a/content/documentation/dsls/sql/aggregate-functions/index.html
+++ b/content/documentation/dsls/sql/aggregate-functions/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/create-table/index.html b/content/documentation/dsls/sql/create-table/index.html
index 8980917..c40550c 100644
--- a/content/documentation/dsls/sql/create-table/index.html
+++ b/content/documentation/dsls/sql/create-table/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/data-types/index.html b/content/documentation/dsls/sql/data-types/index.html
index 251e2ce..c52c50a 100644
--- a/content/documentation/dsls/sql/data-types/index.html
+++ b/content/documentation/dsls/sql/data-types/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/joins/index.html b/content/documentation/dsls/sql/joins/index.html
index eca1027..0cd17e2 100644
--- a/content/documentation/dsls/sql/joins/index.html
+++ b/content/documentation/dsls/sql/joins/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/lexical/index.html b/content/documentation/dsls/sql/lexical/index.html
index aa8a39d..15cf06c 100644
--- a/content/documentation/dsls/sql/lexical/index.html
+++ b/content/documentation/dsls/sql/lexical/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/overview/index.html b/content/documentation/dsls/sql/overview/index.html
index 70d832a..2d0c8b4 100644
--- a/content/documentation/dsls/sql/overview/index.html
+++ b/content/documentation/dsls/sql/overview/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/scalar-functions/index.html b/content/documentation/dsls/sql/scalar-functions/index.html
index daaba3e..781e07c 100644
--- a/content/documentation/dsls/sql/scalar-functions/index.html
+++ b/content/documentation/dsls/sql/scalar-functions/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/select/index.html b/content/documentation/dsls/sql/select/index.html
index 813d209..1d77b77 100644
--- a/content/documentation/dsls/sql/select/index.html
+++ b/content/documentation/dsls/sql/select/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/set/index.html b/content/documentation/dsls/sql/set/index.html
index 3611b4b..144c71a 100644
--- a/content/documentation/dsls/sql/set/index.html
+++ b/content/documentation/dsls/sql/set/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/python-pipeline-dependencies/index.html b/content/documentation/dsls/sql/shell/index.html
similarity index 64%
copy from content/documentation/sdks/python-pipeline-dependencies/index.html
copy to content/documentation/dsls/sql/shell/index.html
index e723404..e215b22 100644
--- a/content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/content/documentation/dsls/sql/shell/index.html
@@ -28,7 +28,7 @@
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
- <title>Managing Python Pipeline Dependencies</title>
+ <title>Beam SQL: Shell</title>
<meta name="description" content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch and streaming data processing and can run on a number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow [...]
">
<link href="https://fonts.googleapis.com/css?family=Roboto:100,300,400" rel="stylesheet">
@@ -39,7 +39,7 @@
<script src="/js/fix-menu.js"></script>
<script src="/js/section-nav.js"></script>
<script src="/js/page-nav.js"></script>
- <link rel="canonical" href="https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/" data-proofer-ignore>
+ <link rel="canonical" href="https://beam.apache.org/documentation/dsls/sql/shell/" data-proofer-ignore>
<link rel="shortcut icon" type="image/x-icon" href="/images/favicon.ico">
<link rel="alternate" type="application/rss+xml" title="Apache Beam" href="https://beam.apache.org/feed.xml">
<script>
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
@@ -222,10 +223,14 @@
<ul class="nav">
- <li><a href="#pypi-dependencies">PyPI Dependencies</a></li>
- <li><a href="#local-or-nonpypi">Local or non-PyPI Dependencies</a></li>
- <li><a href="#multiple-file-dependencies">Multiple File Dependencies</a></li>
- <li><a href="#nonpython">Non-Python Dependencies or PyPI Dependencies with Non-Python Dependencies</a></li>
+ <li><a href="#overview">Overview</a></li>
+ <li><a href="#quickstart">Quickstart</a></li>
+ <li><a href="#declaring-tables">Declaring Tables</a></li>
+ <li><a href="#reading-and-writing-data">Reading and Writing Data</a></li>
+ <li><a href="#developing-with-unbounded-sources">Developing with unbounded Sources</a></li>
+ <li><a href="#specifying-the-runner">Specifying the Runner</a></li>
+ <li><a href="#specifying-the-pipelineoptions">Specifying the PipelineOptions</a></li>
+ <li><a href="#packaging-the-sql-shell">Packaging the SQL Shell</a></li>
</ul>
@@ -245,155 +250,147 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
-<h1 id="managing-python-pipeline-dependencies">Managing Python Pipeline Dependencies</h1>
-<blockquote>
- <p><strong>Note:</strong> This page is only applicable to runners that do remote execution.</p>
-</blockquote>
+<h1 id="beam-sql-shell">Beam SQL Shell</h1>
-<p>When you run your pipeline locally, the packages that your pipeline depends on are available because they are installed on your local machine. However, when you want to run your pipeline remotely, you must make sure these dependencies are available on the remote machines. This tutorial shows you how to make your dependencies available to the remote workers. Each section below refers to a different source that your package may have been installed from.</p>
+<h2 id="overview">Overview</h2>
-<p><strong>Note:</strong> Remote workers used for pipeline execution typically have a standard Python 2.7 distribution installation. If your code relies only on standard Python packages, then you probably don’t need to do anything on this page.</p>
+<p>Starting with version 2.6.0, Beam SQL includes an interactive shell, called the Beam SQL shell. The shell allows you to write pipelines as SQL queries without needing the Java SDK. By default, Beam uses the <code class="highlighter-rouge">DirectRunner</code> to execute the queries as Beam pipelines.</p>
-<h2 id="pypi-dependencies">PyPI Dependencies</h2>
+<p>This page describes how to work with the shell, but does not focus on specific features of Beam SQL. For a more thorough overview of the features used in the examples on this page, see the corresponding sections in the <a href="/documentation/dsls/sql/overview/">Beam SQL documentation</a>.</p>
-<p>If your pipeline uses public packages from the <a href="https://pypi.python.org/">Python Package Index</a>, make these packages available remotely by performing the following steps:</p>
+<h2 id="quickstart">Quickstart</h2>
-<p><strong>Note:</strong> If your PyPI package depends on a non-Python package (e.g. a package that requires installation on Linux using the <code class="highlighter-rouge">apt-get install</code> command), see the <a href="#nonpython">PyPI Dependencies with Non-Python Dependencies</a> section instead.</p>
+<p>To use Beam SQL shell, you must first clone the <a href="https://github.com/apache/beam">Beam SDK repository</a>. Then, from the root of the repository clone, execute the following commands to run the shell:</p>
-<ol>
- <li>
- <p>Find out which packages are installed on your machine. Run the following command:</p>
+<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':beam-runners-flink_2.11,:beam-sdks-java-io-kafka' installDist
- <div class="highlighter-rouge"><pre class="highlight"><code> pip freeze > requirements.txt
+./sdks/java/extensions/sql/shell/build/install/beam-sdks-java-extensions-sql-shell/bin/beam-sdks-java-extensions-sql-shell
</code></pre>
- </div>
+</div>
- <p>This command creates a <code class="highlighter-rouge">requirements.txt</code> file that lists all packages that are installed on your machine, regardless of where they were installed from.</p>
- </li>
- <li>
- <p>Edit the <code class="highlighter-rouge">requirements.txt</code> file and leave only the packages that were installed from PyPI and are used in the workflow source. Delete all packages that are not relevant to your code.</p>
- </li>
- <li>
- <p>Run your pipeline with the following command-line option:</p>
+<p>After you run the commands, the SQL shell starts and you can type queries:</p>
- <div class="highlighter-rouge"><pre class="highlight"><code> --requirements_file requirements.txt
+<div class="highlighter-rouge"><pre class="highlight"><code>Welcome to Beam SQL 2.6.0-SNAPSHOT (based on sqlline version 1.4.0)
+0: BeamSQL>
</code></pre>
- </div>
+</div>
- <p>The runner will use the <code class="highlighter-rouge">requirements.txt</code> file to install your additional dependencies onto the remote workers.</p>
- </li>
-</ol>
+<p><em>Note: If you haven’t built the project before running the Gradle command, the command will take a few minutes as Gradle must build all dependencies first.</em></p>
-<p><strong>Important:</strong> Remote workers will install all packages listed in the <code class="highlighter-rouge">requirements.txt</code> file. Because of this, it’s very important that you delete non-PyPI packages from the <code class="highlighter-rouge">requirements.txt</code> file, as stated in step 2. If you don’t remove non-PyPI packages, the remote workers will fail when attempting to install packages from sources that are unknown to them.</p>
+<p>The shell converts the queries into Beam pipelines, runs them using <code class="highlighter-rouge">DirectRunner</code>, and returns the results as tables when the pipelines finish:</p>
-<h2 id="local-or-nonpypi">Local or non-PyPI Dependencies</h2>
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> SELECT 'foo' AS NAME, 'bar' AS TYPE, 'num' AS NUMBER;
++------+------+--------+
+| NAME | TYPE | NUMBER |
++------+------+--------+
+| foo | bar | num |
++------+------+--------+
+1 row selected (0.826 seconds)
+</code></pre>
+</div>
-<p>If your pipeline uses packages that are not available publicly (e.g. packages that you’ve downloaded from a GitHub repo), make these packages available remotely by performing the following steps:</p>
+<h2 id="declaring-tables">Declaring Tables</h2>
-<ol>
- <li>
- <p>Identify which packages are installed on your machine and are not public. Run the following command:</p>
+<p>Before reading data from a source or writing data to a destination, you must declare a virtual table using the <code class="highlighter-rouge">CREATE TABLE</code> statement. For example, if you have a local CSV file <code class="highlighter-rouge">"test-file.csv"</code> in the current folder, you can create a table with the following statement:</p>
- <div class="highlighter-rouge"><pre class="highlight"><code> pip freeze
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> CREATE TABLE csv_file (field1 VARCHAR, field2 INTEGER) TYPE text LOCATION 'test-file.csv';
+
+No rows affected (0.042 seconds)
</code></pre>
- </div>
+</div>
- <p>This command lists all packages that are installed on your machine, regardless of where they were installed from.</p>
- </li>
- <li>
- <p>Run your pipeline with the following command-line option:</p>
+<p>The <code class="highlighter-rouge">CREATE TABLE</code> statement registers the CSV file as a table in Beam SQL and specifies the table’s schema. This statement does not directly create a persistent physical table; it only describes the source/sink to Beam SQL so that you can use the table in the queries that read data and write data.</p>
+
+<p><em>For more information about <code class="highlighter-rouge">CREATE TABLE</code> syntax and supported table types, see the <a href="/documentation/dsls/sql/create-table/">CREATE TABLE reference page</a>.</em></p>
+
+<h2 id="reading-and-writing-data">Reading and Writing Data</h2>
+
+<p>To read data from the local CSV file that you declared in the previous section, execute the following query:</p>
- <div class="highlighter-rouge"><pre class="highlight"><code> --extra_package /path/to/package/package-name
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> SELECT field1 AS field FROM csv_file;
++--------+
+| field |
++--------+
+| baz |
+| foo |
+| bar |
+| bar |
+| foo |
++--------+
</code></pre>
- </div>
+</div>
+
+<p><em>For more information about <code class="highlighter-rouge">SELECT</code> syntax, see the <a href="/documentation/dsls/sql/select/">SELECT reference page</a>.</em></p>
- <p>where package-name is the package’s tarball. If you have the <code class="highlighter-rouge">setup.py</code> for that
-package then you can build the tarball with the following command:</p>
+<p>To write data to the CSV file, use the <code class="highlighter-rouge">INSERT INTO … SELECT ...</code> statement:</p>
- <div class="highlighter-rouge"><pre class="highlight"><code> python setup.py sdist
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> INSERT INTO csv_file SELECT 'foo', 'bar';
</code></pre>
- </div>
+</div>
+<p>Read and write behavior depends on the type of the table. For example:</p>
- <p>See the <a href="https://docs.python.org/2/distutils/sourcedist.html">sdist documentation</a> for more details on this command.</p>
- </li>
-</ol>
+<ul>
+ <li>The table type <code class="highlighter-rouge">text</code> is implemented using <code class="highlighter-rouge">TextIO</code>, so writing to a <code class="highlighter-rouge">text</code> table can produce multiple numbered files.</li>
+ <li>The table type <code class="highlighter-rouge">pubsub</code> is an unbounded source, so reading from a <code class="highlighter-rouge">pubsub</code> table never completes.</li>
+</ul>
-<h2 id="multiple-file-dependencies">Multiple File Dependencies</h2>
+<h2 id="developing-with-unbounded-sources">Developing with unbounded Sources</h2>
-<p>Often, your pipeline code spans multiple files. To run your project remotely, you must group these files as a Python package and specify the package when you run your pipeline. When the remote workers start, they will install your package. To group your files as a Python package and make it available remotely, perform the following steps:</p>
+<p>When you want to inspect the data from an unbounded source during development, you must specify the <code class="highlighter-rouge">LIMIT x</code> clause at the end of the <code class="highlighter-rouge">SELECT</code> statement to limit the output to <code class="highlighter-rouge">x</code> number of records. Otherwise, the pipeline will never finish.</p>
-<ol>
- <li>
- <p>Create a <a href="https://pythonhosted.org/an_example_pypi_project/setuptools.html">setup.py</a> file for your project. The following is a very basic <code class="highlighter-rouge">setup.py</code> file.</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> setuptools.setup(
- name='PACKAGE-NAME'
- version='PACKAGE-VERSION',
- install_requires=[],
- packages=setuptools.find_packages(),
- )
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> SELECT field1 FROM unbounded_source LIMIT 10 ;
</code></pre>
- </div>
- </li>
+</div>
+
+<p>The example queries shown so far are fast queries that execute locally. These queries are helpful when you are investigating the data and iteratively designing the pipeline. Ideally, you want the queries to finish quickly and return output when complete.</p>
+
+<p>When you’re satisfied with the logic of your SQL statements, you can submit the statements as long-running jobs by dropping the <code class="highlighter-rouge">LIMIT x</code> statement. Then, the pipeline can potentially run forever if one of the tables represents an unbounded source.</p>
+
+<h2 id="specifying-the-runner">Specifying the Runner</h2>
+
+<p>By default, Beam uses the <code class="highlighter-rouge">DirectRunner</code> to run the pipeline on the machine where you’re executing the commands. If you want to run the pipeline with a different runner, you must perform two steps:</p>
+
+<ol>
<li>
- <p>Structure your project so that the root directory contains the <code class="highlighter-rouge">setup.py</code> file, the main workflow file, and a directory with the rest of the files.</p>
+ <p>Make sure the SQL shell includes the desired runner. Add the corresponding project id to the <code class="highlighter-rouge">-Pbeam.sql.shell.bundled</code> parameter of the Gradle invocation (<a href="https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/shell/build.gradle">source code</a>, <a href="https://github.com/apache/beam/blob/master/settings.gradle">project ids</a>). For example, use the following command to include Flink runner and KafkaIO:</p>
- <div class="highlighter-rouge"><pre class="highlight"><code> root_dir/
- setup.py
- main.py
- other_files_dir/
+ <div class="highlighter-rouge"><pre class="highlight"><code>./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':beam-runners-flink_2.11,:beam-sdks-java-io-kafka' installDist
</code></pre>
</div>
- <p>See <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset">Juliaset</a> for an example that follows this required project structure.</p>
+ <p><em>Note: You can bundle multiple runners (using a comma-separated list) or other additional components in the same manner. For example, you can add support for more I/Os.</em></p>
</li>
<li>
- <p>Run your pipeline with the following command-line option:</p>
+ <p>Then, specify the runner using the <code class="highlighter-rouge">SET</code> command (<a href="/documentation/dsls/sql/set/">reference page</a>):</p>
- <div class="highlighter-rouge"><pre class="highlight"><code> --setup_file /path/to/setup.py
+ <div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> SET runner='FlinkRunner';
</code></pre>
</div>
</li>
</ol>
-<p><strong>Note:</strong> If you <a href="#pypi-dependencies">created a requirements.txt file</a> and your project spans multiple files, you can get rid of the <code class="highlighter-rouge">requirements.txt</code> file and instead, add all packages contained in <code class="highlighter-rouge">requirements.txt</code> to the <code class="highlighter-rouge">install_requires</code> field of the setup call (in step 1).</p>
+<p>Beam will submit all future <code class="highlighter-rouge">INSERT</code> statements as pipelines to the specified runner. In this case, the Beam SQL shell does not display the query results. You must manage the submitted jobs through the corresponding runner’s UI (for example, using the Flink UI or command line).</p>
-<h2 id="nonpython">Non-Python Dependencies or PyPI Dependencies with Non-Python Dependencies</h2>
+<h2 id="specifying-the-pipelineoptions">Specifying the PipelineOptions</h2>
-<p>If your pipeline uses non-Python packages (e.g. packages that require installation using the <code class="highlighter-rouge">apt-get install</code> command), or uses a PyPI package that depends on non-Python dependencies during package installation, you must perform the following steps.</p>
-
-<ol>
- <li>
- <p>Add the required installation commands (e.g. the <code class="highlighter-rouge">apt-get install</code> commands) for the non-Python dependencies to the list of <code class="highlighter-rouge">CUSTOM_COMMANDS</code> in your <code class="highlighter-rouge">setup.py</code> file. See the <a href="https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/complete/juliaset/setup.py">Juliaset setup.py</a> for an example.</p>
+<p>To configure the runner, you must specify <code class="highlighter-rouge">PipelineOptions</code> by using the <code class="highlighter-rouge">SET</code> command (<a href="/documentation/dsls/sql/set/">details</a>):</p>
- <p><strong>Note:</strong> You must make sure that these commands are runnable on the remote worker (e.g. if you use <code class="highlighter-rouge">apt-get</code>, the remote worker needs <code class="highlighter-rouge">apt-get</code> support).</p>
- </li>
- <li>
- <p>If you are using a PyPI package that depends on non-Python dependencies, add <code class="highlighter-rouge">['pip', 'install', '<your PyPI package>']</code> to the list of <code class="highlighter-rouge">CUSTOM_COMMANDS</code> in your <code class="highlighter-rouge">setup.py</code> file.</p>
- </li>
- <li>
- <p>Structure your project so that the root directory contains the <code class="highlighter-rouge">setup.py</code> file, the main workflow file, and a directory with the rest of the files.</p>
-
- <div class="highlighter-rouge"><pre class="highlight"><code> root_dir/
- setup.py
- main.py
- other_files_dir/
+<div class="highlighter-rouge"><pre class="highlight"><code>0: BeamSQL> SET projectId='gcpProjectId';
+0: BeamSQL> SET tempLocation='/tmp/tempDir';
</code></pre>
- </div>
+</div>
+<h2 id="packaging-the-sql-shell">Packaging the SQL Shell</h2>
- <p>See the <a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/complete/juliaset">Juliaset</a> project for an example that follows this required project structure.</p>
- </li>
- <li>
- <p>Run your pipeline with the following command-line option:</p>
+<p>You can also build your own standalone package for SQL shell using <code class="highlighter-rouge">distZip</code> or <code class="highlighter-rouge">distTar</code> tasks. For example:</p>
+
+<div class="highlighter-rouge"><pre class="highlight"><code>./gradlew -p sdks/java/extensions/sql/shell -Pbeam.sql.shell.bundled=':beam-runners-flink_2.11,:beam-sdks-java-io-kafka' distZip
- <div class="highlighter-rouge"><pre class="highlight"><code> --setup_file /path/to/setup.py
+ls ./sdks/java/extensions/sql/shell/build/distributions/
+beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.tar beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.zip
</code></pre>
- </div>
- </li>
-</ol>
+</div>
-<p><strong>Note:</strong> Because custom commands execute after the dependencies for your workflow are installed (by <code class="highlighter-rouge">pip</code>), you should omit the PyPI package dependency from the pipeline’s <code class="highlighter-rouge">requirements.txt</code> file and from the <code class="highlighter-rouge">install_requires</code> parameter in the <code class="highlighter-rouge">setuptools.setup()</code> call of your <code class="highlighter-rouge">setup.py</code> [...]
</div>
</div>
diff --git a/content/documentation/dsls/sql/user-defined-functions/index.html b/content/documentation/dsls/sql/user-defined-functions/index.html
index c887ead..135b985 100644
--- a/content/documentation/dsls/sql/user-defined-functions/index.html
+++ b/content/documentation/dsls/sql/user-defined-functions/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/walkthrough/index.html b/content/documentation/dsls/sql/walkthrough/index.html
index 722a194..0b58348 100644
--- a/content/documentation/dsls/sql/walkthrough/index.html
+++ b/content/documentation/dsls/sql/walkthrough/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/dsls/sql/windowing-and-triggering/index.html b/content/documentation/dsls/sql/windowing-and-triggering/index.html
index 9721b80..6f8d33b 100644
--- a/content/documentation/dsls/sql/windowing-and-triggering/index.html
+++ b/content/documentation/dsls/sql/windowing-and-triggering/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/feature-comparison/index.html b/content/documentation/sdks/feature-comparison/index.html
index 0b07c2d..41a4dd0 100644
--- a/content/documentation/sdks/feature-comparison/index.html
+++ b/content/documentation/sdks/feature-comparison/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/go/index.html b/content/documentation/sdks/go/index.html
index 0405e98..e8c5bad 100644
--- a/content/documentation/sdks/go/index.html
+++ b/content/documentation/sdks/go/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/java-extensions/index.html b/content/documentation/sdks/java-extensions/index.html
index 55c4072..904a9b0 100644
--- a/content/documentation/sdks/java-extensions/index.html
+++ b/content/documentation/sdks/java-extensions/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/java-thirdparty/index.html b/content/documentation/sdks/java-thirdparty/index.html
index 5909eed..a4e0afb 100644
--- a/content/documentation/sdks/java-thirdparty/index.html
+++ b/content/documentation/sdks/java-thirdparty/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/java/euphoria/index.html b/content/documentation/sdks/java/euphoria/index.html
index 6af7938..a0c0393 100644
--- a/content/documentation/sdks/java/euphoria/index.html
+++ b/content/documentation/sdks/java/euphoria/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/java/index.html b/content/documentation/sdks/java/index.html
index 8a28c35..322fa99 100644
--- a/content/documentation/sdks/java/index.html
+++ b/content/documentation/sdks/java/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/java/nexmark/index.html b/content/documentation/sdks/java/nexmark/index.html
index 7f7dbcd..f1b80fc 100644
--- a/content/documentation/sdks/java/nexmark/index.html
+++ b/content/documentation/sdks/java/nexmark/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/python-custom-io/index.html b/content/documentation/sdks/python-custom-io/index.html
index c2101bd..3402e0b 100644
--- a/content/documentation/sdks/python-custom-io/index.html
+++ b/content/documentation/sdks/python-custom-io/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/python-pipeline-dependencies/index.html b/content/documentation/sdks/python-pipeline-dependencies/index.html
index e723404..9dfba4c 100644
--- a/content/documentation/sdks/python-pipeline-dependencies/index.html
+++ b/content/documentation/sdks/python-pipeline-dependencies/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/python-streaming/index.html b/content/documentation/sdks/python-streaming/index.html
index 7d724fb..f521c16 100644
--- a/content/documentation/sdks/python-streaming/index.html
+++ b/content/documentation/sdks/python-streaming/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/python-type-safety/index.html b/content/documentation/sdks/python-type-safety/index.html
index 33ce931..b2c071c 100644
--- a/content/documentation/sdks/python-type-safety/index.html
+++ b/content/documentation/sdks/python-type-safety/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>
diff --git a/content/documentation/sdks/python/index.html b/content/documentation/sdks/python/index.html
index 859c864..c5a9b17 100644
--- a/content/documentation/sdks/python/index.html
+++ b/content/documentation/sdks/python/index.html
@@ -183,6 +183,7 @@
<ul class="section-nav-list">
<li><a href="/documentation/dsls/sql/overview/">Overview</a></li>
<li><a href="/documentation/dsls/sql/walkthrough/">Walkthrough</a></li>
+ <li><a href="/documentation/dsls/sql/shell/">Shell</a></li>
<li class="section-nav-item--collapsible">
<span class="section-nav-list-title">SQL Reference</span>