You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by fh...@apache.org on 2017/06/15 09:44:02 UTC
[01/10] flink git commit: [FLNK-5354] [docs] Restructured Table API /
SQL docs
Repository: flink
Updated Branches:
refs/heads/master 9141379f6 -> d8756553c
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table_api.md
----------------------------------------------------------------------
diff --git a/docs/dev/table_api.md b/docs/dev/table_api.md
deleted file mode 100644
index 6a5ceee..0000000
--- a/docs/dev/table_api.md
+++ /dev/null
@@ -1,6015 +0,0 @@
----
-title: "Table and SQL"
-is_beta: true
-nav-parent_id: libs
-nav-pos: 0
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-**Table API and SQL are experimental features**
-
-The Table API is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flink's DataSet and DataStream APIs (Java and Scala).
-The Table API and SQL interface operate on a relational `Table` abstraction, which can be created from external data sources, or existing DataSets and DataStreams. With the Table API, you can apply relational operators such as selection, aggregation, and joins on `Table`s.
-
-`Table`s can also be queried with regular SQL, as long as they are registered (see [Registering Tables](#registering-tables)). The Table API and SQL offer equivalent functionality and can be mixed in the same program. When a `Table` is converted back into a `DataSet` or `DataStream`, the logical plan, which was defined by relational operators and SQL queries, is optimized using [Apache Calcite](https://calcite.apache.org/) and transformed into a `DataSet` or `DataStream` program.
-
-* This will be replaced by the TOC
-{:toc}
-
-Using the Table API and SQL
-----------------------------
-
-The Table API and SQL are part of the *flink-table* Maven project.
-The following dependency must be added to your project in order to use the Table API and SQL:
-
-{% highlight xml %}
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-{% endhighlight %}
-
-*Note: The Table API is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/linking.html).*
-
-
-Registering Tables
---------------------------------
-
-`TableEnvironment`s have an internal table catalog to which tables can be registered with a unique name. After registration, a table can be accessed from the `TableEnvironment` by its name.
-
-*Note: `DataSet`s or `DataStream`s can be directly converted into `Table`s without registering them in the `TableEnvironment`.*
-
-### Register a DataSet
-
-A `DataSet` is registered as a `Table` in a `BatchTableEnvironment` as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// register the DataSet cust as table "Customers" with fields derived from the dataset
-tableEnv.registerDataSet("Customers", cust);
-
-// register the DataSet ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataSet("Orders", ord, "user, product, amount");
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-// register the DataSet cust as table "Customers" with fields derived from the dataset
-tableEnv.registerDataSet("Customers", cust)
-
-// register the DataSet ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataSet("Orders", ord, 'user, 'product, 'amount)
-{% endhighlight %}
-</div>
-</div>
-
-*Note: The name of a `DataSet` `Table` must not match the `^_DataSetTable_[0-9]+` pattern which is reserved for internal use only.*
-
-### Register a DataStream
-
-A `DataStream` is registered as a `Table` in a `StreamTableEnvironment` as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// register the DataStream cust as table "Customers" with fields derived from the datastream
-tableEnv.registerDataStream("Customers", cust);
-
-// register the DataStream ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataStream("Orders", ord, "user, product, amount");
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-// register the DataStream cust as table "Customers" with fields derived from the datastream
-tableEnv.registerDataStream("Customers", cust)
-
-// register the DataStream ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataStream("Orders", ord, 'user, 'product, 'amount)
-{% endhighlight %}
-</div>
-</div>
-
-*Note: The name of a `DataStream` `Table` must not match the `^_DataStreamTable_[0-9]+` pattern which is reserved for internal use only.*
-
-### Register a Table
-
-A `Table` that originates from a Table API operation or a SQL query is registered in a `TableEnvironment` as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// works for StreamExecutionEnvironment identically
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// convert a DataSet into a Table
-Table custT = tableEnv
- .toTable(custDs, "name, zipcode")
- .where("zipcode = '12345'")
- .select("name");
-
-// register the Table custT as table "custNames"
-tableEnv.registerTable("custNames", custT);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// works for StreamExecutionEnvironment identically
-val env = ExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-// convert a DataSet into a Table
-val custT = custDs
- .toTable(tableEnv, 'name, 'zipcode)
- .where('zipcode === "12345")
- .select('name)
-
-// register the Table custT as table "custNames"
-tableEnv.registerTable("custNames", custT)
-{% endhighlight %}
-</div>
-</div>
-
-A registered `Table` that originates from a Table API operation or SQL query is treated similarly as a view as known from relational DBMS, i.e., it can be inlined when optimizing the query.
-
-### Register an external Table using a TableSource
-
-An external table is registered in a `TableEnvironment` using a `TableSource` as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// works for StreamExecutionEnvironment identically
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-TableSource custTS = new CsvTableSource("/path/to/file", ...);
-
-// register a `TableSource` as external table "Customers"
-tableEnv.registerTableSource("Customers", custTS);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// works for StreamExecutionEnvironment identically
-val env = ExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-val custTS: TableSource = new CsvTableSource("/path/to/file", ...)
-
-// register a `TableSource` as external table "Customers"
-tableEnv.registerTableSource("Customers", custTS)
-
-{% endhighlight %}
-</div>
-</div>
-
-A `TableSource` can provide access to data stored in various storage systems such as databases (MySQL, HBase, ...), file formats (CSV, Apache Parquet, Avro, ORC, ...), or messaging systems (Apache Kafka, RabbitMQ, ...).
-
-Currently, Flink provides the `CsvTableSource` to read CSV files and various `TableSources` to read JSON or Avro objects from Kafka.
-A custom `TableSource` can be defined by implementing the `BatchTableSource` or `StreamTableSource` interface.
-
-### Available Table Sources
-
-| **Class name** | **Maven dependency** | **Batch?** | **Streaming?** | **Description**
-| `CsvTableSouce` | `flink-table` | Y | Y | A simple source for CSV files.
-| `Kafka08JsonTableSource` | `flink-connector-kafka-0.8` | N | Y | A Kafka 0.8 source for JSON data.
-| `Kafka08AvroTableSource` | `flink-connector-kafka-0.8` | N | Y | A Kafka 0.8 source for Avro data.
-| `Kafka09JsonTableSource` | `flink-connector-kafka-0.9` | N | Y | A Kafka 0.9 source for JSON data.
-| `Kafka09AvroTableSource` | `flink-connector-kafka-0.9` | N | Y | A Kafka 0.9 source for Avro data.
-| `Kafka010JsonTableSource` | `flink-connector-kafka-0.10` | N | Y | A Kafka 0.10 source for JSON data.
-| `Kafka010AvroTableSource` | `flink-connector-kafka-0.10` | N | Y | A Kafka 0.10 source for Avro data.
-
-All sources that come with the `flink-table` dependency can be directly used by your Table programs. For all other table sources, you have to add the respective dependency in addition to the `flink-table` dependency.
-
-#### KafkaJsonTableSource
-
-To use the Kafka JSON source, you have to add the Kafka connector dependency to your project:
-
- - `flink-connector-kafka-0.8` for Kafka 0.8,
- - `flink-connector-kafka-0.9` for Kafka 0.9, or
- - `flink-connector-kafka-0.10` for Kafka 0.10, respectively.
-
-You can then create the source as follows (example for Kafka 0.8):
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// specify JSON field names and types
-TypeInformation<Row> typeInfo = Types.ROW(
- new String[] { "id", "name", "score" },
- new TypeInformation<?>[] { Types.INT(), Types.STRING(), Types.DOUBLE() }
-);
-
-KafkaJsonTableSource kafkaTableSource = new Kafka08JsonTableSource(
- kafkaTopic,
- kafkaProperties,
- typeInfo);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// specify JSON field names and types
-val typeInfo = Types.ROW(
- Array("id", "name", "score"),
- Array(Types.INT, Types.STRING, Types.DOUBLE)
-)
-
-val kafkaTableSource = new Kafka08JsonTableSource(
- kafkaTopic,
- kafkaProperties,
- typeInfo)
-{% endhighlight %}
-</div>
-</div>
-
-By default, a missing JSON field does not fail the source. You can configure this via:
-
-```java
-// Fail on missing JSON field
-tableSource.setFailOnMissingField(true);
-```
-
-You can work with the Table as explained in the rest of the Table API guide:
-
-```java
-tableEnvironment.registerTableSource("kafka-source", kafkaTableSource);
-Table result = tableEnvironment.ingest("kafka-source");
-```
-
-#### KafkaAvroTableSource
-
-The `KafkaAvroTableSource` allows you to read Avro's `SpecificRecord` objects from Kafka.
-
-To use the Kafka Avro source, you have to add the Kafka connector dependency to your project:
-
- - `flink-connector-kafka-0.8` for Kafka 0.8,
- - `flink-connector-kafka-0.9` for Kafka 0.9, or
- - `flink-connector-kafka-0.10` for Kafka 0.10, respectively.
-
-You can then create the source as follows (example for Kafka 0.8):
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// pass the generated Avro class to the TableSource
-Class<? extends SpecificRecord> clazz = MyAvroType.class;
-
-KafkaAvroTableSource kafkaTableSource = new Kafka08AvroTableSource(
- kafkaTopic,
- kafkaProperties,
- clazz);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// pass the generated Avro class to the TableSource
-val clazz = classOf[MyAvroType]
-
-val kafkaTableSource = new Kafka08AvroTableSource(
- kafkaTopic,
- kafkaProperties,
- clazz)
-{% endhighlight %}
-</div>
-</div>
-
-#### CsvTableSource
-
-The `CsvTableSource` is already included in `flink-table` without additional dependecies.
-
-The easiest way to create a `CsvTableSource` is by using the enclosed builder `CsvTableSource.builder()`, the builder has the following methods to configure properties:
-
- - `path(String path)` Sets the path to the CSV file, required.
- - `field(String fieldName, TypeInformation<?> fieldType)` Adds a field with the field name and field type information, can be called multiple times, required. The call order of this method defines also the order of the fields in a row.
- - `fieldDelimiter(String delim)` Sets the field delimiter, `","` by default.
- - `lineDelimiter(String delim)` Sets the line delimiter, `"\n"` by default.
- - `quoteCharacter(Character quote)` Sets the quote character for String values, `null` by default.
- - `commentPrefix(String prefix)` Sets a prefix to indicate comments, `null` by default.
- - `ignoreFirstLine()` Ignore the first line. Disabled by default.
- - `ignoreParseErrors()` Skip records with parse error instead to fail. Throwing an exception by default.
-
-You can create the source as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-CsvTableSource csvTableSource = CsvTableSource
- .builder()
- .path("/path/to/your/file.csv")
- .field("name", Types.STRING())
- .field("id", Types.INT())
- .field("score", Types.DOUBLE())
- .field("comments", Types.STRING())
- .fieldDelimiter("#")
- .lineDelimiter("$")
- .ignoreFirstLine()
- .ignoreParseErrors()
- .commentPrefix("%");
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val csvTableSource = CsvTableSource
- .builder
- .path("/path/to/your/file.csv")
- .field("name", Types.STRING)
- .field("id", Types.INT)
- .field("score", Types.DOUBLE)
- .field("comments", Types.STRING)
- .fieldDelimiter("#")
- .lineDelimiter("$")
- .ignoreFirstLine
- .ignoreParseErrors
- .commentPrefix("%")
-{% endhighlight %}
-</div>
-</div>
-
-You can work with the Table as explained in the rest of the Table API guide in both stream and batch `TableEnvironment`s:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-tableEnvironment.registerTableSource("mycsv", csvTableSource);
-
-Table streamTable = streamTableEnvironment.ingest("mycsv");
-
-Table batchTable = batchTableEnvironment.scan("mycsv");
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-tableEnvironment.registerTableSource("mycsv", csvTableSource)
-
-val streamTable = streamTableEnvironment.ingest("mycsv")
-
-val batchTable = batchTableEnvironment.scan("mycsv")
-{% endhighlight %}
-</div>
-</div>
-
-Registering external Catalogs
---------------------------------
-
-An external catalog is defined by the `ExternalCatalog` interface and provides information about databases and tables such as their name, schema, statistics, and access information. An `ExternalCatalog` is registered in a `TableEnvironment` as follows:
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// works for StreamExecutionEnvironment identically
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-ExternalCatalog customerCatalog = new InMemoryExternalCatalog();
-
-// register the ExternalCatalog customerCatalog
-tableEnv.registerExternalCatalog("Customers", customerCatalog);
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// works for StreamExecutionEnvironment identically
-val env = ExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-val customerCatalog: ExternalCatalog = new InMemoryExternalCatalog
-
-// register the ExternalCatalog customerCatalog
-tableEnv.registerExternalCatalog("Customers", customerCatalog)
-
-{% endhighlight %}
-</div>
-</div>
-
-Once registered in a `TableEnvironment`, all tables defined in a `ExternalCatalog` can be accessed from Table API or SQL queries by specifying their full path (`catalog`.`database`.`table`).
-
-Currently, Flink provides an `InMemoryExternalCatalog` for demo and testing purposes. However, the `ExternalCatalog` interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.
-
-Table API
-----------
-The Table API provides methods to apply relational operations on DataSets and Datastreams both in Scala and Java.
-
-The central concept of the Table API is a `Table` which represents a table with relational schema (or relation). Tables can be created from a `DataSet` or `DataStream`, converted into a `DataSet` or `DataStream`, or registered in a table catalog using a `TableEnvironment`. A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine Tables of different TableEnvironments.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-When using Flink's Java DataSet API, DataSets are converted to Tables and Tables to DataSets using a `TableEnvironment`.
-The following example shows:
-
-- how a `DataSet` is converted to a `Table`,
-- how relational queries are specified, and
-- how a `Table` is converted back to a `DataSet`.
-
-{% highlight java %}
-public class WC {
-
- public WC(String word, int count) {
- this.word = word; this.count = count;
- }
-
- public WC() {} // empty constructor to satisfy POJO requirements
-
- public String word;
- public int count;
-}
-
-...
-
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env);
-
-DataSet<WC> input = env.fromElements(
- new WC("Hello", 1),
- new WC("Ciao", 1),
- new WC("Hello", 1));
-
-Table table = tEnv.fromDataSet(input);
-
-Table wordCounts = table
- .groupBy("word")
- .select("word, count.sum as count");
-
-DataSet<WC> result = tableEnv.toDataSet(wordCounts, WC.class);
-{% endhighlight %}
-
-With Java, expressions must be specified by Strings. The embedded expression DSL is not supported.
-
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// register the DataSet cust as table "Customers" with fields derived from the dataset
-tableEnv.registerDataSet("Customers", cust)
-
-// register the DataSet ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataSet("Orders", ord, "user, product, amount");
-{% endhighlight %}
-
-Please refer to the Javadoc for a full list of supported operations and a description of the expression syntax.
-</div>
-
-<div data-lang="scala" markdown="1">
-The Table API is enabled by importing `org.apache.flink.table.api.scala._`. This enables
-implicit conversions to convert a `DataSet` or `DataStream` to a Table. The following example shows:
-
-- how a `DataSet` is converted to a `Table`,
-- how relational queries are specified, and
-- how a `Table` is converted back to a `DataSet`.
-
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.table.api.scala._
-
-case class WC(word: String, count: Int)
-
-val env = ExecutionEnvironment.getExecutionEnvironment
-val tEnv = TableEnvironment.getTableEnvironment(env)
-
-val input = env.fromElements(WC("hello", 1), WC("hello", 1), WC("ciao", 1))
-val expr = input.toTable(tEnv)
-val result = expr
- .groupBy('word)
- .select('word, 'count.sum as 'count)
- .toDataSet[WC]
-{% endhighlight %}
-
-The expression DSL uses Scala symbols to refer to field names and code generation to
-transform expressions to efficient runtime code. Please note that the conversion to and from
-Tables only works when using Scala case classes or Java POJOs. Please refer to the [Type Extraction and Serialization]({{ site.baseurl }}/internals/types_serialization.html) section
-to learn the characteristics of a valid POJO.
-
-Another example shows how to join two Tables:
-
-{% highlight scala %}
-case class MyResult(a: String, d: Int)
-
-val input1 = env.fromElements(...).toTable(tEnv).as('a, 'b)
-val input2 = env.fromElements(...).toTable(tEnv, 'c, 'd)
-
-val joined = input1.join(input2)
- .where("a = c && d > 42")
- .select("a, d")
- .toDataSet[MyResult]
-{% endhighlight %}
-
-Notice, how the field names of a Table can be changed with `as()` or specified with `toTable()` when converting a DataSet to a Table. In addition, the example shows how to use Strings to specify relational expressions.
-
-Creating a `Table` from a `DataStream` works in a similar way.
-The following example shows how to convert a `DataStream` to a `Table` and filter it with the Table API.
-
-{% highlight scala %}
-import org.apache.flink.api.scala._
-import org.apache.flink.table.api.scala._
-
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tEnv = TableEnvironment.getTableEnvironment(env)
-
-val inputStream = env.addSource(...)
-val result = inputStream
- .toTable(tEnv, 'a, 'b, 'c)
- .filter('a === 3)
-val resultStream = result.toDataStream[Row]
-{% endhighlight %}
-
-Please refer to the Scaladoc for a full list of supported operations and a description of the expression syntax.
-</div>
-</div>
-
-{% top %}
-
-
-### Access a registered Table
-
-A registered table can be accessed from a `TableEnvironment` as follows:
-
-- `tEnv.scan("tName")` scans a `Table` that was registered as `"tName"` in a `BatchTableEnvironment`.
-- `tEnv.ingest("tName")` ingests a `Table` that was registered as `"tName"` in a `StreamTableEnvironment`.
-
-{% top %}
-
-### Table API Operators
-
-The Table API features a domain-specific language to execute language-integrated queries on structured data in Scala and Java.
-This section gives a brief overview of the available operators. You can find more details of operators in the [Javadoc](http://flink.apache.org/docs/latest/api/java/org/apache/flink/table/api/Table.html).
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Operators</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td><strong>Select</strong></td>
- <td>
- <p>Similar to a SQL SELECT statement. Performs a select operation.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.select("a, c as d");
-{% endhighlight %}
- <p>You can use star (<code>*</code>) to act as a wild card, selecting all of the columns in the table.</p>
-{% highlight java %}
-Table result = in.select("*");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>As</strong></td>
- <td>
- <p>Renames fields.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.as("d, e, f");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Where / Filter</strong></td>
- <td>
- <p>Similar to a SQL WHERE clause. Filters out rows that do not pass the filter predicate.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.where("b = 'red'");
-{% endhighlight %}
-or
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.filter("a % 2 = 0");
-{% endhighlight %}
- </td>
- </tr>
- <tr>
- <td><strong>GroupBy</strong></td>
- <td>
- <p>Similar to a SQL GROUPBY clause. Groups the rows on the grouping keys, with a following aggregation
- operator to aggregate rows group-wise.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.groupBy("a").select("a, b.sum as d");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Join</strong></td>
- <td>
- <p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined through join operator or using a where or filter operator.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "d, e, f");
-Table result = left.join(right).where("a = d").select("a, b, e");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>LeftOuterJoin</strong></td>
- <td>
- <p>Similar to a SQL LEFT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "d, e, f");
-Table result = left.leftOuterJoin(right, "a = d").select("a, b, e");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>RightOuterJoin</strong></td>
- <td>
- <p>Similar to a SQL RIGHT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "d, e, f");
-Table result = left.rightOuterJoin(right, "a = d").select("a, b, e");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>FullOuterJoin</strong></td>
- <td>
- <p>Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "d, e, f");
-Table result = left.fullOuterJoin(right, "a = d").select("a, b, e");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Union</strong></td>
- <td>
- <p>Similar to a SQL UNION clause. Unions two tables with duplicate records removed. Both tables must have identical field types.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "a, b, c");
-Table result = left.union(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>UnionAll</strong></td>
- <td>
- <p>Similar to a SQL UNION ALL clause. Unions two tables. Both tables must have identical field types.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "a, b, c");
-Table result = left.unionAll(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Intersect</strong></td>
- <td>
- <p>Similar to a SQL INTERSECT clause. Intersect returns records that exist in both tables. If a record is present one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Both tables must have identical field types.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "d, e, f");
-Table result = left.intersect(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>IntersectAll</strong></td>
- <td>
- <p>Similar to a SQL INTERSECT ALL clause. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Both tables must have identical field types.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "d, e, f");
-Table result = left.intersectAll(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Minus</strong></td>
- <td>
- <p>Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "a, b, c");
-Table result = left.minus(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>MinusAll</strong></td>
- <td>
- <p>Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.</p>
-{% highlight java %}
-Table left = tableEnv.fromDataSet(ds1, "a, b, c");
-Table right = tableEnv.fromDataSet(ds2, "a, b, c");
-Table result = left.minusAll(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Distinct</strong></td>
- <td>
- <p>Similar to a SQL DISTINCT clause. Returns records with distinct value combinations.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.distinct();
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Order By</strong></td>
- <td>
- <p>Similar to a SQL ORDER BY clause. Returns records globally sorted across all parallel partitions.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.orderBy("a.asc");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Limit</strong></td>
- <td>
- <p>Similar to a SQL LIMIT clause. Limits a sorted result to a specified number of records from an offset position. Limit is technically part of the Order By operator and thus must be preceded by it.</p>
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.orderBy("a.asc").limit(3); // returns unlimited number of records beginning with the 4th record
-{% endhighlight %}
-or
-{% highlight java %}
-Table in = tableEnv.fromDataSet(ds, "a, b, c");
-Table result = in.orderBy("a.asc").limit(3, 5); // returns 5 records beginning with the 4th record
-{% endhighlight %}
- </td>
- </tr>
-
- </tbody>
-</table>
-
-</div>
-<div data-lang="scala" markdown="1">
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Operators</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td><strong>Select</strong></td>
- <td>
- <p>Similar to a SQL SELECT statement. Performs a select operation.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.select('a, 'c as 'd);
-{% endhighlight %}
- <p>You can use star (<code>*</code>) to act as a wild card, selecting all of the columns in the table.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.select('*);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>As</strong></td>
- <td>
- <p>Renames fields.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv).as('a, 'b, 'c);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Where / Filter</strong></td>
- <td>
- <p>Similar to a SQL WHERE clause. Filters out rows that do not pass the filter predicate.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.filter('a % 2 === 0)
-{% endhighlight %}
-or
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.where('b === "red");
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>GroupBy</strong></td>
- <td>
- <p>Similar to a SQL GROUPBY clause. Groups rows on the grouping keys, with a following aggregation
- operator to aggregate rows group-wise.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.groupBy('a).select('a, 'b.sum as 'd);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Join</strong></td>
- <td>
- <p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and an equality join predicate must be defined using a where or filter operator.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'd, 'e, 'f);
-val result = left.join(right).where('a === 'd).select('a, 'b, 'e);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>LeftOuterJoin</strong></td>
- <td>
- <p>Similar to a SQL LEFT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
-{% highlight scala %}
-val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
-val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
-val result = left.leftOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>RightOuterJoin</strong></td>
- <td>
- <p>Similar to a SQL RIGHT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
-{% highlight scala %}
-val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
-val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
-val result = left.rightOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>FullOuterJoin</strong></td>
- <td>
- <p>Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
-{% highlight scala %}
-val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
-val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
-val result = left.fullOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Union</strong></td>
- <td>
- <p>Similar to a SQL UNION clause. Unions two tables with duplicate records removed, both tables must have identical field types.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
-val result = left.union(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>UnionAll</strong></td>
- <td>
- <p>Similar to a SQL UNION ALL clause. Unions two tables, both tables must have identical field types.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
-val result = left.unionAll(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Intersect</strong></td>
- <td>
- <p>Similar to a SQL INTERSECT clause. Intersect returns records that exist in both tables. If a record is present in one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Both tables must have identical field types.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'e, 'f, 'g);
-val result = left.intersect(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>IntersectAll</strong></td>
- <td>
- <p>Similar to a SQL INTERSECT ALL clause. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Both tables must have identical field types.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'e, 'f, 'g);
-val result = left.intersectAll(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Minus</strong></td>
- <td>
- <p>Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
-val result = left.minus(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>MinusAll</strong></td>
- <td>
- <p>Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.</p>
-{% highlight scala %}
-val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
-val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
-val result = left.minusAll(right);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Distinct</strong></td>
- <td>
- <p>Similar to a SQL DISTINCT clause. Returns records with distinct value combinations.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.distinct();
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Order By</strong></td>
- <td>
- <p>Similar to a SQL ORDER BY clause. Returns records globally sorted across all parallel partitions.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.orderBy('a.asc);
-{% endhighlight %}
- </td>
- </tr>
-
- <tr>
- <td><strong>Limit</strong></td>
- <td>
- <p>Similar to a SQL LIMIT clause. Limits a sorted result to a specified number of records from an offset position. Limit is technically part of the Order By operator and thus must be preceded by it.</p>
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.orderBy('a.asc).limit(3); // returns unlimited number of records beginning with the 4th record
-{% endhighlight %}
-or
-{% highlight scala %}
-val in = ds.toTable(tableEnv, 'a, 'b, 'c);
-val result = in.orderBy('a.asc).limit(3, 5); // returns 5 records beginning with the 4th record
-{% endhighlight %}
- </td>
- </tr>
-
- </tbody>
-</table>
-</div>
-</div>
-
-{% top %}
-
-### Expression Syntax
-Some of the operators in previous sections expect one or more expressions. Expressions can be specified using an embedded Scala DSL or as Strings. Please refer to the examples above to learn how expressions can be specified.
-
-This is the EBNF grammar for expressions:
-
-{% highlight ebnf %}
-
-expressionList = expression , { "," , expression } ;
-
-expression = timeIndicator | overConstant | alias ;
-
-alias = logic | ( logic , "as" , fieldReference ) | ( logic , "as" , "(" , fieldReference , { "," , fieldReference } , ")" ) ;
-
-logic = comparison , [ ( "&&" | "||" ) , comparison ] ;
-
-comparison = term , [ ( "=" | "==" | "===" | "!=" | "!==" | ">" | ">=" | "<" | "<=" ) , term ] ;
-
-term = product , [ ( "+" | "-" ) , product ] ;
-
-product = unary , [ ( "*" | "/" | "%") , unary ] ;
-
-unary = [ "!" | "-" ] , composite ;
-
-composite = over | nullLiteral | suffixed | atom ;
-
-suffixed = interval | cast | as | if | functionCall ;
-
-interval = timeInterval | rowInterval ;
-
-timeInterval = composite , "." , ("year" | "years" | "month" | "months" | "day" | "days" | "hour" | "hours" | "minute" | "minutes" | "second" | "seconds" | "milli" | "millis") ;
-
-rowInterval = composite , "." , "rows" ;
-
-cast = composite , ".cast(" , dataType , ")" ;
-
-dataType = "BYTE" | "SHORT" | "INT" | "LONG" | "FLOAT" | "DOUBLE" | "BOOLEAN" | "STRING" | "DECIMAL" | "SQL_DATE" | "SQL_TIME" | "SQL_TIMESTAMP" | "INTERVAL_MONTHS" | "INTERVAL_MILLIS" | ( "PRIMITIVE_ARRAY" , "(" , dataType , ")" ) | ( "OBJECT_ARRAY" , "(" , dataType , ")" ) ;
-
-as = composite , ".as(" , fieldReference , ")" ;
-
-if = composite , ".?(" , expression , "," , expression , ")" ;
-
-functionCall = composite , "." , functionIdentifier , [ "(" , [ expression , { "," , expression } ] , ")" ] ;
-
-atom = ( "(" , expression , ")" ) | literal | fieldReference ;
-
-fieldReference = "*" | identifier ;
-
-nullLiteral = "Null(" , dataType , ")" ;
-
-timeIntervalUnit = "YEAR" | "YEAR_TO_MONTH" | "MONTH" | "DAY" | "DAY_TO_HOUR" | "DAY_TO_MINUTE" | "DAY_TO_SECOND" | "HOUR" | "HOUR_TO_MINUTE" | "HOUR_TO_SECOND" | "MINUTE" | "MINUTE_TO_SECOND" | "SECOND" ;
-
-timePointUnit = "YEAR" | "MONTH" | "DAY" | "HOUR" | "MINUTE" | "SECOND" | "QUARTER" | "WEEK" | "MILLISECOND" | "MICROSECOND" ;
-
-over = composite , "over" , fieldReference ;
-
-overConstant = "current_row" | "current_range" | "unbounded_row" | "unbounded_row" ;
-
-timeIndicator = fieldReference , "." , ( "proctime" | "rowtime" ) ;
-
-{% endhighlight %}
-
-Here, `literal` is a valid Java literal, `fieldReference` specifies a column in the data (or all columns if `*` is used), and `functionIdentifier` specifies a supported scalar function. The
-column names and function names follow Java identifier syntax. Expressions specified as Strings can also use prefix notation instead of suffix notation to call operators and functions.
-
-If working with exact numeric values or large decimals is required, the Table API also supports Java's BigDecimal type. In the Scala Table API decimals can be defined by `BigDecimal("123456")` and in Java by appending a "p" for precise e.g. `123456p`.
-
-In order to work with temporal values the Table API supports Java SQL's Date, Time, and Timestamp types. In the Scala Table API literals can be defined by using `java.sql.Date.valueOf("2016-06-27")`, `java.sql.Time.valueOf("10:10:42")`, or `java.sql.Timestamp.valueOf("2016-06-27 10:10:42.123")`. The Java and Scala Table API also support calling `"2016-06-27".toDate()`, `"10:10:42".toTime()`, and `"2016-06-27 10:10:42.123".toTimestamp()` for converting Strings into temporal types. *Note:* Since Java's temporal SQL types are time zone dependent, please make sure that the Flink Client and all TaskManagers use the same time zone.
-
-Temporal intervals can be represented as number of months (`Types.INTERVAL_MONTHS`) or number of milliseconds (`Types.INTERVAL_MILLIS`). Intervals of same type can be added or subtracted (e.g. `1.hour + 10.minutes`). Intervals of milliseconds can be added to time points (e.g. `"2016-08-10".toDate + 5.days`).
-
-{% top %}
-
-### Windows
-
-The Table API is a declarative API to define queries on batch and streaming tables. Projection, selection, and union operations can be applied both on streaming and batch tables without additional semantics. Aggregations on (possibly) infinite streaming tables, however, can only be computed on finite groups of records. Window aggregates group rows into finite groups based on time or row-count intervals and evaluate aggregation functions once per group. For batch tables, windows are a convenient shortcut to group records by time intervals.
-
-Windows are defined using the `window(w: Window)` clause and require an alias, which is specified using the `as` clause. In order to group a table by a window, the window alias must be referenced in the `groupBy(...)` clause like a regular grouping attribute.
-The following example shows how to define a window aggregation on a table.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Table table = input
- .window([Window w].as("w")) // define window with alias w
- .groupBy("w") // group the table by window w
- .select("b.sum"); // aggregate
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val table = input
- .window([w: Window] as 'w) // define window with alias w
- .groupBy('w) // group the table by window w
- .select('b.sum) // aggregate
-{% endhighlight %}
-</div>
-</div>
-
-In streaming environments, window aggregates can only be computed in parallel if they group on one or more attributes in addition to the window, i.e., the `groupBy(...)` clause references a window alias and at least one additional attribute. A `groupBy(...)` clause that only references a window alias (such as in the example above) can only be evaluated by a single, non-parallel task.
-The following example shows how to define a window aggregation with additional grouping attributes.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Table table = input
- .window([Window w].as("w")) // define window with alias w
- .groupBy("w, a") // group the table by attribute a and window w
- .select("a, b.sum"); // aggregate
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val table = input
- .window([w: Window] as 'w) // define window with alias w
- .groupBy('w, 'a) // group the table by attribute a and window w
- .select('a, 'b.sum) // aggregate
-{% endhighlight %}
-</div>
-</div>
-
-The `Window` parameter defines how rows are mapped to windows. `Window` is not an interface that users can implement. Instead, the Table API provides a set of predefined `Window` classes with specific semantics, which are translated into underlying `DataStream` or `DataSet` operations. The supported window definitions are listed below. Window properties such as the start and end timestamp of a time window can be added in the select statement as a property of the window alias as `w.start` and `w.end`, respectively.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-Table table = input
- .window([Window w].as("w")) // define window with alias w
- .groupBy("w, a") // group the table by attribute a and window w
- .select("a, w.start, w.end, b.count"); // aggregate and add window start and end timestamps
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val table = input
- .window([w: Window] as 'w) // define window with alias w
- .groupBy('w, 'a) // group the table by attribute a and window w
- .select('a, 'w.start, 'w.end, 'b.count) // aggregate and add window start and end timestamps
-{% endhighlight %}
-</div>
-</div>
-
-#### Tumble (Tumbling Windows)
-
-A tumbling window assigns rows to non-overlapping, continuous windows of fixed length. For example, a tumbling window of 5 minutes groups rows in 5 minutes intervals. Tumbling windows can be defined on event-time, processing-time, or on a row-count.
-
-Tumbling windows are defined by using the `Tumble` class as follows:
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Method</th>
- <th class="text-left" style="width: 20%">Required?</th>
- <th class="text-left">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td><code>over</code></td>
- <td>Required.</td>
- <td>Defines the length the window, either as time or row-count interval.</td>
- </tr>
- <tr>
- <td><code>on</code></td>
- <td>Required for streaming event-time windows and windows on batch tables.</td>
- <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped.</td>
- </tr>
- <tr>
- <td><code>as</code></td>
- <td>Required.</td>
- <td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
- </tr>
- </tbody>
-</table>
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// Tumbling Event-time Window
-.window(Tumble.over("10.minutes").on("rowtime").as("w"));
-
-// Tumbling Processing-time Window
-.window(Tumble.over("10.minutes").as("w"));
-
-// Tumbling Row-count Window
-.window(Tumble.over("10.rows").as("w"));
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// Tumbling Event-time Window
-.window(Tumble over 10.minutes on 'rowtime as 'w)
-
-// Tumbling Processing-time Window
-.window(Tumble over 10.minutes as 'w)
-
-// Tumbling Row-count Window
-.window(Tumble over 10.rows as 'w)
-{% endhighlight %}
-</div>
-</div>
-
-#### Slide (Sliding Windows)
-
-A sliding window has a fixed size and slides by a specified slide interval. If the slide interval is smaller than the window size, sliding windows are overlapping. Thus, rows can be assigned to multiple windows. For example, a sliding window of 15 minutes size and 5 minute slide interval assigns each row to 3 different windows of 15 minute size, which are evaluated in an interval of 5 minutes. Sliding windows can be defined on event-time, processing-time, or on a row-count.
-
-Sliding windows are defined by using the `Slide` class as follows:
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Method</th>
- <th class="text-left" style="width: 20%">Required?</th>
- <th class="text-left">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td><code>over</code></td>
- <td>Required.</td>
- <td>Defines the length of the window, either as time or row-count interval.</td>
- </tr>
- <tr>
- <td><code>every</code></td>
- <td>Required.</td>
- <td>Defines the slide interval, either as time or row-count interval. The slide interval must be of the same type as the size interval.</td>
- </tr>
- <tr>
- <td><code>on</code></td>
- <td>Required for event-time windows and windows on batch tables.</td>
- <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped</td>
- </tr>
- <tr>
- <td><code>as</code></td>
- <td>Required.</td>
- <td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
- </tr>
- </tbody>
-</table>
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// Sliding Event-time Window
-.window(Slide.over("10.minutes").every("5.minutes").on("rowtime").as("w"));
-
-// Sliding Processing-time window
-.window(Slide.over("10.minutes").every("5.minutes").as("w"));
-
-// Sliding Row-count window
-.window(Slide.over("10.rows").every("5.rows").as("w"));
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// Sliding Event-time Window
-.window(Slide over 10.minutes every 5.minutes on 'rowtime as 'w)
-
-// Sliding Processing-time window
-.window(Slide over 10.minutes every 5.minutes as 'w)
-
-// Sliding Row-count window
-.window(Slide over 10.rows every 5.rows as 'w)
-{% endhighlight %}
-</div>
-</div>
-
-#### Session (Session Windows)
-
-Session windows do not have a fixed size but their bounds are defined by an interval of inactivity, i.e., a session window is closes if no event appears for a defined gap period. For example a session window with a 30 minute gap starts when a row is observed after 30 minutes inactivity (otherwise the row would be added to an existing window) and is closed if no row is added within 30 minutes. Session windows can work on event-time or processing-time.
-
-A session window is defined by using the `Session` class as follows:
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 20%">Method</th>
- <th class="text-left" style="width: 20%">Required?</th>
- <th class="text-left">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td><code>withGap</code></td>
- <td>Required.</td>
- <td>Defines the gap between two windows as time interval.</td>
- </tr>
- <tr>
- <td><code>on</code></td>
- <td>Required for event-time windows and windows on batch tables.</td>
- <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped</td>
- </tr>
- <tr>
- <td><code>as</code></td>
- <td>Required.</td>
- <td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
- </tr>
- </tbody>
-</table>
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-// Session Event-time Window
-.window(Session.withGap("10.minutes").on("rowtime").as("w"));
-
-// Session Processing-time Window
-.window(Session.withGap("10.minutes").as("w"));
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// Session Event-time Window
-.window(Session withGap 10.minutes on 'rowtime as 'w)
-
-// Session Processing-time Window
-.window(Session withGap 10.minutes as 'w)
-{% endhighlight %}
-</div>
-</div>
-
-#### Limitations
-
-Currently the following features are not supported yet:
-
-- Row-count windows on event-time
-- Non-grouped session windows on batch tables
-- Sliding windows on batch tables
-
-SQL
-----
-SQL queries are specified using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table` which can be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries, or written to a `TableSink` (see [Writing Tables to External Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single DataStream or DataSet program.
-
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be registered in the `TableEnvironment` in order to be accessible by a SQL query (see [Registering Tables](#registering-tables)). For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).
-
-*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*
-
-### SQL on Batch Tables
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// read a DataSet from an external source
-DataSet<Tuple3<Long, String, Integer>> ds = env.readCsvFile(...);
-
-// call SQL on unregistered tables
-Table table = tableEnv.toTable(ds, "user, product, amount");
-Table result = tableEnv.sql(
- "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
-
-// call SQL on registered tables
-// register the DataSet as table "Orders"
-tableEnv.registerDataSet("Orders", ds, "user, product, amount");
-// run a SQL query on the Table and retrieve the result as a new Table
-Table result2 = tableEnv.sql(
- "SELECT SUM(amount) FROM Orders WHERE product LIKE '%Rubber%'");
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-// read a DataSet from an external source
-val ds: DataSet[(Long, String, Integer)] = env.readCsvFile(...)
-
-// call SQL on unregistered tables
-val table = ds.toTable(tableEnv, 'user, 'product, 'amount)
-val result = tableEnv.sql(
- s"SELECT SUM(amount) FROM $table WHERE product LIKE '%Rubber%'")
-
-// call SQL on registered tables
-// register the DataSet under the name "Orders"
-tableEnv.registerDataSet("Orders", ds, 'user, 'product, 'amount)
-// run a SQL query on the Table and retrieve the result as a new Table
-val result2 = tableEnv.sql(
- "SELECT SUM(amount) FROM Orders WHERE product LIKE '%Rubber%'")
-{% endhighlight %}
-</div>
-</div>
-
-#### Limitations
-
-The current version supports selection (filter), projection, inner equi-joins, grouping, aggregates, and sorting on batch tables.
-
-Among others, the following SQL features are not supported, yet:
-
-- Timestamps and intervals are limited to milliseconds precision
-- Interval arithmetic is currenly limited
-- Non-equi joins and Cartesian products
-- Efficient grouping sets
-
-*Note: Tables are joined in the order in which they are specified in the `FROM` clause. In some cases the table order must be manually tweaked to resolve Cartesian products.*
-
-### SQL on Streaming Tables
-
-SQL queries can be executed on streaming Tables (Tables backed by `DataStream` or `StreamTableSource`) like standard SQL.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// ingest a DataStream from an external source
-DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
-
-// call SQL on unregistered tables
-Table table = tableEnv.toTable(ds, "user, product, amount");
-Table result = tableEnv.sql(
- "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
-
-// call SQL on registered tables
-// register the DataStream as table "Orders"
-tableEnv.registerDataStream("Orders", ds, "user, product, amount");
-// run a SQL query on the Table and retrieve the result as a new Table
-Table result2 = tableEnv.sql(
- "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-// read a DataStream from an external source
-val ds: DataStream[(Long, String, Integer)] = env.addSource(...)
-
-// call SQL on unregistered tables
-val table = ds.toTable(tableEnv, 'user, 'product, 'amount)
-val result = tableEnv.sql(
- s"SELECT SUM(amount) FROM $table WHERE product LIKE '%Rubber%'")
-
-// call SQL on registered tables
-// register the DataStream under the name "Orders"
-tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)
-// run a SQL query on the Table and retrieve the result as a new Table
-val result2 = tableEnv.sql(
- "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'")
-{% endhighlight %}
-</div>
-</div>
-
-#### Limitations
-
-Joins, set operations, and non-windowed aggregations are not supported yet.
-`UNNEST` supports only arrays and does not support `WITH ORDINALITY` yet.
-
-{% top %}
-
-### Group Windows
-
-Group windows are defined in the `GROUP BY` clause of a SQL query. Just like queries with regular `GROUP BY` clauses, queries with a `GROUP BY` clause that includes a group window function compute a single result row per group. The following group windows functions are supported for SQL on batch and streaming tables.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 30%">Group Window Function</th>
- <th class="text-left">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td><code>TUMBLE(time_attr, interval)</code></td>
- <td>Defines a tumbling time window. A tumbling time window assigns rows to non-overlapping, continuous windows with a fixed duration (<code>interval</code>). For example, a tumbling window of 5 minutes groups rows in 5 minutes intervals. Tumbling windows can be defined on event-time (stream + batch) or processing-time (stream).</td>
- </tr>
- <tr>
- <td><code>HOP(time_attr, interval, interval)</code></td>
- <td>Defines a hopping time window (called sliding window in the Table API). A hopping time window has a fixed duration (second <code>interval</code> parameter) and hops by a specified hop interval (first <code>interval</code> parameter). If the hop interval is smaller than the window size, hopping windows are overlapping. Thus, rows can be assigned to multiple windows. For example, a hopping window of 15 minutes size and 5 minute hop interval assigns each row to 3 different windows of 15 minute size, which are evaluated in an interval of 5 minutes. Hopping windows can be defined on event-time (stream + batch) or processing-time (stream).</td>
- </tr>
- <tr>
- <td><code>SESSION(time_attr, interval)</code></td>
- <td>Defines a session time window. Session time windows do not have a fixed duration but their bounds are defined by a time <code>interval</code> of inactivity, i.e., a session window is closed if no event appears for a defined gap period. For example a session window with a 30 minute gap starts when a row is observed after 30 minutes inactivity (otherwise the row would be added to an existing window) and is closed if no row is added within 30 minutes. Session windows can work on event-time (stream + batch) or processing-time (stream).</td>
- </tr>
- </tbody>
-</table>
-
-For SQL queries on streaming tables, the `time_attr` argument of the group window function must be one of the `rowtime()` or `proctime()` time-indicators, which distinguish between event or processing time, respectively. For SQL on batch tables, the `time_attr` argument of the group window function must be an attribute of type `TIMESTAMP`.
-
-#### Selecting Group Window Start and End Timestamps
-
-The start and end timestamps of group windows can be selected with the following auxiliary functions:
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 40%">Auxiliary Function</th>
- <th class="text-left">Description</th>
- </tr>
- </thead>
-
- <tbody>
- <tr>
- <td>
- <code>TUMBLE_START(time_attr, interval)</code><br/>
- <code>HOP_START(time_attr, interval, interval)</code><br/>
- <code>SESSION_START(time_attr, interval)</code><br/>
- </td>
- <td>Returns the start timestamp of the corresponding tumbling, hopping, and session window.</td>
- </tr>
- <tr>
- <td>
- <code>TUMBLE_END(time_attr, interval)</code><br/>
- <code>HOP_END(time_attr, interval, interval)</code><br/>
- <code>SESSION_END(time_attr, interval)</code><br/>
- </td>
- <td>Returns the end timestamp of the corresponding tumbling, hopping, and session window.</td>
- </tr>
- </tbody>
-</table>
-
-Note that the auxiliary functions must be called with exactly same arguments as the group window function in the `GROUP BY` clause.
-
-The following examples show how to specify SQL queries with group windows on streaming tables.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// ingest a DataStream from an external source
-DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
-// register the DataStream as table "Orders"
-tableEnv.registerDataStream("Orders", ds, "user, product, amount");
-
-// compute SUM(amount) per day (in event-time)
-Table result1 = tableEnv.sql(
- "SELECT user, " +
- " TUMBLE_START(rowtime(), INTERVAL '1' DAY) as wStart, " +
- " SUM(amount) FROM Orders " +
- "GROUP BY TUMBLE(rowtime(), INTERVAL '1' DAY), user");
-
-// compute SUM(amount) per day (in processing-time)
-Table result2 = tableEnv.sql(
- "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime(), INTERVAL '1' DAY), user");
-
-// compute every hour the SUM(amount) of the last 24 hours in event-time
-Table result3 = tableEnv.sql(
- "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime(), INTERVAL '1' HOUR, INTERVAL '1' DAY), product");
-
-// compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
-Table result4 = tableEnv.sql(
- "SELECT user, " +
- " SESSION_START(rowtime(), INTERVAL '12' HOUR) AS sStart, " +
- " SESSION_END(rowtime(), INTERVAL '12' HOUR) AS snd, " +
- " SUM(amount) " +
- "FROM Orders " +
- "GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user");
-
-{% endhighlight %}
-</div>
-
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val tableEnv = TableEnvironment.getTableEnvironment(env)
-
-// read a DataStream from an external source
-val ds: DataStream[(Long, String, Int)] = env.addSource(...)
-// register the DataStream under the name "Orders"
-tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)
-
-// compute SUM(amount) per day (in event-time)
-val result1 = tableEnv.sql(
- """
- |SELECT
- | user,
- | TUMBLE_START(rowtime(), INTERVAL '1' DAY) as wStart,
- | SUM(amount)
- | FROM Orders
- | GROUP BY TUMBLE(rowtime(), INTERVAL '1' DAY), user
- """.stripMargin)
-
-// compute SUM(amount) per day (in processing-time)
-val result2 = tableEnv.sql(
- "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime(), INTERVAL '1' DAY), user")
-
-// compute every hour the SUM(amount) of the last 24 hours in event-time
-val result3 = tableEnv.sql(
- "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime(), INTERVAL '1' HOUR, INTERVAL '1' DAY), product")
-
-// compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
-val result4 = tableEnv.sql(
- """
- |SELECT
- | user,
- | SESSION_START(rowtime(), INTERVAL '12' HOUR) AS sStart,
- | SESSION_END(rowtime(), INTERVAL '12' HOUR) AS sEnd,
- | SUM(amount)
- | FROM Orders
- | GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user
- """.stripMargin)
-
-{% endhighlight %}
-</div>
-</div>
-
-{% top %}
-
-### SQL Syntax
-
-Flink uses [Apache Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. Currently, Flink SQL only supports query-related SQL syntax and only a subset of the comprehensive SQL standard. The following BNF-grammar describes the supported SQL features:
-
-```
-
-query:
- values
- | {
- select
- | selectWithoutFrom
- | query UNION [ ALL ] query
- | query EXCEPT query
- | query INTERSECT query
- }
- [ ORDER BY orderItem [, orderItem ]* ]
- [ LIMIT { count | ALL } ]
- [ OFFSET start { ROW | ROWS } ]
- [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY]
-
-orderItem:
- expression [ ASC | DESC ]
-
-select:
- SELECT [ ALL | DISTINCT ]
- { * | projectItem [, projectItem ]* }
- FROM tableExpression
- [ WHERE booleanExpression ]
- [ GROUP BY { groupItem [, groupItem ]* } ]
- [ HAVING booleanExpression ]
-
-selectWithoutFrom:
- SELECT [ ALL | DISTINCT ]
- { * | projectItem [, projectItem ]* }
-
-projectItem:
- expression [ [ AS ] columnAlias ]
- | tableAlias . *
-
-tableExpression:
- tableReference [, tableReference ]*
- | tableExpression [ NATURAL ] [ LEFT | RIGHT | FULL ] JOIN tableExpression [ joinCondition ]
-
-joinCondition:
- ON booleanExpression
- | USING '(' column [, column ]* ')'
-
-tableReference:
- tablePrimary
- [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]
-
-tablePrimary:
- [ TABLE ] [ [ catalogName . ] schemaName . ] tableName
- | LATERAL TABLE '(' functionName '(' expression [, expression ]* ')' ')'
- | UNNEST '(' expression ')'
-
-values:
- VALUES expression [, expression ]*
-
-groupItem:
- expression
- | '(' ')'
- | '(' expression [, expression ]* ')'
- | CUBE '(' expression [, expression ]* ')'
- | ROLLUP '(' expression [, expression ]* ')'
- | GROUPING SETS '(' groupItem [, groupItem ]* ')'
-```
-
-For a better definition of SQL queries within a Java String, Flink SQL uses a lexical policy similar to Java:
-
-- The case of identifiers is preserved whether or not they are quoted.
-- After which, identifiers are matched case-sensitively.
-- Unlike Java, back-ticks allow identifiers to contain non-alphanumeric characters (e.g. <code>"SELECT a AS `my field` FROM t"</code>).
-
-
-{% top %}
-
-### Reserved Keywords
-
-Although not every SQL feature is implemented yet, some string combinations are already reserved as keywords for future use. If you want to use one of the following strings as a field name, make sure to surround them with backticks (e.g. `` `value` ``, `` `count` ``).
-
-{% highlight sql %}
-
-A, ABS, ABSOLUTE, ACTION, ADA, ADD, ADMIN, AFTER, ALL, ALLOCATE, ALLOW, ALTER, ALWAYS, AND, ANY, ARE, ARRAY, AS, ASC, ASENSITIVE, ASSERTION, ASSIGNMENT, ASYMMETRIC, AT, ATOMIC, ATTRIBUTE, ATTRIBUTES, AUTHORIZATION, AVG, BEFORE, BEGIN, BERNOULLI, BETWEEN, BIGINT, BINARY, BIT, BLOB, BOOLEAN, BOTH, BREADTH, BY, C, CALL, CALLED, CARDINALITY, CASCADE, CASCADED, CASE, CAST, CATALOG, CATALOG_NAME, CEIL, CEILING, CENTURY, CHAIN, CHAR, CHARACTER, CHARACTERISTICTS, CHARACTERS, CHARACTER_LENGTH, CHARACTER_SET_CATALOG, CHARACTER_SET_NAME, CHARACTER_SET_SCHEMA, CHAR_LENGTH, CHECK, CLASS_ORIGIN, CLOB, CLOSE, COALESCE, COBOL, COLLATE, COLLATION, COLLATION_CATALOG, COLLATION_NAME, COLLATION_SCHEMA, COLLECT, COLUMN, COLUMN_NAME, COMMAND_FUNCTION, COMMAND_FUNCTION_CODE, COMMIT, COMMITTED, CONDITION, CONDITION_NUMBER, CONNECT, CONNECTION, CONNECTION_NAME, CONSTRAINT, CONSTRAINTS, CONSTRAINT_CATALOG, CONSTRAINT_NAME, CONSTRAINT_SCHEMA, CONSTRUCTOR, CONTAINS, CONTINUE, CONVERT, CORR, CORRESPONDING, COUN
T, COVAR_POP, COVAR_SAMP, CREATE, CROSS, CUBE, CUME_DIST, CURRENT, CURRENT_CATALOG, CURRENT_DATE, CURRENT_DEFAULT_TRANSFORM_GROUP, CURRENT_PATH, CURRENT_ROLE, CURRENT_SCHEMA, CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_TRANSFORM_GROUP_FOR_TYPE, CURRENT_USER, CURSOR, CURSOR_NAME, CYCLE, DATA, DATABASE, DATE, DATETIME_INTERVAL_CODE, DATETIME_INTERVAL_PRECISION, DAY, DEALLOCATE, DEC, DECADE, DECIMAL, DECLARE, DEFAULT, DEFAULTS, DEFERRABLE, DEFERRED, DEFINED, DEFINER, DEGREE, DELETE, DENSE_RANK, DEPTH, DEREF, DERIVED, DESC, DESCRIBE, DESCRIPTION, DESCRIPTOR, DETERMINISTIC, DIAGNOSTICS, DISALLOW, DISCONNECT, DISPATCH, DISTINCT, DOMAIN, DOUBLE, DOW, DOY, DROP, DYNAMIC, DYNAMIC_FUNCTION, DYNAMIC_FUNCTION_CODE, EACH, ELEMENT, ELSE, END, END-EXEC, EPOCH, EQUALS, ESCAPE, EVERY, EXCEPT, EXCEPTION, EXCLUDE, EXCLUDING, EXEC, EXECUTE, EXISTS, EXP, EXPLAIN, EXTEND, EXTERNAL, EXTRACT, FALSE, FETCH, FILTER, FINAL, FIRST, FIRST_VALUE, FLOAT, FLOOR, FOLLOWING, FOR, FOREIGN, FORTRAN, FOUND, FRAC_SECOND, F
REE, FROM, FULL, FUNCTION, FUSION, G, GENERAL, GENERATED, GET, GLOBAL, GO, GOTO, GRANT, GRANTED, GROUP, GROUPING, HAVING, HIERARCHY, HOLD, HOUR, IDENTITY, IMMEDIATE, IMPLEMENTATION, IMPORT, IN, INCLUDING, INCREMENT, INDICATOR, INITIALLY, INNER, INOUT, INPUT, INSENSITIVE, INSERT, INSTANCE, INSTANTIABLE, INT, INTEGER, INTERSECT, INTERSECTION, INTERVAL, INTO, INVOKER, IS, ISOLATION, JAVA, JOIN, K, KEY, KEY_MEMBER, KEY_TYPE, LABEL, LANGUAGE, LARGE, LAST, LAST_VALUE, LATERAL, LEADING, LEFT, LENGTH, LEVEL, LIBRARY, LIKE, LIMIT, LN, LOCAL, LOCALTIME, LOCALTIMESTAMP, LOCATOR, LOWER, M, MAP, MATCH, MATCHED, MAX, MAXVALUE, MEMBER, MERGE, MESSAGE_LENGTH, MESSAGE_OCTET_LENGTH, MESSAGE_TEXT, METHOD, MICROSECOND, MILLENNIUM, MIN, MINUTE, MINVALUE, MOD, MODIFIES, MODULE, MONTH, MORE, MULTISET, MUMPS, NAME, NAMES, NATIONAL, NATURAL, NCHAR, NCLOB, NESTING, NEW, NEXT, NO, NONE, NORMALIZE, NORMALIZED, NOT, NULL, NULLABLE, NULLIF, NULLS, NUMBER, NUMERIC, OBJECT, OCTETS, OCTET_LENGTH, OF, OFFSET, OLD, O
N, ONLY, OPEN, OPTION, OPTIONS, OR, ORDER, ORDERING, ORDINALITY, OTHERS, OUT, OUTER, OUTPUT, OVER, OVERLAPS, OVERLAY, OVERRIDING, PAD, PARAMETER, PARAMETER_MODE, PARAMETER_NAME, PARAMETER_ORDINAL_POSITION, PARAMETER_SPECIFIC_CATALOG, PARAMETER_SPECIFIC_NAME, PARAMETER_SPECIFIC_SCHEMA, PARTIAL, PARTITION, PASCAL, PASSTHROUGH, PATH, PERCENTILE_CONT, PERCENTILE_DISC, PERCENT_RANK, PLACING, PLAN, PLI, POSITION, POWER, PRECEDING, PRECISION, PREPARE, PRESERVE, PRIMARY, PRIOR, PRIVILEGES, PROCEDURE, PUBLIC, QUARTER, RANGE, RANK, READ, READS, REAL, RECURSIVE, REF, REFERENCES, REFERENCING, REGR_AVGX, REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE, REGR_SXX, REGR_SXY, REGR_SYY, RELATIVE, RELEASE, REPEATABLE, RESET, RESTART, RESTRICT, RESULT, RETURN, RETURNED_CARDINALITY, RETURNED_LENGTH, RETURNED_OCTET_LENGTH, RETURNED_SQLSTATE, RETURNS, REVOKE, RIGHT, ROLE, ROLLBACK, ROLLUP, ROUTINE, ROUTINE_CATALOG, ROUTINE_NAME, ROUTINE_SCHEMA, ROW, ROWS, ROW_COUNT, ROW_NUMBER, SAVEPOINT, SCALE
, SCHEMA, SCHEMA_NAME, SCOPE, SCOPE_CATALOGS, SCOPE_NAME, SCOPE_SCHEMA, SCROLL, SEARCH, SECOND, SECTION, SECURITY, SELECT, SELF, SENSITIVE, SEQUENCE, SERIALIZABLE, SERVER, SERVER_NAME, SESSION, SESSION_USER, SET, SETS, SIMILAR, SIMPLE, SIZE, SMALLINT, SOME, SOURCE, SPACE, SPECIFIC, SPECIFICTYPE, SPECIFIC_NAME, SQL, SQLEXCEPTION, SQLSTATE, SQLWARNING, SQL_TSI_DAY, SQL_TSI_FRAC_SECOND, SQL_TSI_HOUR, SQL_TSI_MICROSECOND, SQL_TSI_MINUTE, SQL_TSI_MONTH, SQL_TSI_QUARTER, SQL_TSI_SECOND, SQL_TSI_WEEK, SQL_TSI_YEAR, SQRT, START, STATE, STATEMENT, STATIC, STDDEV_POP, STDDEV_SAMP, STREAM, STRUCTURE, STYLE, SUBCLASS_ORIGIN, SUBMULTISET, SUBSTITUTE, SUBSTRING, SUM, SYMMETRIC, SYSTEM, SYSTEM_USER, TABLE, TABLESAMPLE, TABLE_NAME, TEMPORARY, THEN, TIES, TIME, TIMESTAMP, TIMESTAMPADD, TIMESTAMPDIFF, TIMEZONE_HOUR, TIMEZONE_MINUTE, TINYINT, TO, TOP_LEVEL_COUNT, TRAILING, TRANSACTION, TRANSACTIONS_ACTIVE, TRANSACTIONS_COMMITTED, TRANSACTIONS_ROLLED_BACK, TRANSFORM, TRANSFORMS, TRANSLATE, TRANSLATION,
TREAT, TRIGGER, TRIGGER_CATALOG, TRIGGER_NAME, TRIGGER_SCHEMA, TRIM, TRUE, TYPE, UESCAPE, UNBOUNDED, UNCOMMITTED, UNDER, UNION, UNIQUE, UNKNOWN, UNNAMED, UNNEST, UPDATE, UPPER, UPSERT, USAGE, USER, USER_DEFINED_TYPE_CATALOG, USER_DEFINED_TYPE_CODE, USER_DEFINED_TYPE_NAME, USER_DEFINED_TYPE_SCHEMA, USING, VALUE, VALUES, VARBINARY, VARCHAR, VARYING, VAR_POP, VAR_SAMP, VERSION, VIEW, WEEK, WHEN, WHENEVER, WHERE, WIDTH_BUCKET, WINDOW, WITH, WITHIN, WITHOUT, WORK, WRAPPER, WRITE, XML, YEAR, ZONE
-
-{% endhighlight %}
-
-{% top %}
-
-Data Types
-----------
-
-The Table API is built on top of Flink's DataSet and DataStream API. Internally, it also uses Flink's `TypeInformation` to distinguish between types. The Table API does not support all Flink types so far. All supported simple types are listed in `org.apache.flink.table.api.Types`. The following table summarizes the relation between Table API types, SQL types, and the resulting Java class.
-
-| Table API | SQL | Java type |
-| :--------------------- | :-------------------------- | :--------------------- |
-| `Types.STRING` | `VARCHAR` | `java.lang.String` |
-| `Types.BOOLEAN` | `BOOLEAN` | `java.lang.Boolean` |
-| `Types.BYTE` | `TINYINT` | `java.lang.Byte` |
-| `Types.SHORT` | `SMALLINT` | `java.lang.Short` |
-| `Types.INT` | `INTEGER, INT` | `java.lang.Integer` |
-| `Types.LONG` | `BIGINT` | `java.lang.Long` |
-| `Types.FLOAT` | `REAL, FLOAT` | `java.lang.Float` |
-| `Types.DOUBLE` | `DOUBLE` | `java.lang.Double` |
-| `Types.DECIMAL` | `DECIMAL` | `java.math.BigDecimal` |
-| `Types.DATE` | `DATE` | `java.sql.Date` |
-| `Types.TIME` | `TIME` | `java.sql.Time` |
-| `Types.TIMESTAMP` | `TIMESTAMP(3)` | `java.sql.Timestamp` |
-| `Types.INTERVAL_MONTHS`| `INTERVAL YEAR TO MONTH` | `java.lang.Integer` |
-| `Types.INTERVAL_MILLIS`| `INTERVAL DAY TO SECOND(3)` | `java.lang.Long` |
-| `Types.PRIMITIVE_ARRAY`| `ARRAY` | e.g. `int[]` |
-| `Types.OBJECT_ARRAY` | `ARRAY` | e.g. `java.lang.Byte[]`|
-| `Types.MAP` | `MAP` | `java.util.HashMap` |
-
-
-Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and array types (object or primitive arrays) can be fields of a row.
-
-Generic types are treated as a black box within Table API and SQL yet.
-
-Composite types, however, are fully supported types where fields of a composite type can be accessed using the `.get()` operator in Table API and dot operator (e.g. `MyTable.pojoColumn.myField`) in SQL. Composite types can also be flattened using `.flatten()` in Table API or `MyTable.pojoColumn.*` in SQL.
-
-Array types can be accessed using the `myArray.at(1)` operator in Table API and `myArray[1]` operator in SQL. Array literals can be created using `array(1, 2, 3)` in Table API and `ARRAY[1, 2, 3]` in SQL.
-
-{% top %}
-
-Built-in Functions
-----------------
-
-Both the Table API and SQL come with a set of built-in functions for data transformations. This section gives a brief overview of the available functions so far.
-
-<div class="codetabs" markdown="1">
-<div data-lang="java" markdown="1">
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 40%">Comparison functions</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
-
- <tr>
- <td>
- {% highlight java %}
-ANY === ANY
-{% endhighlight %}
- </td>
- <td>
- <p>Equals.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY !== ANY
-{% endhighlight %}
- </td>
- <td>
- <p>Not equal.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY > ANY
-{% endhighlight %}
- </td>
- <td>
- <p>Greater than.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY >= ANY
-{% endhighlight %}
- </td>
- <td>
- <p>Greater than or equal.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY < ANY
-{% endhighlight %}
- </td>
- <td>
- <p>Less than.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY <= ANY
-{% endhighlight %}
- </td>
- <td>
- <p>Less than or equal.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY.isNull
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if the given expression is null.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-ANY.isNotNull
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if the given expression is not null.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-STRING.like(STRING)
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true, if a string matches the specified LIKE pattern. E.g. "Jo_n%" matches all strings that start with "Jo(arbitrary letter)n".</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-STRING.similar(STRING)
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true, if a string matches the specified SQL regex pattern. E.g. "A+" matches all strings that consist of at least one "A".</p>
- </td>
- </tr>
-
- </tbody>
-</table>
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 40%">Logical functions</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
-
- <tr>
- <td>
- {% highlight java %}
-boolean1 || boolean2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if <i>boolean1</i> is true or <i>boolean2</i> is true. Supports three-valued logic.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-boolean1 && boolean2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if <i>boolean1</i> and <i>boolean2</i> are both true. Supports three-valued logic.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-!BOOLEAN
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if boolean expression is not true; returns null if boolean is null.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-BOOLEAN.isTrue
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if the given boolean expression is true. False otherwise (for null and false).</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-BOOLEAN.isFalse
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if given boolean expression is false. False otherwise (for null and true).</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-BOOLEAN.isNotTrue
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if the given boolean expression is not true (for null and false). False otherwise.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-BOOLEAN.isNotFalse
-{% endhighlight %}
- </td>
- <td>
- <p>Returns true if given boolean expression is not false (for null and true). False otherwise.</p>
- </td>
- </tr>
-
- </tbody>
-</table>
-
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 40%">Arithmetic functions</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
-
- <tr>
- <td>
- {% highlight java %}
-+ numeric
-{% endhighlight %}
- </td>
- <td>
- <p>Returns <i>numeric</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-- numeric
-{% endhighlight %}
- </td>
- <td>
- <p>Returns negative <i>numeric</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-numeric1 + numeric2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns <i>numeric1</i> plus <i>numeric2</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-numeric1 - numeric2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns <i>numeric1</i> minus <i>numeric2</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-numeric1 * numeric2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns <i>numeric1</i> multiplied by <i>numeric2</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-numeric1 / numeric2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns <i>numeric1</i> divided by <i>numeric2</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-numeric1.power(numeric2)
-{% endhighlight %}
- </td>
- <td>
- <p>Returns <i>numeric1</i> raised to the power of <i>numeric2</i>.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.abs()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the absolute value of given value.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-numeric1 % numeric2
-{% endhighlight %}
- </td>
- <td>
- <p>Returns the remainder (modulus) of <i>numeric1</i> divided by <i>numeric2</i>. The result is negative only if <i>numeric1</i> is negative.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.sqrt()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the square root of a given value.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.ln()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the natural logarithm of given value.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.log10()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the base 10 logarithm of given value.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.exp()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the Euler's number raised to the given power.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.ceil()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the smallest integer greater than or equal to a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.floor()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the largest integer less than or equal to a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.sin()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the sine of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.cos()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the cosine of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.tan()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the tangent of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.cot()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the cotangent of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.asin()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the arc sine of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.acos()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the arc cosine of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.atan()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the arc tangent of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.degrees()
-{% endhighlight %}
- </td>
- <td>
- <p>Converts <i>numeric</i> from radians to degrees.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.radians()
-{% endhighlight %}
- </td>
- <td>
- <p>Converts <i>numeric</i> from degrees to radians.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.sign()
-{% endhighlight %}
- </td>
- <td>
- <p>Calculates the signum of a given number.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-NUMERIC.round(INT)
-{% endhighlight %}
- </td>
- <td>
- <p>Rounds the given number to <i>integer</i> places right to the decimal point.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-pi()
-{% endhighlight %}
- </td>
- <td>
- <p>Returns a value that is closer than any other value to pi.</p>
- </td>
- </tr>
-
- </tbody>
-</table>
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 40%">String functions</th>
- <th class="text-center">Description</th>
- </tr>
- </thead>
-
- <tbody>
-
- <tr>
- <td>
- {% highlight java %}
-STRING + STRING
-{% endhighlight %}
- </td>
- <td>
- <p>Concatenates two character strings.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-STRING.charLength()
-{% endhighlight %}
- </td>
- <td>
- <p>Returns the length of a String.</p>
- </td>
- </tr>
-
- <tr>
- <td>
- {% highlight java %}
-STRING.upperCase()
-{% endhighlight %}
- </td>
- <td>
- <p>Returns all of the characters in a string in
<TRUNCATED>
[06/10] flink git commit: [FLINK-6749] [table] [docs] Updated Table
API / SQL docs: SQL
Posted by fh...@apache.org.
[FLINK-6749] [table] [docs] Updated Table API / SQL docs: SQL
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/ddae51fb
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/ddae51fb
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/ddae51fb
Branch: refs/heads/master
Commit: ddae51fba2e7cb264015fe3530159ea92662b22b
Parents: fffce09
Author: Haohui Mai <wh...@apache.org>
Authored: Thu Jun 1 15:46:05 2017 -0700
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:42:19 2017 +0200
----------------------------------------------------------------------
docs/dev/table/sql.md | 482 +++++++++++++++++++++++++++++++++++++--------
1 file changed, 395 insertions(+), 87 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/ddae51fb/docs/dev/table/sql.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/sql.md b/docs/dev/table/sql.md
index b79f4f2..26f4f1b 100644
--- a/docs/dev/table/sql.md
+++ b/docs/dev/table/sql.md
@@ -22,19 +22,21 @@ specific language governing permissions and limitations
under the License.
-->
-SQL queries are specified using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table` which can be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries, or written to a `TableSink` (see [Writing Tables to External Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single DataStream or DataSet program.
+SQL queries are specified with the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table`. A `Table` can be used in [subsequent SQL and Table API queries](common.html#mixing-table-api-and-sql), be [converted into a DataSet or DataStream](common.html#integration-with-datastream-and-dataset-api), or [written to a TableSink](common.html#emit-a-table)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single program.
-A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be registered in the `TableEnvironment` in order to be accessible by a SQL query (see [Registering Tables](#registering-tables)). For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).
+In order to access a table in a SQL query, it must be [registered in the TableEnvironment](common.html#register-a-table-in-the-catalog). A table can be registered from a [TableSource](common.html#register-a-tablesource), [Table](common.html#register-a-table), [DataStream, or DataSet](common.html#register-a-datastream-or-dataset-as-table). Alternatively, users can also [register external catalogs in a TableEnvironment](common.html#register-an-external-catalog) to specify the location of the data sources.
-*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*
+For convenience `Table.toString()` automatically registers the table under a unique name in its `TableEnvironment` and returns the name. Hence, `Table` objects can be directly inlined into SQL queries (by string concatenation) as shown in the examples below.
-**TODO: Rework intro. Move some parts below. **
+**Note:** Flink's SQL support is not yet feature complete. Queries that include unsupported SQL features cause a `TableException`. The supported features of SQL on batch and streaming tables are listed in the following sections.
* This will be replaced by the TOC
{:toc}
Specifying a Query
----------------
+------------------
+
+The following examples show how to specify a SQL queries on registered and inlined tables.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
@@ -45,12 +47,12 @@ StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
-// call SQL on unregistered tables
+// SQL query with an inlined (unregistered) table
Table table = tableEnv.toTable(ds, "user, product, amount");
Table result = tableEnv.sql(
"SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
-// call SQL on registered tables
+// SQL query with a registered table
// register the DataStream as table "Orders"
tableEnv.registerDataStream("Orders", ds, "user, product, amount");
// run a SQL query on the Table and retrieve the result as a new Table
@@ -67,12 +69,12 @@ val tableEnv = TableEnvironment.getTableEnvironment(env)
// read a DataStream from an external source
val ds: DataStream[(Long, String, Integer)] = env.addSource(...)
-// call SQL on unregistered tables
+// SQL query with an inlined (unregistered) table
val table = ds.toTable(tableEnv, 'user, 'product, 'amount)
val result = tableEnv.sql(
s"SELECT SUM(amount) FROM $table WHERE product LIKE '%Rubber%'")
-// call SQL on registered tables
+// SQL query with a registered table
// register the DataStream under the name "Orders"
tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)
// run a SQL query on the Table and retrieve the result as a new Table
@@ -82,14 +84,14 @@ val result2 = tableEnv.sql(
</div>
</div>
-**TODO: Add some intro.**
-
{% top %}
Supported Syntax
----------------
-Flink uses [Apache Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. Currently, Flink SQL only supports query-related SQL syntax and only a subset of the comprehensive SQL standard. The following BNF-grammar describes the supported SQL features:
+Flink parses SQL using [Apache Calcite](https://calcite.apache.org/docs/reference.html), which supports standard ANSI SQL. DML and DDL statements are not supported by Flink.
+
+The following BNF-grammar describes the superset of supported SQL features in batch and streaming queries. The [Operations](#operations) section shows examples for the supported features and indicates which features are only supported for batch or streaming queries.
```
@@ -153,9 +155,10 @@ groupItem:
| CUBE '(' expression [, expression ]* ')'
| ROLLUP '(' expression [, expression ]* ')'
| GROUPING SETS '(' groupItem [, groupItem ]* ')'
+
```
-For a better definition of SQL queries within a Java String, Flink SQL uses a lexical policy similar to Java:
+Flink SQL uses a lexical policy for identifier (table, attribute, function names) similar to Java:
- The case of identifiers is preserved whether or not they are quoted.
- After which, identifiers are matched case-sensitively.
@@ -163,31 +166,355 @@ For a better definition of SQL queries within a Java String, Flink SQL uses a le
{% top %}
-Example Queries
----------------
-
-**TODO: Add a examples for different operations with similar structure as for the Table API. Add highlighted tags if an operation is not supported by stream / batch.**
-
-* Scan & Values
-* Selection & Projection
-* Aggregations (distinct only Batch)
- * GroupBy
- * GroupBy Windows (TUMBLE, HOP, SESSION)
- * OVER windows (Only Stream)
- * Grouping sets, rollup, cube (only batch)
- * Having (only batch?)
-* Joins
- * Inner equi joins (only batch)
- * Outer equi joins (only batch)
- * TableFunction
-* Set operations (only batch, except Union ALL)
-* OrderBy + Limit + Offset
+Operations
+--------------------
+
+### Scan, Projection, and Filter
+
+<div markdown="1">
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operation</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>
+ <strong>Scan / Select / As</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT * FROM Orders
+
+SELECT a, c AS d FROM Orders
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>Where / Filter</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT * FROM Orders WHERE b = 'red'
+
+SELECT * FROM Orders WHERE a % 2 = 0
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>User-defined Scalar Functions (Scalar UDF)</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>UDFs must be registered in the TableEnvironment. See the <a href="udfs.html">UDF documentation</a> for details on how to specify and register scalar UDFs.</p>
+{% highlight sql %}
+SELECT PRETTY_PRINT(user) FROM Orders
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+</div>
+
+{% top %}
+
+### Aggregations
+
+<div markdown="1">
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operation</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>
+ <strong>GroupBy Aggregation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span><br>
+ <span class="label label-info">Result Updating</span>
+ </td>
+ <td>
+ <p><b>Note:</b> GroupBy on a streaming table produces an updating result. See the <a href="streaming.html">Streaming Concepts</a> page for details.
+ </p>
+{% highlight sql %}
+SELECT a, SUM(b) as d
+FROM Orders
+GROUP BY a
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>GroupBy Window Aggregation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Use a group window to compute a single result row per group. See <a href="#group-windows">Group Windows</a> section for more details.</p>
+{% highlight sql %}
+SELECT user, SUM(amount)
+FROM Orders
+GROUP BY TUMBLE(rowtime, INTERVAL '1' DAY), user
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>Over Window aggregation</strong><br>
+ <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p><b>Note:</b> All aggregates must be defined over the same window, i.e., same partitioning, sorting, and range. Currently, only windows with PRECEDING (UNBOUNDED and bounded) to CURRENT ROW range are supported. Ranges with FOLLOWING are not supported yet. ORDER BY must be specified on a single <a href="streaming.html#time-attributes">time attribute</a></p>
+{% highlight sql %}
+SELECT COUNT(amount) OVER (
+ PARTITION BY user
+ ORDER BY proctime
+ ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)
+FROM Orders
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>Distinct</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT DISTINCT users FROM Orders
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>Grouping sets, Rollup, Cube</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT SUM(amount)
+FROM Orders
+GROUP BY GROUPING SETS ((user), (product))
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>Having</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT SUM(amount)
+FROM Orders
+GROUP BY users
+HAVING SUM(amount) > 50
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>User-defined Aggregate Functions (UDAGG)</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>UDAGGs must be registered in the TableEnvironment. See the <a href="udfs.html">UDF documentation</a> for details on how to specify and register UDAGGs.</p>
+{% highlight sql %}
+SELECT MyAggregate(amount)
+FROM Orders
+GROUP BY users
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+</div>
+
+{% top %}
+
+### Joins
+
+<div markdown="1">
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operation</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Inner Equi-join / Outer Equi-join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+ <p>Currently, only equi-joins are supported, i.e., joins that have at least one conjunctive condition with an equality predicate. Arbitrary cross or theta joins are not supported.</p>
+ <p><b>Note:</b> The order of joins is not optimized. Tables are joined in the order in which they are specified in the FROM clause. Make sure to specify tables in an order that does not yield a cross join (Cartesian product) which are not supported and would cause a query to fail.</p>
+{% highlight sql %}
+SELECT *
+FROM Orders INNER JOIN Product ON Orders.productId = Product.id
+
+SELECT *
+FROM Orders LEFT JOIN Product ON Orders.productId = Product.id
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>Expanding arrays into a relation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Unnesting WITH ORDINALITY is not supported yet.</p>
+{% highlight sql %}
+SELECT users, tag
+FROM Orders CROSS JOIN UNNEST(tags) AS t (tag)
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>User Defined Table Functions (UDTF)</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>UDTFs must be registered in the TableEnvironment. See the <a href="udfs.html">UDF documentation</a> for details on how to specify and register UDTFs. </p>
+{% highlight sql %}
+SELECT users, tag
+FROM Orders LATERAL VIEW UNNEST_UDTF(tags) t AS tag
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+</div>
+
+{% top %}
+
+### Set Operations
+
+<div markdown="1">
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operation</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>
+ <strong>Union</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT *
+FROM (
+ (SELECT user FROM Orders WHERE a % 2 = 0)
+ UNION
+ (SELECT user FROM Orders WHERE b = 0)
+)
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td>
+ <strong>UnionAll</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT *
+FROM (
+ (SELECT user FROM Orders WHERE a % 2 = 0)
+ UNION ALL
+ (SELECT user FROM Orders WHERE b = 0)
+)
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ <strong>Intersect / Except</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT *
+FROM (
+ (SELECT user FROM Orders WHERE a % 2 = 0)
+ INTERSECT
+ (SELECT user FROM Orders WHERE b = 0)
+)
+{% endhighlight %}
+{% highlight sql %}
+SELECT *
+FROM (
+ (SELECT user FROM Orders WHERE a % 2 = 0)
+ EXCEPT
+ (SELECT user FROM Orders WHERE b = 0)
+)
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+</div>
{% top %}
-### GroupBy Windows
+### OrderBy & Limit
+
+<div markdown="1">
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operation</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>
+ <strong>Order By</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT *
+FROM Orders
+ORDER BY users
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Limit</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
+ <td>
+{% highlight sql %}
+SELECT *
+FROM Orders
+LIMIT 3
+{% endhighlight %}
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+</div>
-**TODO: Integrate this with the examples**
+{% top %}
### Group Windows
@@ -217,7 +544,12 @@ Group windows are defined in the `GROUP BY` clause of a SQL query. Just like que
</tbody>
</table>
-For SQL queries on streaming tables, the `time_attr` argument of the group window function must be one of the `rowtime()` or `proctime()` time-indicators, which distinguish between event or processing time, respectively. For SQL on batch tables, the `time_attr` argument of the group window function must be an attribute of type `TIMESTAMP`.
+
+#### Time Attributes
+
+For SQL queries on streaming tables, the `time_attr` argument of the group window function must refer to a valid time attribute that specifies the processing time or event time of rows. See the [documentation of time attributes](streaming.html#time-attributes) to learn how to define time attributes.
+
+For SQL on batch tables, the `time_attr` argument of the group window function must be an attribute of type `TIMESTAMP`.
#### Selecting Group Window Start and End Timestamps
@@ -251,9 +583,9 @@ The start and end timestamps of group windows can be selected with the following
</tbody>
</table>
-Note that the auxiliary functions must be called with exactly same arguments as the group window function in the `GROUP BY` clause.
+*Note:* Auxiliary functions must be called with exactly same arguments as the group window function in the `GROUP BY` clause.
-The following examples show how to specify SQL queries with group windows on streaming tables.
+The following examples show how to specify SQL queries with group windows on streaming tables.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
@@ -264,31 +596,31 @@ StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
// ingest a DataStream from an external source
DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
// register the DataStream as table "Orders"
-tableEnv.registerDataStream("Orders", ds, "user, product, amount");
+tableEnv.registerDataStream("Orders", ds, "user, product, amount, proctime.proctime, rowtime.rowtime");
// compute SUM(amount) per day (in event-time)
Table result1 = tableEnv.sql(
"SELECT user, " +
- " TUMBLE_START(rowtime(), INTERVAL '1' DAY) as wStart, " +
- " SUM(amount) FROM Orders " +
- "GROUP BY TUMBLE(rowtime(), INTERVAL '1' DAY), user");
+ " TUMBLE_START(rowtime, INTERVAL '1' DAY) as wStart, " +
+ " SUM(amount) FROM Orders " +
+ "GROUP BY TUMBLE(rowtime, INTERVAL '1' DAY), user");
// compute SUM(amount) per day (in processing-time)
Table result2 = tableEnv.sql(
- "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime(), INTERVAL '1' DAY), user");
+ "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime, INTERVAL '1' DAY), user");
// compute every hour the SUM(amount) of the last 24 hours in event-time
Table result3 = tableEnv.sql(
- "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime(), INTERVAL '1' HOUR, INTERVAL '1' DAY), product");
+ "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime, INTERVAL '1' HOUR, INTERVAL '1' DAY), product");
// compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
Table result4 = tableEnv.sql(
"SELECT user, " +
- " SESSION_START(rowtime(), INTERVAL '12' HOUR) AS sStart, " +
- " SESSION_END(rowtime(), INTERVAL '12' HOUR) AS snd, " +
- " SUM(amount) " +
- "FROM Orders " +
- "GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user");
+ " SESSION_START(rowtime, INTERVAL '12' HOUR) AS sStart, " +
+ " SESSION_END(rowtime, INTERVAL '12' HOUR) AS snd, " +
+ " SUM(amount) " +
+ "FROM Orders " +
+ "GROUP BY SESSION(rowtime, INTERVAL '12' HOUR), user");
{% endhighlight %}
</div>
@@ -301,34 +633,34 @@ val tableEnv = TableEnvironment.getTableEnvironment(env)
// read a DataStream from an external source
val ds: DataStream[(Long, String, Int)] = env.addSource(...)
// register the DataStream under the name "Orders"
-tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)
+tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount, 'proctime.proctime, 'rowtime.rowtime)
// compute SUM(amount) per day (in event-time)
val result1 = tableEnv.sql(
"""
|SELECT
- | user,
- | TUMBLE_START(rowtime(), INTERVAL '1' DAY) as wStart,
+ | user,
+ | TUMBLE_START(rowtime, INTERVAL '1' DAY) as wStart,
| SUM(amount)
| FROM Orders
- | GROUP BY TUMBLE(rowtime(), INTERVAL '1' DAY), user
+ | GROUP BY TUMBLE(rowtime, INTERVAL '1' DAY), user
""".stripMargin)
// compute SUM(amount) per day (in processing-time)
val result2 = tableEnv.sql(
- "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime(), INTERVAL '1' DAY), user")
+ "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime, INTERVAL '1' DAY), user")
// compute every hour the SUM(amount) of the last 24 hours in event-time
val result3 = tableEnv.sql(
- "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime(), INTERVAL '1' HOUR, INTERVAL '1' DAY), product")
+ "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime, INTERVAL '1' HOUR, INTERVAL '1' DAY), product")
// compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
val result4 = tableEnv.sql(
"""
|SELECT
- | user,
- | SESSION_START(rowtime(), INTERVAL '12' HOUR) AS sStart,
- | SESSION_END(rowtime(), INTERVAL '12' HOUR) AS sEnd,
+ | user,
+ | SESSION_START(rowtime, INTERVAL '12' HOUR) AS sStart,
+ | SESSION_END(rowtime, INTERVAL '12' HOUR) AS sEnd,
| SUM(amount)
| FROM Orders
| GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user
@@ -340,28 +672,6 @@ val result4 = tableEnv.sql(
{% top %}
-### Limitations
-
-**TODO: Integrate this with the examples**
-
-#### Batch
-
-The current version supports selection (filter), projection, inner equi-joins, grouping, aggregates, and sorting on batch tables.
-
-Among others, the following SQL features are not supported, yet:
-
-- Timestamps and intervals are limited to milliseconds precision
-- Interval arithmetic is currenly limited
-- Non-equi joins and Cartesian products
-- Efficient grouping sets
-
-*Note: Tables are joined in the order in which they are specified in the `FROM` clause. In some cases the table order must be manually tweaked to resolve Cartesian products.*
-
-#### Streaming
-
-Joins, set operations, and non-windowed aggregations are not supported yet.
-`UNNEST` supports only arrays and does not support `WITH ORDINALITY` yet.
-
Data Types
----------
@@ -388,7 +698,7 @@ The SQL runtime is built on top of Flink's DataSet and DataStream APIs. Internal
| `Types.MAP` | `MAP` | `java.util.HashMap` |
-Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and array types (object or primitive arrays) can be fields of a row.
+Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and array types (object or primitive arrays) can be fields of a row.
Generic types are treated as a black box within Table API and SQL yet.
@@ -799,7 +1109,7 @@ boolean IS NOT UNKNOWN
<p>Returns negative <i>numeric</i>.</p>
</td>
</tr>
-
+
<tr>
<td>
{% highlight text %}
@@ -1562,7 +1872,7 @@ AVG(numeric)
<p>Returns the average (arithmetic mean) of <i>numeric</i> across all input values.</p>
</td>
</tr>
-
+
<tr>
<td>
{% highlight text %}
@@ -1605,7 +1915,7 @@ STDDEV_POP(value)
<p>Returns the population standard deviation of the numeric field across all input values.</p>
</td>
</tr>
-
+
<tr>
<td>
{% highlight text %}
@@ -1751,14 +2061,13 @@ ELEMENT(ARRAY)
</tbody>
</table>
-### Limitations
+### Unsupported Functions
-The following operations are not supported yet:
+The following functions are not supported yet:
- Binary string operators and functions
- System functions
- Collection functions
-- Aggregate functions like STDDEV_xxx, VAR_xxx, and REGR_xxx
- Distinct aggregate functions like COUNT DISTINCT
{% top %}
@@ -1775,4 +2084,3 @@ A, ABS, ABSOLUTE, ACTION, ADA, ADD, ADMIN, AFTER, ALL, ALLOCATE, ALLOW, ALTER, A
{% endhighlight %}
{% top %}
-
[10/10] flink git commit: [FLINK-6746] [table] [docs] Updated Table
API / SQL docs: Common API
Posted by fh...@apache.org.
[FLINK-6746] [table] [docs] Updated Table API / SQL docs: Common API
This closes #4012.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fffce090
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fffce090
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fffce090
Branch: refs/heads/master
Commit: fffce0904521d3e600799ba846d73d15757b24b1
Parents: 4088409
Author: Fabian Hueske <fh...@apache.org>
Authored: Sun May 28 15:35:09 2017 +0200
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:42:19 2017 +0200
----------------------------------------------------------------------
docs/dev/table/common.md | 800 +++++++++++++++++++++++++++++++-----------
1 file changed, 593 insertions(+), 207 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/fffce090/docs/dev/table/common.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/common.md b/docs/dev/table/common.md
index d0d4914..fed2b6d 100644
--- a/docs/dev/table/common.md
+++ b/docs/dev/table/common.md
@@ -22,9 +22,7 @@ specific language governing permissions and limitations
under the License.
-->
-The Table API and SQL are integrated API and share many concepts and much of their API.
-
-**TODO: Extend**
+The Table API and SQL are integrated in a joint API. The central concept of this API is a `Table` which serves as input and output of queries. This document shows the common structure of programs with Table API and SQL queries, how to register a `Table`, how to query a `Table`, and how to emit a `Table`.
* This will be replaced by the TOC
{:toc}
@@ -32,440 +30,828 @@ The Table API and SQL are integrated API and share many concepts and much of the
Structure of Table API and SQL Programs
---------------------------------------
-All Table API and SQL programs for batch and streaming have the same structure.
+All Table API and SQL programs for batch and streaming follow the same pattern. The following code example shows the common structure of Table API and SQL programs.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
+// for batch programs use ExecutionEnvironment instead of StreamExecutionEnvironment
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
-// Create a TableEnvironment
+// create a TableEnvironment
+// for batch programs use BatchTableEnvironment instead of StreamTableEnvironment
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-// Register a Table
-tableEnv.registerTable("yourTable", ...) // or
-tableEnv.registerTableSource("yourTableSrc", ...); // or
-tableEnv.registerDataStream("yourTableStream", ...); // or
-tableEnv.registerDataSet("yourTableSet", ...); // or
-tableEnv.registerExternalCatalog("yourCatalog", ...);
+// register a Table
+tableEnv.registerTable("table1", ...) // or
+tableEnv.registerTableSource("table2", ...); // or
+tableEnv.registerExternalCatalog("extCat", ...);
-// Create a table from a Table API query
-Table tapiResult = tableEnv.scan("yourTableSrc").select(...);
-// Or create a table from a SQL query
-Table sqlResult = tableEnv.sql("SELECT ... FROM yourTableSrc ... ");
+// create a Table from a Table API query
+Table tapiResult = tableEnv.scan("table1").select(...);
+// create a Table from a SQL query
+Table sqlResult = tableEnv.sql("SELECT ... FROM table2 ... ");
-// Emit a Table to a TableSink / DataStream / DataSet
-resultTable.writeToSink(...); // or
-resultTable.toAppendStream(...); // or
-resultTable.toRetractStream(...); // or
-resultTable.toDataSet(...);
+// emit a Table API result Table to a TableSink, same for SQL result
+tapiResult.writeToSink(...);
-// Execute
-env.execute("Your Query");
+// execute
+env.execute();
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
+// for batch programs use ExecutionEnvironment instead of StreamExecutionEnvironment
+val env = StreamExecutionEnvironment.getExecutionEnvironment
-// Create a TableEnvironment
+// create a TableEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
-// Register a Table
-tableEnv.registerTable("yourTable", ...) // or
-tableEnv.registerTableSource("yourTableSrc", ...) // or
-tableEnv.registerDataStream("yourTableStream", ...) // or
-tableEnv.registerDataSet("yourTableSet", ...) // or
-tableEnv.registerExternalCatalog("yourCatalog", ...)
+// register a Table
+tableEnv.registerTable("table1", ...) // or
+tableEnv.registerTableSource("table2", ...) // or
+tableEnv.registerExternalCatalog("extCat", ...)
-// Create a table from a Table API query
-val tapiResult = tableEnv.scan("yourTableSrc").select(...)
-// Or create a table from a SQL query
-val sqlResult = tableEnv.sql("SELECT ... FROM yourTableSrc ...")
+// create a Table from a Table API query
+val tapiResult = tableEnv.scan("table1").select(...)
+// Create a Table from a SQL query
+val sqlResult = tableEnv.sql("SELECT ... FROM table2 ...")
-// Emit a Table
-resultTable.writeToSink(...) // or
-resultTable.toAppendStream(...) // or
-resultTable.toRetractStream(...) // or
-resultTable.toDataSet(...)
+// emit a Table API result Table to a TableSink, same for SQL result
+tapiResult.writeToSink(...)
-// Execute
-env.execute("Your Query")
+// execute
+env.execute()
{% endhighlight %}
</div>
</div>
+**Note:** Table API and SQL queries can be easily integrated with and embedded into DataStream or DataSet programs. Have a look at the [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api) section to learn how DataStreams and DataSets can be converted into Tables and vice versa.
+
{% top %}
Create a TableEnvironment
-------------------------
-A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine Tables of different TableEnvironments.
+The `TableEnvironment` is a central concept of the Table API and SQL integration. It is responsible for:
-**TODO: Extend**
+* Registering a `Table` in the internal catalog
+* Registering an external catalog
+* Executing SQL queries
+* Registering a user-defined (scalar, table, or aggregation) function
+* Converting a `DataStream` or `DataSet` into a `Table`
+* Holding a reference to an `ExecutionEnvironment` or `StreamExecutionEnvironment`
+
+A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine tables of different TableEnvironments in the same query, e.g., to join or union them.
+
+A `TableEnvironment` is created by calling the static `TableEnvironment.getTableEnvironment()` method with a `StreamExecutionEnvironment` or an `ExecutionEnvironment` and an optional `TableConfig`. The `TableConfig` can be used to configure the `TableEnvironment` or to customize the query optimization and translation process (see [Query Optimization](#query-optimization)).
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// ***************
+// STREAMING QUERY
+// ***************
+StreamExecutionEnvironment sEnv = StreamExecutionEnvironment.getExecutionEnvironment();
+// create a TableEnvironment for streaming queries
+StreamTableEnvironment sTableEnv = TableEnvironment.getTableEnvironment(sEnv);
+
+// ***********
+// BATCH QUERY
+// ***********
+ExecutionEnvironment bEnv = ExecutionEnvironment.getExecutionEnvironment();
+// create a TableEnvironment for batch queries
+BatchTableEnvironment bTableEnv = TableEnvironment.getTableEnvironment(bEnv);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// ***************
+// STREAMING QUERY
+// ***************
+val sEnv = StreamExecutionEnvironment.getExecutionEnvironment
+// create a TableEnvironment for streaming queries
+val sTableEnv = TableEnvironment.getTableEnvironment(sEnv)
+
+// ***********
+// BATCH QUERY
+// ***********
+val bEnv = ExecutionEnvironment.getExecutionEnvironment
+// create a TableEnvironment for batch queries
+val bTableEnv = TableEnvironment.getTableEnvironment(bEnv)
+{% endhighlight %}
+</div>
+</div>
{% top %}
Register a Table in the Catalog
-------------------------------
-`TableEnvironment`s have an internal table catalog to which tables can be registered with a unique name. After registration, a table can be accessed from the `TableEnvironment` by its name.
+A `TableEnvironment` has an internal catalog of tables, organized by table name. Table API or SQL queries can access tables which are registered in the catalog, by referencing them by name.
+
+A `TableEnvironment` allows you to register a table from various sources:
+
+* an existing `Table` object, usually the result of a Table API or SQL query.
+* a `TableSource`, which accesses external data, such as a file, database, or messaging system.
+* a `DataStream` or `DataSet` from a DataStream or DataSet program.
-*Note: `DataSet`s or `DataStream`s can be directly converted into `Table`s without registering them in the `TableEnvironment`. See [Create a Table from a DataStream or DataSet](#tbd) for details.
+Registering a `DataStream` or `DataSet` as a table is discussed in the [Integration with DataStream and DataSet API](#integration-with-datastream-and-dataset-api) section.
### Register a Table
-A `Table` that originates from a Table API operation or a SQL query is registered in a `TableEnvironment` as follows:
+A `Table` is registered in a `TableEnvironment` as follows:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-// works for StreamExecutionEnvironment identically
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-// convert a DataSet into a Table
-Table custT = tableEnv
- .toTable(custDs, "name, zipcode")
- .where("zipcode = '12345'")
- .select("name");
+// Table is the result of a simple projection query
+Table projTable = tableEnv.scan("X").project(...);
-// register the Table custT as table "custNames"
-tableEnv.registerTable("custNames", custT);
+// register the Table projTable as table "projectedX"
+tableEnv.registerTable("projectedTable", projTable);
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-// works for StreamExecutionEnvironment identically
-val env = ExecutionEnvironment.getExecutionEnvironment
+// get a TableEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
-// convert a DataSet into a Table
-val custT = custDs
- .toTable(tableEnv, 'name, 'zipcode)
- .where('zipcode === "12345")
- .select('name)
+// Table is the result of a simple projection query
+val projTable: Table = tableEnv.scan("X").project(...)
-// register the Table custT as table "custNames"
-tableEnv.registerTable("custNames", custT)
+// register the Table projTable as table "projectedX"
+tableEnv.registerTable("projectedTable", projTable)
{% endhighlight %}
</div>
</div>
-A registered `Table` that originates from a Table API operation or SQL query is treated similarly as a view as known from relational DBMS, i.e., it can be inlined when optimizing the query.
+**Note:** A registered `Table` is treated similarly to a `VIEW` as known from relational database systems, i.e., the query that defines the `Table` is not optimized but will be inlined when another query references the registered `Table`. If multiple queries reference the same registered `Table`, it will be inlined for each referencing query and executed multiple times, i.e., the result of the registered `Table` will *not* be shared.
{% top %}
-### Register a DataSet
+### Register a TableSource
+
+A `TableSource` provides access to external data which is stored in a storage systems such as a database (MySQL, HBase, ...), a file with specific encoding (CSV, Apache \[Parquet, Avro, ORC\], ...), or a messaging system (Apache Kafka, RabbitMQ, ...).
-A `DataSet` is registered as a `Table` in a `BatchTableEnvironment` as follows:
+Flink aims to provide TableSources for common data formats and storage systems. Please have a look at the [Table Sources and Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html) page for a list of supported TableSources and instructions for how to build a custom `TableSource`.
+
+A `TableSource` is registered in a `TableEnvironment` as follows:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-// register the DataSet cust as table "Customers" with fields derived from the dataset
-tableEnv.registerDataSet("Customers", cust);
+// create a TableSource
+TableSource csvSource = new CsvTableSource("/path/to/file", ...);
-// register the DataSet ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataSet("Orders", ord, "user, product, amount");
+// register the TableSource as table "CsvTable"
+tableEnv.registerTableSource("CsvTable", csvSource);
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
+// get a TableEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
-// register the DataSet cust as table "Customers" with fields derived from the dataset
-tableEnv.registerDataSet("Customers", cust)
+// create a TableSource
+val csvSource: TableSource = new CsvTableSource("/path/to/file", ...)
-// register the DataSet ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataSet("Orders", ord, 'user, 'product, 'amount)
+// register the TableSource as table "CsvTable"
+tableEnv.registerTableSource("CsvTable", csvSource)
{% endhighlight %}
</div>
</div>
-*Note: The name of a `DataSet` `Table` must not match the `^_DataSetTable_[0-9]+` pattern which is reserved for internal use only.*
-
{% top %}
-### Register a DataStream
+Register an External Catalog
+----------------------------
+
+An external catalog can provide information about external databases and tables such as their name, schema, statistics, and information for how to access data stored in an external database, table, or file.
-A `DataStream` is registered as a `Table` in a `StreamTableEnvironment` as follows:
+An external catalog can be created by implementing the `ExternalCatalog` interface and is registered in a `TableEnvironment` as follows:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-// register the DataStream cust as table "Customers" with fields derived from the datastream
-tableEnv.registerDataStream("Customers", cust);
+// create an external catalog
+ExternalCatalog catalog = new InMemoryExternalCatalog();
-// register the DataStream ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataStream("Orders", ord, "user, product, amount");
+// register the ExternalCatalog catalog
+tableEnv.registerExternalCatalog("InMemCatalog", catalog);
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val env = StreamExecutionEnvironment.getExecutionEnvironment
+// get a TableEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
-// register the DataStream cust as table "Customers" with fields derived from the datastream
-tableEnv.registerDataStream("Customers", cust)
+// create an external catalog
+val catalog: ExternalCatalog = new InMemoryExternalCatalog
-// register the DataStream ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataStream("Orders", ord, 'user, 'product, 'amount)
+// register the ExternalCatalog catalog
+tableEnv.registerExternalCatalog("InMemCatalog", catalog)
{% endhighlight %}
</div>
</div>
-*Note: The name of a `DataStream` `Table` must not match the `^_DataStreamTable_[0-9]+` pattern which is reserved for internal use only.*
+Once registered in a `TableEnvironment`, all tables defined in a `ExternalCatalog` can be accessed from Table API or SQL queries by specifying their full path, such as `catalog.database.table`.
+
+Currently, Flink provides an `InMemoryExternalCatalog` for demo and testing purposes. However, the `ExternalCatalog` interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.
{% top %}
-### Register a TableSource
+Query a Table
+-------------
-TableSources provided access to data stored in various storage systems such as databases (MySQL, HBase, ...), file formats (CSV, Apache Parquet, Avro, ORC, ...), or messaging systems (Apache Kafka, RabbitMQ, ...). Flink provides a TableSources for common data formats and storage systems. Please have a look at the [Table Sources and Sinks page]({{ site.baseurl }}/dev/table/sourceSinks.html) for a list of provided TableSources and documentation for how to built your own.
+### Table API
-An external table is registered in a `TableEnvironment` using a `TableSource` as follows:
+The Table API is a language-integrated query API for Scala and Java. In contrast to SQL, queries are not specified as Strings but are composed step-by-step in the host language.
+
+The API is based on the `Table` class which represents a table (streaming or batch) and offers methods to apply relational operations. These methods return a new `Table` object, which represents the result of applying the relational operation on the input `Table`. Some relational operations are composed of multiple method calls such as `table.groupBy(...).select()`, where `groupBy(...)` specifies a grouping of `table`, and `select(...)` the projection on the grouping of `table`.
+
+The [Table API]({{ site.baseurl }}/dev/table/tableapi.html) document describes all Table API operations that are supported on streaming and batch tables.
+
+The following example shows a simple Table API aggregation query:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-// works for StreamExecutionEnvironment identically
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// register Orders table
-TableSource custTS = new CsvTableSource("/path/to/file", ...);
+// scan registered Orders table
+Table orders = tableEnv.scan("Orders");
+// compute revenue for all customers from France
+Table revenue = orders
+ .filter("cCountry === 'FRANCE'")
+ .groupBy("cID, cName")
+ .select("cID, cName, revenue.sum AS revSum");
-// register a `TableSource` as external table "Customers"
-tableEnv.registerTableSource("Customers", custTS);
+// emit or convert Table
+// execute query
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-// works for StreamExecutionEnvironment identically
-val env = ExecutionEnvironment.getExecutionEnvironment
+// get a TableEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
-val custTS: TableSource = new CsvTableSource("/path/to/file", ...)
+// register Orders table
-// register a `TableSource` as external table "Customers"
-tableEnv.registerTableSource("Customers", custTS)
+// scan registered Orders table
+Table orders = tableEnv.scan("Orders")
+// compute revenue for all customers from France
+Table revenue = orders
+ .filter('cCountry === "FRANCE")
+ .groupBy('cID, 'cName)
+ .select('cID, 'cName, 'revenue.sum AS 'revSum)
+// emit or convert Table
+// execute query
{% endhighlight %}
+
+**Note:** The Scala Table API uses Scala Symbols, which start with a single tick (`'`) to reference the attributes of a `Table`. The Table API uses Scala implicits. Make sure to import `org.apache.flink.api.scala._` and `org.apache.flink.table.api.scala._` in order to use Scala implicit conversions.
</div>
</div>
-A `TableSource` can provide access to data stored in various storage systems such as databases (MySQL, HBase, ...), file formats (CSV, Apache Parquet, Avro, ORC, ...), or messaging systems (Apache Kafka, RabbitMQ, ...).
-
{% top %}
-Register an External Catalog
-----------------------------
+### SQL
+
+Flink's SQL integration is based on [Apache Calcite](https://calcite.apache.org), which implements the SQL standard. SQL queries are specified as regular Strings.
-An external catalog is defined by the `ExternalCatalog` interface and provides information about databases and tables such as their name, schema, statistics, and access information. An `ExternalCatalog` is registered in a `TableEnvironment` as follows:
+The [SQL]({{ site.baseurl }}/dev/table/sql.html) document describes Flink's SQL support for streaming and batch tables.
+
+The following example shows how to specify a query and return the result as a Table.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-// works for StreamExecutionEnvironment identically
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-ExternalCatalog customerCatalog = new InMemoryExternalCatalog();
+// register Orders table
-// register the ExternalCatalog customerCatalog
-tableEnv.registerExternalCatalog("Customers", customerCatalog);
+// compute revenue for all customers from France
+Table revenue = tableEnv.sql(
+ "SELECT cID, cName, SUM(revenue) AS revSum " +
+ "FROM Orders " +
+ "WHERE cCountry = 'FRANCE' " +
+ "GROUP BY cID, cName"
+ );
+
+// emit or convert Table
+// execute query
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-// works for StreamExecutionEnvironment identically
-val env = ExecutionEnvironment.getExecutionEnvironment
+// get a TableEnvironment
val tableEnv = TableEnvironment.getTableEnvironment(env)
-val customerCatalog: ExternalCatalog = new InMemoryExternalCatalog
+// register Orders table
-// register the ExternalCatalog customerCatalog
-tableEnv.registerExternalCatalog("Customers", customerCatalog)
+// compute revenue for all customers from France
+Table revenue = tableEnv.sql("""
+ |SELECT cID, cName, SUM(revenue) AS revSum
+ |FROM Orders
+ |WHERE cCountry = 'FRANCE'
+ |GROUP BY cID, cName
+ """.stripMargin)
+// emit or convert Table
+// execute query
{% endhighlight %}
+
</div>
</div>
-Once registered in a `TableEnvironment`, all tables defined in a `ExternalCatalog` can be accessed from Table API or SQL queries by specifying their full path (`catalog`.`database`.`table`).
+{% top %}
-Currently, Flink provides an `InMemoryExternalCatalog` for demo and testing purposes. However, the `ExternalCatalog` interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.
+### Mixing Table API and SQL
+
+Table API and SQL queries can be easily mixed because both return `Table` objects:
+
+* A Table API query can be defined on the `Table` object returned by a SQL query.
+* A SQL query can be defined on the result of a Table API query by [registering the resulting Table](#register-a-table) in the `TableEnvironment` and referencing it in the `FROM` clause of the SQL query.
{% top %}
-Create a Table from a DataStream or DataSet
--------------------------------------------
+Emit a Table
+------------
-Besides registering a Table in a catalog, it is also possible to directly create a `Table` from a `DataStream` or `DataSet`.
+In order to emit a `Table`, it can be written to a `TableSink`. A `TableSink` is a generic interface to support a wide variety of file formats (e.g. CSV, Apache Parquet, Apache Avro), storage systems (e.g., JDBC, Apache HBase, Apache Cassandra, Elasticsearch), or messaging systems (e.g., Apache Kafka, RabbitMQ).
-### Create a Table from a DataStream
+A batch `Table` can only be written to a `BatchTableSink`, while a streaming table requires either an `AppendStreamTableSink`, a `RetractStreamTableSink`, or an `UpsertStreamTableSink`.
-**TODO**
+Please see the documentation about [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html) for details about available sinks and instructions for how to implement a custom `TableSink`.
-{% top %}
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-### Create a Table from a DataSet
+// compute a result Table using Table API operators and/or SQL queries
+Table result = ...
+
+// create a TableSink
+TableSink sink = new CsvTableSink("/path/to/file", fieldDelim = "|");
+
+// write the result Table to the TableSink
+result.writeToSink(sink);
+
+// execute the program
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get a TableEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// compute a result Table using Table API operators and/or SQL queries
+val result: Table = ...
-**TODO**
+// create a TableSink
+val sink: TableSink = new CsvTableSink("/path/to/file", fieldDelim = "|")
-### Scala Implicit Conversion
+// write the result Table to the TableSink
+result.writeToSink(sink)
-If you use the Scala API, A `DataSet` or `DataStream` can be implicitly converted into a `Table`.
+// execute the program
+{% endhighlight %}
+</div>
+</div>
{% top %}
-Query a Table
--------------
-### Table API
+Translate and Execute a Query
+-----------------------------
+
+Table API and SQL queries are translated into [DataStream]({{ site.baseurl }}/dev/datastream_api.html) or [DataSet]({{ site.baseurl }}/dev/batch) programs depending on whether their input is a streaming or batch input. A query is internally represented as a logical query plan and is translated in two phases:
+
+1. optimization of the logical plan,
+2. translation into a DataStream or DataSet program.
-**TODO**
+A Table API or SQL query is translated when:
+
+* the `Table` is emitted to a `TableSink`, i.e., when `Table.writeToSink()` is called.
+* the `Table` is converted into a `DataStream` or `DataSet` (see [Integration with DataStream and DataSet API](#integration-with-dataStream-and-dataSet-api)).
+
+Once translated, a Table API or SQL query is handled like a regular DataStream or DataSet program and is executed when `StreamExecutionEnvironment.execute()` or `ExecutionEnvironment.execute()` is called.
{% top %}
-### SQL
+Integration with DataStream and DataSet API
+-------------------------------------------
-**TODO**
+Table API and SQL queries can be easily integrated with and embedded into [DataStream]({{ site.baseurl }}/dev/datastream_api.html) and [DataSet]({{ site.baseurl }}/dev/batch) programs. For instance, it is possible to query an external table (for example from a RDBMS), do some pre-processing, such as filtering, projecting, aggregating, or joining with meta data, and then further process the data with either the DataStream or DataSet API (and any of the libraries built on top of these APIs, such as CEP or Gelly). Inversely, a Table API or SQL query can also be applied on the result of a DataStream or DataSet program.
+
+This interaction can be achieved by converting a `DataStream` or `DataSet` into a `Table` and vice versa. In this section, we describe how these conversions are done.
+
+### Implicit Conversion for Scala
+
+The Scala Table API features implicit conversions for the `DataSet`, `DataStream`, and `Table` classes. These conversions are enabled by importing the package `org.apache.flink.table.api.scala._` in addition to `org.apache.flink.api.scala._` for the Scala DataStream API.
+
+### Register a DataStream or DataSet as Table
+
+A `DataStream` or `DataSet` can be registered in a `TableEnvironment` as a Table. The schema of the resulting table depends on the data type of the registered `DataStream` or `DataSet`. Please check the section about [mapping of data types to table schema](#mapping-of-data-types-to-table-schema) for details.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get StreamTableEnvironment
+// registration of a DataSet in a BatchTableEnvironment is equivalent
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+DataStream<Tuple2<Long, String>> stream = ...
+
+// register the DataStream as Table "myTable" with fields "f0", "f1"
+tableEnv.registerDataStream("myTable", stream);
+
+// register the DataStream as table "myTable2" with fields "myLong", "myString"
+tableEnv.registerDataStream("myTable2", stream, "myLong, myString");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get TableEnvironment
+// registration of a DataSet is equivalent
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+val stream: DataStream[(Long, String)] = ...
+
+// register the DataStream as Table "myTable" with fields "f0", "f1"
+tableEnv.registerDataStream("myTable", stream)
+
+// register the DataStream as table "myTable2" with fields "myLong", "myString"
+tableEnv.registerDataStream("myTable2", stream, 'myLong, 'myString)
+{% endhighlight %}
+</div>
+</div>
+
+**Note:** The name of a `DataStream` `Table` must not match the `^_DataStreamTable_[0-9]+` pattern and the name of a `DataSet` `Table` must not match the `^_DataSetTable_[0-9]+` pattern. These patterns are reserved for internal use only.
{% top %}
-### Interoperability
+### Convert a DataStream or DataSet into a Table
+
+Instead of registering a `DataStream` or `DataSet` in a `TableEnvironment`, it can also be directly converted into a `Table`. This is convenient if you want to use the Table in a Table API query.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get StreamTableEnvironment
+// registration of a DataSet in a BatchTableEnvironment is equivalent
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+DataStream<Tuple2<Long, String>> stream = ...
+
+// Convert the DataStream into a Table with default fields "f0", "f1"
+Table table1 = tableEnv.fromDataStream(stream);
-**TODO**
+// Convert the DataStream into a Table with fields "myLong", "myString"
+Table table2 = tableEnv.fromDataStream(stream, "myLong, myString");
+{% endhighlight %}
+</div>
-* Mix SQL and Table as you like
-* Table API to SQL requires registered tables, register Table
-* SQL to Table API just use resulting table
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get TableEnvironment
+// registration of a DataSet is equivalent
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+val stream: DataStream[(Long, String)] = ...
+
+// convert the DataStream into a Table with default fields '_1, '_2
+val table1: Table = tableEnv.fromDataStream(stream)
+
+// convert the DataStream into a Table with fields 'myLong, 'myString
+val table2: Table = tableEnv.fromDataStream(stream, 'myLong, 'myString)
+{% endhighlight %}
+</div>
+</div>
{% top %}
-Emit a Table
-------------
+### Convert a Table into a DataStream or DataSet
+
+A `Table` can be converted into a `DataStream` or `DataSet`. In this way, custom DataStream or DataSet programs can be run on the result of a Table API or SQL query.
-### Emit to a TableSink
+When converting a `Table` into a `DataStream` or `DataSet`, you need to specify the data type of the resulting `DataStream` or `DataSet`, i.e., the data type into which the rows of the `Table` are to be converted. Often the most convenient conversion type is `Row`. The following list gives an overview of the features of the different options:
-A `Table` can be written to a `TableSink`, which is a generic interface to support a wide variety of file formats (e.g. CSV, Apache Parquet, Apache Avro), storage systems (e.g., JDBC, Apache HBase, Apache Cassandra, Elasticsearch), or messaging systems (e.g., Apache Kafka, RabbitMQ). A batch `Table` can only be written to a `BatchTableSink`, a streaming table requires a `StreamTableSink`. A `TableSink` can implement both interfaces at the same time.
+- **Row**: fields are mapped by position, arbitrary number of fields, support for `null` values, no type-safe access.
+- **POJO**: fields are mapped by name (POJO fields must be named as `Table` fields), arbitrary number of fields, support for `null` values, type-safe access.
+- **Case Class**: fields are mapped by position, no support for `null` values, type-safe access.
+- **Tuple**: fields are mapped by position, limitation to 22 (Scala) or 25 (Java) fields, no support for `null` values, type-safe access.
+- **Atomic Type**: `Table` must have a single field, no support for `null` values, type-safe access.
-Currently, Flink only provides a `CsvTableSink` that writes a batch or streaming `Table` to CSV-formatted files. A custom `TableSink` can be defined by implementing the `BatchTableSink` and/or `StreamTableSink` interface.
+#### Convert a Table into a DataStream
+
+A `Table` that is the result of a streaming query will be updated dynamically, i.e., it is changing as new records arrive on the query's input streams. Hence, the `DataStream` into which such a dynamic query is converted needs to encode the updates of the table.
+
+There are two modes to convert a `Table` into a `DataStream`:
+
+1. **Append Mode**: This mode can only be used if the dynamic `Table` is only modified by `INSERT` changes, i.e, it is append-only and previously emitted results are never updated.
+2. **Retract Mode**: This mode can always be used. It encodes `INSERT` and `DELETE` changes with a `boolean` flag.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+// get StreamTableEnvironment.
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// Table with two fields (String name, Integer age)
+Table table = ...
+
+// convert the Table into an append DataStream of Row by specifying the class
+DataStream<Row> dsRow = tableEnv.toAppendStream(table, Row.class);
+
+// convert the Table into an append DataStream of Tuple2<String, Integer>
+// via a TypeInformation
+TupleTypeInfo<Tuple2<String, Integer>> tupleType = new TupleTypeInfo<>(
+ Types.STRING(),
+ Types.INT());
+DataStream<Tuple2<String, Integer>> dsTuple =
+ tableEnv.toAppendStream(table, tupleType);
+
+// convert the Table into a retract DataStream of Row.
+// A retract stream of type X is a DataStream<Tuple2<Boolean, X>>.
+// The boolean field indicates the type of the change.
+// True is INSERT, false is DELETE.
+DataStream<Tuple2<Boolean, Row>> retractStream =
+ tableEnv.toRetractStream(table, Row.class);
+
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get TableEnvironment.
+// registration of a DataSet is equivalent
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// Table with two fields (String name, Integer age)
+val table: Table = ...
+
+// convert the Table into an append DataStream of Row
+val dsRow: DataStream[Row] = tableEnv.toAppendStream[Row](table)
+
+// convert the Table into an append DataStream of Tuple2[String, Int]
+val dsTuple: DataStream[(String, Int)] dsTuple =
+ tableEnv.toAppendStream[(String, Int)](table)
+
+// convert the Table into a retract DataStream of Row.
+// A retract stream of type X is a DataStream[(Boolean, X)].
+// The boolean field indicates the type of the change.
+// True is INSERT, false is DELETE.
+val retractStream: DataStream[(Boolean, Row)] = tableEnv.toRetractStream[Row](table)
+{% endhighlight %}
+</div>
+</div>
+
+**Note:** A detailed discussion about dynamic tables and their properties is given in the [Streaming Queries]({{ site.baseurl }}/dev/table/streaming.html) document.
+
+#### Convert a Table into a DataSet
+
+A `Table` is converted into a `DataSet` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get BatchTableEnvironment
BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-// compute the result Table using Table API operators and/or SQL queries
-Table result = ...
+// Table with two fields (String name, Integer age)
+Table table = ...
-// create a TableSink
-TableSink sink = new CsvTableSink("/path/to/file", fieldDelim = "|");
-// write the result Table to the TableSink
-result.writeToSink(sink);
+// convert the Table into a DataSet of Row by specifying a class
+DataSet<Row> dsRow = tableEnv.toDataSet(table, Row.class);
-// execute the program
-env.execute();
+// convert the Table into a DataSet of Tuple2<String, Integer> via a TypeInformation
+TupleTypeInfo<Tuple2<String, Integer>> tupleType = new TupleTypeInfo<>(
+ Types.STRING(),
+ Types.INT());
+DataStream<Tuple2<String, Integer>> dsTuple =
+ tableEnv.toAppendStream(table, tupleType);
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-val env = ExecutionEnvironment.getExecutionEnvironment
+// get TableEnvironment
+// registration of a DataSet is equivalent
val tableEnv = TableEnvironment.getTableEnvironment(env)
-// compute the result Table using Table API operators and/or SQL queries
-val result: Table = ...
+// Table with two fields (String name, Integer age)
+val table: Table = ...
-// create a TableSink
-val sink: TableSink = new CsvTableSink("/path/to/file", fieldDelim = "|")
-// write the result Table to the TableSink
-result.writeToSink(sink)
+// convert the Table into a DataSet of Row
+val dsRow: DataSet[Row] = tableEnv.toDataSet[Row](table)
-// execute the program
-env.execute()
+// convert the Table into a DataSet of Tuple2[String, Int]
+val dsTuple: DataSet[(String, Int)] = tableEnv.toDataSet[(String, Int)](table)
{% endhighlight %}
</div>
</div>
{% top %}
-### Convert to a DataStream
+### Mapping of Data Types to Table Schema
-**TODO**
+Flink's DataStream and DataSet APIs support very diverse types, such as Tuples (built-in Scala and Flink Java tuples), POJOs, case classes, and atomic types. In the following we describe how the Table API converts these types into an internal row representation and show examples of converting a `DataStream` into a `Table`.
-{% top %}
+#### Atomic Types
-### Convert to a DataSet
+Flink treats primitives (`Integer`, `Double`, `String`) or generic types (types that cannot be analyzed and decomposed) as atomic types. A `DataStream` or `DataSet` of an atomic type is converted into a `Table` with a single attribute. The type of the attribute is inferred from the atomic type and the name of the attribute must be specified.
-**TODO**
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-### Scala Implicit Conversion
+DataStream<Long> stream = ...
+// convert DataStream into Table with field "myLong"
+Table table = tableEnv.fromDataStream(stream, "myLong");
+{% endhighlight %}
+</div>
-If you use the Scala API, A `Table` can be implicitly converted into a `DataSet` or `DataStream`.
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get a TableEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
-{% top %}
+val stream: DataStream[Long] = ...
+// convert DataStream into Table with field 'myLong
+val table: Table = tableEnv.fromDataStream(stream, 'myLong)
+{% endhighlight %}
+</div>
+</div>
-Execute a Query
----------------
+#### Tuples (Scala and Java) and Case Classes (Scala only)
-**TODO**
+Flink supports Scala's built-in tuples and provides its own tuple classes for Java. DataStreams and DataSets of both kinds of tuples can be converted into tables. Fields can be renamed by providing names for all fields (mapping based on position). If no field names are specified, the default field names are used.
-{% top %}
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-Mappings Types to Table Schema
-------------------------------
+DataStream<Tuple2<Long, String>> stream = ...
-* Explain how types are mapped to table schema
- * Atomic Types
- * Row
- * Tuples (Java / Scala)
- * Pojos
- * Case Classes
+// convert DataStream into Table with field names "myLong", "myString"
+Table table1 = tableEnv.fromDataStream(stream, "myLong, myString");
-**TODO**
+// convert DataStream into Table with default field names "f0", "f1"
+Table table2 = tableEnv.fromDataStream(stream);
+{% endhighlight %}
+</div>
-{% top %}
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get a TableEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
-Integration with DataSet and DataStream API
--------------------------------------------
+val stream: DataStream[(Long, String)] = ...
+
+// convert DataStream into Table with field names 'myLong, 'myString
+val table1: Table = tableEnv.fromDataStream(stream, 'myLong, 'myString)
+
+// convert DataStream into Table with default field names '_1, '_2
+val table2: Table = tableEnv.fromDataStream(stream)
+
+// define case class
+case class Person(name: String, age: Int)
+val streamCC: DataStream[Person] = ...
+
+// convert DataStream into Table with default field names 'name, 'age
+val tableCC1 = tableEnv.fromDataStream(streamCC)
+
+// convert DataStream into Table with field names 'myName, 'myAge
+val tableCC1 = tableEnv.fromDataStream(streamCC, 'myName, 'myAge)
+
+{% endhighlight %}
+</div>
+</div>
+
+#### POJO (Java and Scala)
+
+Flink supports POJOs as composite types. The rules for what determines a POJO are documented [here]({{ site.baseurl }}/dev/api_concepts.html#pojos).
+
+When converting a POJO `DataStream` or `DataSet` into a `Table` without specifying field names, the names of the original POJO fields are used. Renaming the original POJO fields requires the keyword `AS` because POJO fields have no inherent order. The name mapping requires the original names and cannot be done by positions.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// Person is a POJO with fields "name" and "age"
+DataStream<Person> stream = ...
+
+// convert DataStream into Table with field names "name", "age"
+Table table1 = tableEnv.fromDataStream(stream);
+
+// convert DataStream into Table with field names "myName", "myAge"
+Table table2 = tableEnv.fromDataStream(stream, "name as myName, age as myAge");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get a TableEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// Person is a POJO with field names "name" and "age"
+val stream: DataStream[Person] = ...
+
+// convert DataStream into Table with field names 'name, 'age
+val table1: Table = tableEnv.fromDataStream(stream)
+
+// convert DataStream into Table with field names 'myName, 'myAge
+val table2: Table = tableEnv.fromDataStream(stream, 'name as 'myName, 'age as 'myAge)
+{% endhighlight %}
+</div>
+</div>
+
+#### Row
+
+The Row data type supports an arbitrary number of fields and fields with `null` values. Field names can be specified via a `RowTypeInfo` or when converting a `Row` `DataStream` or `DataSet` into a `Table` (based on position).
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// get a StreamTableEnvironment, works for BatchTableEnvironment equivalently
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// DataStream of Row with two fields "name" and "age" specified in `RowTypeInfo`
+DataStream<Row> stream = ...
-**TODO**
+// convert DataStream into Table with field names "name", "age"
+Table table1 = tableEnv.fromDataStream(stream);
-* Create `Table` from `DataSet` and `DataStream` and back
-* Easy integration with more expressive APIs and libraries
- * CEP / Gelly / ML
- * Ingestion and projection
+// convert DataStream into Table with field names "myName", "myAge"
+Table table2 = tableEnv.fromDataStream(stream, "myName, myAge");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// get a TableEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// DataStream of Row with two fields "name" and "age" specified in `RowTypeInfo`
+val stream: DataStream[Row] = ...
+
+// convert DataStream into Table with field names 'name, 'age
+val table1: Table = tableEnv.fromDataStream(stream)
+
+// convert DataStream into Table with field names 'myName, 'myAge
+val table2: Table = tableEnv.fromDataStream(stream, 'myName, 'myAge)
+{% endhighlight %}
+</div>
+</div>
{% top %}
+
Query Optimization
------------------
-* No join order yet
-* Filter / Projection push down
-* Custom rules
+Apache Flink leverages Apache Calcite to optimize and translate queries. The optimization currently performed include projection and filter push-down, subquery decorrelation, and other kinds of query rewriting. Flink does not yet optimize the order of joins, but executes them in the same order as defined in the query (order of Tables in the `FROM` clause and/or order of join predicates in the `WHERE` clause).
+
+It is possible to tweak the set of optimization rules which are applied in different phases by providing a `CalciteConfig` object. This can be created via a builder by calling `CalciteConfig.createBuilder())` and is provided to the TableEnvironment by calling `tableEnv.getConfig.setCalciteConfig(calciteConfig)`.
### Explaining a Table
The Table API provides a mechanism to explain the logical and optimized query plans to compute a `Table`.
-This is done through the `TableEnvironment#explain(table)` method. It returns a string describing three plans:
+This is done through the `TableEnvironment.explain(table)` method. It returns a String describing three plans:
1. the Abstract Syntax Tree of the relational query, i.e., the unoptimized logical query plan,
2. the optimized logical query plan, and
@@ -485,8 +871,8 @@ DataStream<Tuple2<Integer, String>> stream2 = env.fromElements(new Tuple2<>(1, "
Table table1 = tEnv.fromDataStream(stream1, "count, word");
Table table2 = tEnv.fromDataStream(stream2, "count, word");
Table table = table1
- .where("LIKE(word, 'F%')")
- .unionAll(table2);
+ .where("LIKE(word, 'F%')")
+ .unionAll(table2);
String explanation = tEnv.explain(table);
System.out.println(explanation);
@@ -501,8 +887,8 @@ val tEnv = TableEnvironment.getTableEnvironment(env)
val table1 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word)
val table2 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word)
val table = table1
- .where('word.like("F%"))
- .unionAll(table2)
+ .where('word.like("F%"))
+ .unionAll(table2)
val explanation: String = tEnv.explain(table)
println(explanation)
[07/10] flink git commit: [FLINK-6750] [table] [docs] Rework Table
Sources & Sinks Page
Posted by fh...@apache.org.
[FLINK-6750] [table] [docs] Rework Table Sources & Sinks Page
This closes #4094.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/d8756553
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/d8756553
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/d8756553
Branch: refs/heads/master
Commit: d8756553ce490023a017e5927d30c9f178b858d8
Parents: 2324815
Author: twalthr <tw...@apache.org>
Authored: Fri Jun 9 08:25:54 2017 +0200
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:42:19 2017 +0200
----------------------------------------------------------------------
docs/dev/table/sourceSinks.md | 364 +++++++++++++++++--
docs/dev/table/streaming.md | 8 +-
.../flink/table/sources/TableSource.scala | 2 +-
3 files changed, 335 insertions(+), 39 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/d8756553/docs/dev/table/sourceSinks.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/sourceSinks.md b/docs/dev/table/sourceSinks.md
index 2d07254..266ab3b 100644
--- a/docs/dev/table/sourceSinks.md
+++ b/docs/dev/table/sourceSinks.md
@@ -22,6 +22,12 @@ specific language governing permissions and limitations
under the License.
-->
+A `TableSource` provides access to data which is stored in external systems (database, key-value store, message queue) or files. After a [TableSource is registered in a TableEnvironment](common.html#register-a-tablesource) it can accessed by [Table API](tableApi.html) or [SQL](sql.html) queries.
+
+A TableSink [emits a Table](common.html#emit-a-table) to an external storage system, such as a database, key-value store, message queue, or file system (in different encodings, e.g., CSV, Parquet, or ORC).
+
+Have a look at the [common concepts and API](common.html) page for details how to [register a TableSource](common.html#register-a-tablesource) and how to [emit a Table through a TableSink](common.html#emit-a-table).
+
* This will be replaced by the TOC
{:toc}
@@ -30,8 +36,8 @@ Provided TableSources
**TODO: extend and complete**
-Currently, Flink provides the `CsvTableSource` to read CSV files and various `TableSources` to read JSON or Avro objects from Kafka.
-A custom `TableSource` can be defined by implementing the `BatchTableSource` or `StreamTableSource` interface.
+Currently, Flink provides the `CsvTableSource` to read CSV files and a few table sources to read JSON or Avro data from Kafka.
+A custom `TableSource` can be defined by implementing the `BatchTableSource` or `StreamTableSource` interface. See section on [defining a custom TableSource](#define-a-tablesource) for details.
| **Class name** | **Maven dependency** | **Batch?** | **Streaming?** | **Description**
| `CsvTableSouce` | `flink-table` | Y | Y | A simple source for CSV files.
@@ -94,13 +100,6 @@ By default, a missing JSON field does not fail the source. You can configure thi
tableSource.setFailOnMissingField(true);
```
-You can work with the Table as explained in the rest of the Table API guide:
-
-```java
-tableEnvironment.registerTableSource("kafka-source", kafkaTableSource);
-Table result = tableEnvironment.scan("kafka-source");
-```
-
{% top %}
### KafkaAvroTableSource
@@ -114,6 +113,7 @@ To use the Kafka Avro source, you have to add the Kafka connector dependency to
- `flink-connector-kafka-0.10` for Kafka 0.10, respectively.
You can then create the source as follows (example for Kafka 0.8):
+
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
@@ -195,92 +195,386 @@ val csvTableSource = CsvTableSource
</div>
</div>
-You can work with the Table as explained in the rest of the Table API guide in both stream and batch `TableEnvironment`s:
+{% top %}
+
+Provided TableSinks
+-------------------
+
+**TODO**
+
+{% top %}
+
+Define a TableSource
+--------------------
+
+A `TableSource` is a generic interface to access to data stored in an external system as a table. It produces a `DataSet` or `DataStream` and provides the type information to derive the schema of the generated table. There are different table sources for batch tables and streaming tables.
+
+Schema information consists of a data type, field names, and corresponding indexes of these names in the data type.
+
+The general interface looks as follows:
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
{% highlight java %}
-tableEnvironment.registerTableSource("mycsv", csvTableSource);
+TableSource<T> {
-Table streamTable = streamTableEnvironment.scan("mycsv");
+ public TypeInformation<T> getReturnType();
-Table batchTable = batchTableEnvironment.scan("mycsv");
+ public String explainSource();
+}
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
{% highlight scala %}
-tableEnvironment.registerTableSource("mycsv", csvTableSource)
+TableSource[T] {
+
+ def getReturnType: TypeInformation[T]
-val streamTable = streamTableEnvironment.scan("mycsv")
+ def explainSource: String
-val batchTable = batchTableEnvironment.scan("mycsv")
+}
{% endhighlight %}
</div>
</div>
-{% top %}
+To define a `TableSource` one needs to implement `TableSource#getReturnType`. In this case field names and field indexes are derived from the returned type.
-Provided TableSinks
--------------------
+If the `TypeInformation` returned by `getReturnType` does not allow to specify custom field names, it is possible to implement the `DefinedFieldNames` interface in addition.
-**TODO**
+### BatchTableSource
-{% top %}
+Defines an external `TableSource` to create a batch table and provides access to its data.
-Define a TableSource
---------------------
+The interface looks as follows:
-### BatchTableSource
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+BatchTableSource<T> extends TableSource<T> {
-**TODO**
+ public DataSet<T> getDataSet(ExecutionEnvironment execEnv);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+BatchTableSource[T] extends TableSource[T] {
+
+ def getDataSet(execEnv: ExecutionEnvironment): DataSet[T]
+}
+{% endhighlight %}
+</div>
+</div>
{% top %}
### StreamTableSource
-* TimestampAssigner
-* DefinedRowtimeAttribute / DefinedProctimeAttribute
-**TODO**
+Defines an external `TableSource` to create a streaming table and provides access to its data.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamTableSource<T> extends TableSource<T> {
+
+ public DataSet<T> getDataStream(StreamExecutionEnvironment execEnv);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+StreamTableSource[T] extends TableSource[T] {
+
+ def getDataStream(execEnv: StreamExecutionEnvironment): DataSet[T]
+}
+{% endhighlight %}
+</div>
+</div>
+
+**Note:** If a Table needs to be processed in event-time, the `DataStream` returned by the `getDataStream()` method must carry timestamps and watermarks. Please see the documentation on [timestamp and watermark assignment]({{ site.baseurl }}/dev/event_timestamps_watermarks.html) for details on how to assign timestamps and watermarks.
+
+**Note:** Time-based operations on streaming tables such as windows in both the [Table API](tableApi.html#group-windows) and [SQL](sql.html#group-windows) require explicitly specified time attributes.
+
+- `DefinedRowtimeAttribute` provides the `getRowtimeAttribute()` method to specify the name of the event-time time attribute.
+- `DefinedProctimeAttribute` provides the `getProctimeAttribute()` method to specify the name of the processing-time time attribute.
+
+Please see the documentation on [time attributes]({{ site.baseurl }}/dev/table/streaming.html#time-attributes) for details.
{% top %}
### ProjectableTableSource
-**TODO**
+The `ProjectableTableSource` interface adds support for projection push-down to a `TableSource`. A `TableSource` extending this interface is able to project the fields of the return table.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+ProjectableTableSource<T> {
+
+ public TableSource<T> projectFields(int[] fields);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+ProjectableTableSource[T] {
+
+ def TableSource[T] projectFields(fields: Array[Int])
+}
+{% endhighlight %}
+</div>
+</div>
+
+The `projectFields()` is called with an array that holds the indexes of the required fields. The method returns a new `TableSource` object that returns rows with the requested schema.
{% top %}
+### NestedFieldsProjectableTableSource
+
+The `NestedFieldsProjectableTableSource` interface adds support for projection push-down to a `TableSource` with nested fields. A `TableSource` extending this interface is able to project the nested fields of the returned table.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+NestedFieldsProjectableTableSource<T> {
+
+ public TableSource<T> projectNestedFields(int[] fields, String[][] nestedFields);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+NestedFieldsProjectableTableSource[T] {
+
+ def projectNestedFields(fields: Array[Int], nestedFields: Array[Array[String]]): TableSource[T]
+}
+{% endhighlight %}
+</div>
+</div>
+
### FilterableTableSource
-**TODO**
+The `FilterableTableSource` interface adds support for filtering push-down to a `TableSource`. A `TableSource` extending this interface is able to filter records before returning.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+FilterableTableSource<T> {
+
+ public TableSource<T> applyPredicate(List<Expression> predicates);
+
+ public boolean isFilterPushedDown();
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+FilterableTableSource[T] {
+
+ def applyPredicate(predicates: java.util.List[Expression]): TableSource[T]
+
+ def isFilterPushedDown: Boolean
+}
+{% endhighlight %}
+</div>
+</div>
+
+The optimizer pushes predicates down by calling the `applyPredicate()` method. The `TableSource` can evaluate which predicates to evaluate by itself and which to leave for the framework. Predicates which are evaluated by the `TableSource` must be removed from the `List`. All predicates which remain in the `List` after the method call returns are evaluated by the framework. The `applyPredicate()` method returns a new `TableSource` that evaluates all selected predicates.
+
+The `isFilterPushedDown()` method tells the optimizer whether predicates have been pushed down or not.
{% top %}
Define a TableSink
------------------
+A `TableSink` specifies how to emit a `Table` to an external system or location. The interface is generic such that it can support different storage locations and formats. There are different table sinks for batch tables and streaming tables.
+
+The general interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+TableSink<T> {
+
+ public TypeInformation<T> getOutputType();
+
+ public String[] getFieldNames();
+
+ public TypeInformation[] getFieldTypes();
+
+ public TableSink<T> configure(String[] fieldNames, TypeInformation[] fieldTypes);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+TableSink[T] {
+
+ def getOutputType: TypeInformation<T>
+
+ def getFieldNames: Array[String]
+
+ def getFieldTypes: Array[TypeInformation]
+
+ def configure(fieldNames: Array[String], fieldTypes: Array[TypeInformation]): TableSink[T]
+}
+{% endhighlight %}
+</div>
+</div>
+
+The `TableSink#configure` method is called to pass the schema of the Table (field names and types) to emit to the `TableSink`. The method must return a new instance of the TableSink which is configured to emit the provided Table schema.
+
### BatchTableSink
-**TODO**
+Defines an external `TableSink` to emit a batch table.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+BatchTableSink<T> extends TableSink<T> {
+
+ public void emitDataSet(DataSet<T> dataSet);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+BatchTableSink[T] extends TableSink[T] {
+
+ def emitDataSet(dataSet: DataSet[T]): Unit
+}
+{% endhighlight %}
+</div>
+</div>
{% top %}
### AppendStreamTableSink
-**TODO**
+Defines an external `TableSink` to emit a streaming table with only insert changes.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+AppendStreamTableSink<T> extends TableSink<T> {
+
+ public void emitDataStream(DataStream<T> dataStream);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+AppendStreamTableSink[T] extends TableSink[T] {
+
+ def emitDataStream(dataStream: DataStream<T>): Unit
+}
+{% endhighlight %}
+</div>
+</div>
+
+If the table is also modified by update or delete changes, a `TableException` will be thrown.
{% top %}
### RetractStreamTableSink
-**TODO**
+Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+RetractStreamTableSink<T> extends TableSink<Tuple2<Boolean, T>> {
+
+ public TypeInformation<T> getRecordType();
+
+ public void emitDataStream(DataStream<Tuple2<Boolean, T>> dataStream);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+RetractStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] {
+
+ def getRecordType: TypeInformation[T]
+
+ def emitDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): Unit
+}
+{% endhighlight %}
+</div>
+</div>
+
+The table will be converted into a stream of accumulate and retraction messages which are encoded as Java `Tuple2`. The first field is a boolean flag to indicate the message type (`true` indicates insert, `false` indicates delete). The second field holds the record of the requested type `T`.
{% top %}
-### UpsertStreamTableSInk
+### UpsertStreamTableSink
-**TODO**
+Defines an external `TableSink` to emit a streaming table with insert, update, and delete changes.
+
+The interface looks as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+UpsertStreamTableSink<T> extends TableSink<Tuple2<Boolean, T>> {
+
+ public void setKeyFields(String[] keys);
+
+ public void setIsAppendOnly(boolean isAppendOnly);
+
+ public TypeInformation<T> getRecordType();
+
+ public void emitDataStream(DataStream<Tuple2<Boolean, T>> dataStream);
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+UpsertStreamTableSink[T] extends TableSink[Tuple2[Boolean, T]] {
+
+ def setKeyFields(keys: Array[String]): Unit
+
+ def setIsAppendOnly(isAppendOnly: Boolean): Unit
+
+ def getRecordType: TypeInformation[T]
+
+ def emitDataStream(dataStream: DataStream[Tuple2[Boolean, T]]): Unit
+}
+{% endhighlight %}
+</div>
+</div>
+
+The table must be have unique key fields (atomic or composite) or be append-only. If the table does not have a unique key and is not append-only, a `TableException` will be thrown. The unique key of the table is configured by the `UpsertStreamTableSink#setKeyFields()` method.
+
+The table will be converted into a stream of upsert and delete messages which are encoded as a Java `Tuple2`. The first field is a boolean flag to indicate the message type. The second field holds the record of the requested type `T`.
+
+A message with true boolean field is an upsert message for the configured key. A message with false flag is a delete message for the configured key. If the table is append-only, all messages will have a true flag and must be interpreted as insertions.
{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/d8756553/docs/dev/table/streaming.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/streaming.md b/docs/dev/table/streaming.md
index d7d97fa..c7f070b 100644
--- a/docs/dev/table/streaming.md
+++ b/docs/dev/table/streaming.md
@@ -27,10 +27,12 @@ under the License.
* This will be replaced by the TOC
{:toc}
-Dynamic table
+Dynamic Table
-------------
-**TO BE DONE**
+This section will be reworked soon. Until then, please read the [introductory blog post about Dynamic Tables](http://flink.apache.org/news/2017/04/04/dynamic-tables.html).
+
+**TO BE DONE:**
* Stream -> Table
* Table -> Stream
@@ -76,7 +78,7 @@ env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) // default
</div>
</div>
-Time-based operations such as windows in both the [Table API]({{ site.baseurl }}/dev/table/tableApi.html#windows) and [SQL]({{ site.baseurl }}/dev/table/sql.html#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+Time-based operations such as windows in both the [Table API]({{ site.baseurl }}/dev/table/tableApi.html#group-windows) and [SQL]({{ site.baseurl }}/dev/table/sql.html#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
Time attributes can be part of every table schema. They are defined when creating a table from a `DataStream` or are pre-defined when using a `TableSource`. Once a time attribute has been defined at the beginning, it can be referenced as a field and can used in time-based operations.
http://git-wip-us.apache.org/repos/asf/flink/blob/d8756553/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSource.scala
----------------------------------------------------------------------
diff --git a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSource.scala b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSource.scala
index c41582e..d9ebc5a 100644
--- a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSource.scala
+++ b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/sources/TableSource.scala
@@ -25,7 +25,7 @@ import org.apache.flink.api.common.typeinfo.TypeInformation
* Schema information consists of a data type, field names, and corresponding indices of
* these names in the data type.
*
- * To define a TableSource one need to implement [[TableSource#getReturnType]]. In this case
+ * To define a TableSource one needs to implement [[TableSource#getReturnType]]. In this case
* field names and field indices are derived from the returned type.
*
* In case if custom field names are required one need to additionally implement
[09/10] flink git commit: [FLINK-6748] [table] [docs] Reworked Table
API Page
Posted by fh...@apache.org.
[FLINK-6748] [table] [docs] Reworked Table API Page
This closes #4093.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/23248157
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/23248157
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/23248157
Branch: refs/heads/master
Commit: 232481572bb48e82880afdb2f7237af08a8404b5
Parents: ddae51f
Author: twalthr <tw...@apache.org>
Authored: Fri Jun 9 06:54:22 2017 +0200
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:42:19 2017 +0200
----------------------------------------------------------------------
docs/dev/table/index.md | 82 ++++++
docs/dev/table/tableApi.md | 534 +++++++++++++++++++++++++++++----------
docs/dev/tableApi.md | 82 ------
docs/redirects/table.md | 2 +-
docs/redirects/table_api.md | 24 ++
5 files changed, 504 insertions(+), 220 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/23248157/docs/dev/table/index.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/index.md b/docs/dev/table/index.md
new file mode 100644
index 0000000..df2ccba
--- /dev/null
+++ b/docs/dev/table/index.md
@@ -0,0 +1,82 @@
+---
+title: "Table API & SQL"
+nav-id: tableapi
+nav-parent_id: dev
+is_beta: true
+nav-show_overview: true
+nav-pos: 35
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
+
+The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
+
+**Please note that the Table API and SQL are not yet feature complete and are being actively developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
+
+Setup
+-----
+
+The Table API and SQL are bundled in the `flink-table` Maven artifact.
+The following dependency must be added to your project in order to use the Table API and SQL:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+In addition, you need to add a dependency for either Flink's Scala batch or streaming API. For a batch query you need to add:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+For a streaming query you need to add:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+**Note:** Due to an issue in Apache Calcite, which prevents the user classloaders from being garbage-collected, we do *not* recommend building a fat-jar that includes the `flink-table` dependency. Instead, we recommend configuring Flink to include the `flink-table` dependency in the system classloader. This can be done by copying the `flink-table.jar` file from the `./opt` folder to the `./lib` folder. See [these instructions]({{ site.baseurl }}/dev/linking.html) for further details.
+
+{% top %}
+
+Where to go next?
+-----------------
+
+* [Concepts & Common API]({{ site.baseurl }}/dev/table/common.html): Shared concepts and APIs of the Table API and SQL.
+* [Streaming Table API & SQL]({{ site.baseurl }}/dev/table/streaming.html): Streaming-specific documentation for the Table API or SQL such as configuration of time attributes and handling of updating results.
+* [Table API]({{ site.baseurl }}/dev/table/tableapi.html): Supported operations and API for the Table API.
+* [SQL]({{ site.baseurl }}/dev/table/sql.html): Supported operations and syntax for SQL
+* [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html): Reading tables from and emitting tables to external storage systems.
+* [User-Defined Functions]({{ site.baseurl }}/dev/table/udfs.html): Definition and usage of user-defined functions.
+
+{% top %}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/flink/blob/23248157/docs/dev/table/tableApi.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md
index 25810d2..693b9bd 100644
--- a/docs/dev/table/tableApi.md
+++ b/docs/dev/table/tableApi.md
@@ -22,94 +22,129 @@ specific language governing permissions and limitations
under the License.
-->
-The Table API is a language-integrated relational API for Scala and Java. The Table API is a unified API for stream and batch processing.
+The Table API is a unified, relational API for stream and batch processing. Table API queries can be run on batch or streaming input without modifications. The Table API is a super set of the SQL language and is specially designed for working with Apache Flink. The Table API is a language-integrated API for Scala and Java. Instead of specifying queries as String values as common with SQL, Table API queries are defined in a language-embedded style in Java or Scala with IDE support like autocompletion and syntax validation.
-Please have a look at the [Common Concepts & API]({{ site.baseurl }}/dev/table/common.html) and the [Streaming Concepts]({{ site.baseurl }}/dev/table/streaming.html) if you work with streaming data.
+The Table API shares many concepts and parts of its API with Flink's SQL integration. Have a look at the [Common Concepts & API]({{ site.baseurl }}/dev/table/common.html) to learn how to register tables or to create a `Table` object. The [Streaming Concepts]({{ site.baseurl }}/dev/table/streaming.html) page discusses streaming specific concepts such as dynamic tables and time attributes.
-The following examples assume a registered table called `Orders` with attributes `a, b, c, rowtime`.
-
-**TODO: Extend**
+The following examples assume a registered table called `Orders` with attributes `(a, b, c, rowtime)`. The `rowtime` field is either a logical [time attribute](streaming.html#time-attributes) in streaming or a regular timestamp field in batch.
* This will be replaced by the TOC
{:toc}
-Table API Overview
-------------------
-
-The Table API is available for Scala and Java. The Scala Table API is based on Scala Expressions, the Java Table API on Strings which are parsed and converted into Expressions.
+Overview & Examples
+-----------------------------
-The following example shows the differences between the Scala and Java Table API.
+The Table API is available for Scala and Java. The Scala Table API leverages on Scala expressions, the Java Table API is based on strings which are parsed and converted into equivalent expressions.
-**TODO: Extend**
+The following example shows the differences between the Scala and Java Table API. The table program is executed in a batch environment. It scans the `Orders` table, groups by field `a`, and counts the resulting rows per group. The result of the table program is converted into a `DataSet` of type `Row` and printed.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
-The Java Table API is enabled by importing `org.apache.flink.table.api.java.*`. The following example shows how a Java Table API program is constructed.
+The Java Table API is enabled by importing `org.apache.flink.table.api.java.*`. The following example shows how a Java Table API program is constructed and how expressions are specified as strings.
{% highlight java %}
-
+// environment configuration
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
BatchTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env);
+// register Orders table in table environment
+// ...
+
+// specify table program
Table orders = tEnv.scan("Orders"); // schema (a, b, c, rowtime)
Table counts = orders
.groupBy("a")
.select("a, b.count as cnt");
-DataSet<Row> result = tableEnv.toDataSet(wordCounts, Row.class);
-{% endhighlight %}
-
-With Java, expressions must be specified by Strings. The embedded expression DSL is not supported.
-
-{% highlight java %}
-ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
-BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
-
-// register the DataSet cust as table "Customers" with fields derived from the dataset
-tableEnv.registerDataSet("Customers", cust)
-
-// register the DataSet ord as table "Orders" with fields user, product, and amount
-tableEnv.registerDataSet("Orders", ord, "user, product, amount");
+// conversion to DataSet
+DataSet<Row> result = tableEnv.toDataSet(counts, Row.class);
+result.print();
{% endhighlight %}
</div>
<div data-lang="scala" markdown="1">
-The Scala Table API is enabled by importing `org.apache.flink.table.api.scala._`. The following example shows how a Scala Table API program is constructed.
+The Scala Table API is enabled by importing `org.apache.flink.api.scala._` and `org.apache.flink.table.api.scala._`.
+
+The following example shows how a Scala Table API program is constructed. Table attributes are referenced using [Scala Symbols](http://scala-lang.org/files/archive/spec/2.12/01-lexical-syntax.html#symbol-literals), which start with an apostrophe character (`'`).
{% highlight scala %}
import org.apache.flink.api.scala._
import org.apache.flink.table.api.scala._
+// environment configuration
val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = TableEnvironment.getTableEnvironment(env)
+// register Orders table in table environment
+// ...
+
+// specify table program
val orders = tEnv.scan("Orders") // schema (a, b, c, rowtime)
+
val result = orders
.groupBy('a)
.select('a, 'b.count as 'cnt)
- .toDataSet[Row]
+ .toDataSet[Row] // conversion to DataSet
+ .print()
{% endhighlight %}
-The expression DSL uses Scala symbols to refer to field names and code generation to
-transform expressions to efficient runtime code. Please note that the conversion to and from
-Tables only works when using Scala case classes or Java POJOs. Please refer to the [Type Extraction and Serialization]({{ site.baseurl }}/internals/types_serialization.html) section
-to learn the characteristics of a valid POJO.
+</div>
+</div>
+
+The next example shows a more complex Table API program. The program scans again the `Orders` table. It filters null values, normalizes the field `a` of type String, and calculates for each hour and product `a` the average billing amount `b`.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+{% highlight java %}
+// environment configuration
+// ...
+
+// specify table program
+Table orders = tEnv.scan("Orders"); // schema (a, b, c, rowtime)
+
+Table result = orders
+ .filter("a.isNotNull && b.isNotNull && c.isNotNull")
+ .select("a.lowerCase(), b, rowtime")
+ .window(Tumble.over("1.hour").on("rowtime").as("hourlyWindow"))
+ .groupBy("hourlyWindow, a")
+ .select("a, hourlyWindow.end as hour, b.avg as avgBillingAmount");
+{% endhighlight %}
+
+</div>
+
+<div data-lang="scala" markdown="1">
+
+{% highlight scala %}
+// environment configuration
+// ...
+
+// specify table program
+val orders: Table = tEnv.scan("Orders") // schema (a, b, c, rowtime)
+
+val result: Table = orders
+ .filter('a.isNotNull && 'b.isNotNull && 'c.isNotNull)
+ .select('a.lowerCase(), 'b, 'rowtime)
+ .window(Tumble over 1.hour on 'rowtime as 'hourlyWindow)
+ .groupBy('hourlyWindow, 'a)
+ .select('a, 'hourlyWindow.end as 'hour, 'b.avg as 'avgBillingAmount);
+{% endhighlight %}
</div>
</div>
-**TODO**
+Since the Table API is a unified API for batch and streaming data, both example programs can be executed on batch and streaming inputs without any modification of the table program itself. In both cases, the program produces the same results given that streaming records are not late (see [Streaming Concepts](streaming.html) for details).
{% top %}
Operations
----------
-**TODO: Add Tags for Batch and Streaming support**
+The Table API supports the following operations. Please note that not all operations are available in both batch and streaming yet; they are tagged accordingly.
### Scan, Projection, and Filter
@@ -125,7 +160,10 @@ Operations
</thead>
<tbody>
<tr>
- <td><strong>Scan</strong></td>
+ <td>
+ <strong>Scan</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to the FROM clause in a SQL query. Performs a scan of a registered table.</p>
{% highlight java %}
@@ -134,7 +172,10 @@ Table orders = tableEnv.scan("Orders");
</td>
</tr>
<tr>
- <td><strong>Select</strong></td>
+ <td>
+ <strong>Select</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to a SQL SELECT statement. Performs a select operation.</p>
{% highlight java %}
@@ -149,7 +190,10 @@ Table result = orders.select("*");
</tr>
<tr>
- <td><strong>As</strong></td>
+ <td>
+ <strong>As</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Renames fields.</p>
{% highlight java %}
@@ -160,17 +204,20 @@ Table result = orders.as("x, y, z, t");
</tr>
<tr>
- <td><strong>Where / Filter</strong></td>
+ <td>
+ <strong>Where / Filter</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to a SQL WHERE clause. Filters out rows that do not pass the filter predicate.</p>
{% highlight java %}
Table orders = tableEnv.scan("Orders");
-Table result = orders.where("b = 'red'");
+Table result = orders.where("b === 'red'");
{% endhighlight %}
or
{% highlight java %}
Table orders = tableEnv.scan("Orders");
-Table result = orders.filter("a % 2 = 0");
+Table result = orders.filter("a % 2 === 0");
{% endhighlight %}
</td>
</tr>
@@ -189,7 +236,10 @@ Table result = orders.filter("a % 2 = 0");
</thead>
<tbody>
<tr>
- <td><strong>Scan</strong></td>
+ <td>
+ <strong>Scan</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to the FROM clause in a SQL query. Performs a scan of a registered table.</p>
{% highlight scala %}
@@ -198,7 +248,10 @@ val orders: Table = tableEnv.scan("Orders")
</td>
</tr>
<tr>
- <td><strong>Select</strong></td>
+ <td>
+ <strong>Select</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to a SQL SELECT statement. Performs a select operation.</p>
{% highlight scala %}
@@ -214,17 +267,23 @@ val result = orders.select('*)
</tr>
<tr>
- <td><strong>As</strong></td>
+ <td>
+ <strong>As</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Renames fields.</p>
{% highlight scala %}
-val orders: Table = tableEnv.scan("Orders").as('x, 'y, 'z, 't')
+val orders: Table = tableEnv.scan("Orders").as('x, 'y, 'z, 't)
{% endhighlight %}
</td>
</tr>
<tr>
- <td><strong>Where / Filter</strong></td>
+ <td>
+ <strong>Where / Filter</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to a SQL WHERE clause. Filters out rows that do not pass the filter predicate.</p>
{% highlight scala %}
@@ -259,26 +318,63 @@ val result = orders.where('b === "red")
</thead>
<tbody>
<tr>
- <td><strong>GroupBy</strong></td>
<td>
- <p>Similar to a SQL GROUPBY clause. Groups the rows on the grouping keys, with a following aggregation
- operator to aggregate rows group-wise.</p>
+ <strong>GroupBy Aggregation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span><br>
+ <span class="label label-info">Result Updating</span>
+ </td>
+ <td>
+ <p>Similar to a SQL GROUP BY clause. Groups the rows on the grouping keys with a following running aggregation operator to aggregate rows group-wise.</p>
{% highlight java %}
Table orders = tableEnv.scan("Orders");
Table result = orders.groupBy("a").select("a, b.sum as d");
{% endhighlight %}
+ <p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the type of aggregation and the number of distinct grouping keys. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
- <td><strong>GroupBy Window</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>GroupBy Window Aggregation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Groups and aggregates a table on a <a href="#group-windows">group window</a> and possibly one or more grouping keys.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders
+ .window(Tumble.over("5.minutes").on("rowtime").as("w")) // define window
+ .groupBy("a, w") // group by key and window
+ .select("a, w.start, w.end, b.sum as d"); // access window properties and aggregate
+{% endhighlight %}
+ </td>
</tr>
<tr>
- <td><strong>Over Window</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>Over Window Aggregation</strong><br>
+ <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Similar to a SQL OVER clause. Over window aggregates are computed for each row, based on a window (range) of preceding and succeeding rows. See the <a href="#over-windows">over windows section</a> for more details.</p>
+ {% highlight scala %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders
+ // define window
+ .window(Over
+ .partitionBy("a")
+ .orderBy("rowtime")
+ .preceding("UNBOUNDED_RANGE")
+ .following("CURRENT_RANGE")
+ .as("w")
+ .select("a, b.avg over w, b.max over w, b.min over w") // sliding aggregate
+{% endhighlight %}
+ <p><b>Note:</b> All aggregates must be defined over the same window, i.e., same partitioning, sorting, and range. Currently, only windows with PRECEDING (UNBOUNDED and bounded) to CURRENT ROW range are supported. Ranges with FOLLOWING are not supported yet. ORDER BY must be specified on a single <a href="streaming.html#time-attributes">time attribute</a>.</p>
+ </td>
</tr>
<tr>
- <td><strong>Distinct</strong></td>
+ <td>
+ <strong>Distinct</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL DISTINCT clause. Returns records with distinct value combinations.</p>
{% highlight java %}
@@ -303,26 +399,63 @@ Table result = orders.distinct();
<tbody>
<tr>
- <td><strong>GroupBy</strong></td>
<td>
- <p>Similar to a SQL GROUPBY clause. Groups rows on the grouping keys, with a following aggregation
- operator to aggregate rows group-wise.</p>
+ <strong>GroupBy Aggregation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span><br>
+ <span class="label label-info">Result Updating</span>
+ </td>
+ <td>
+ <p>Similar to a SQL GROUP BY clause. Groups the rows on the grouping keys with a following running aggregation operator to aggregate rows group-wise.</p>
{% highlight scala %}
val orders: Table = tableEnv.scan("Orders")
val result = orders.groupBy('a).select('a, 'b.sum as 'd)
{% endhighlight %}
+ <p><b>Note:</b> For streaming queries the required state to compute the query result might grow infinitely depending on the type of aggregation and the number of distinct grouping keys. Please provide a query configuration with valid retention interval to prevent excessive state size. See <a href="streaming.html">Streaming Concepts</a> for details.</p>
</td>
</tr>
<tr>
- <td><strong>GroupBy Window</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>GroupBy Window Aggregation</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Groups and aggregates a table on a <a href="#group-windows">group window</a> and possibly one or more grouping keys.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result: Table = orders
+ .window(Tumble over 5.minutes on 'rowtime as 'w) // define window
+ .groupBy('a, 'w) // group by key and window
+ .select('a, w.start, 'w.end, 'b.sum as 'd) // access window properties and aggregate
+{% endhighlight %}
+ </td>
</tr>
<tr>
- <td><strong>Over Window</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>Over Window Aggregation</strong><br>
+ <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Similar to a SQL OVER clause. Over window aggregates are computed for each row, based on a window (range) of preceding and succeeding rows. See the <a href="#over-windows">over windows section</a> for more details.</p>
+ {% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result: Table = orders
+ // define window
+ .window(Over
+ partitionBy 'a
+ orderBy 'rowtime
+ preceding UNBOUNDED_RANGE
+ following CURRENT_RANGE
+ as 'w)
+ .select('a, 'b.avg over 'w, 'b.max over 'w, 'b.min over 'w,) // sliding aggregate
+{% endhighlight %}
+ <p><b>Note:</b> All aggregates must be defined over the same window, i.e., same partitioning, sorting, and range. Currently, only windows with PRECEDING (UNBOUNDED and bounded) to CURRENT ROW range are supported. Ranges with FOLLOWING are not supported yet. ORDER BY must be specified on a single <a href="streaming.html#time-attributes">time attribute</a>.</p>
+ </td>
</tr>
<tr>
- <td><strong>Distinct</strong></td>
+ <td>
+ <strong>Distinct</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL DISTINCT clause. Returns records with distinct value combinations.</p>
{% highlight scala %}
@@ -352,7 +485,10 @@ val result = orders.distinct()
</thead>
<tbody>
<tr>
- <td><strong>Inner Join</strong></td>
+ <td>
+ <strong>Inner Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined through join operator or using a where or filter operator.</p>
{% highlight java %}
@@ -364,7 +500,10 @@ Table result = left.join(right).where("a = d").select("a, b, e");
</tr>
<tr>
- <td><strong>LeftOuterJoin</strong></td>
+ <td>
+ <strong>Left Outer Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL LEFT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
{% highlight java %}
@@ -376,7 +515,10 @@ Table result = left.leftOuterJoin(right, "a = d").select("a, b, e");
</tr>
<tr>
- <td><strong>RightOuterJoin</strong></td>
+ <td>
+ <strong>Right Outer Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL RIGHT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
{% highlight java %}
@@ -388,7 +530,10 @@ Table result = left.rightOuterJoin(right, "a = d").select("a, b, e");
</tr>
<tr>
- <td><strong>FullOuterJoin</strong></td>
+ <td>
+ <strong>Full Outer Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
{% highlight java %}
@@ -399,12 +544,46 @@ Table result = left.fullOuterJoin(right, "a = d").select("a, b, e");
</td>
</tr>
<tr>
- <td><strong>TableFunction CrossJoin</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>TableFunction Join</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Joins a table with a the results of a table function. Each row of the left (outer) table is joined with all rows produced by the corresponding call of the table function. A row of the left (outer) table is dropped, if its table function call returns an empty result.
+ </p>
+{% highlight java %}
+// register function
+TableFunction<String> split = new MySplitUDTF();
+tEnv.registerFunction("split", split);
+
+// join
+Table orders = tableEnv.scan("Orders");
+Table result = orders
+ .join(new Table(tEnv, "split(c)").as("s", "t", "v")))
+ .select("a, b, s, t, v");
+{% endhighlight %}
+ </td>
</tr>
<tr>
- <td><strong>TableFunction LeftOuterJoin</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>TableFunction Left Outer Join</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Joins a table with a the results of a table function. Each row of the left (outer) table is joined with all rows produced by the corresponding call of the table function. If a table function call returns an empty result, the corresponding outer row is preserved and the result padded with null values.
+ </p>
+{% highlight java %}
+// register function
+TableFunction<String> split = new MySplitUDTF();
+tEnv.registerFunction("split", split);
+
+// join
+Table orders = tableEnv.scan("Orders");
+Table result = orders
+ .leftOuterJoin(new Table(tEnv, "split(c)").as("s", "t", "v")))
+ .select("a, b, s, t, v");
+{% endhighlight %}
+ </td>
</tr>
</tbody>
@@ -423,7 +602,10 @@ Table result = left.fullOuterJoin(right, "a = d").select("a, b, e");
<tbody>
<tr>
- <td><strong>Join</strong></td>
+ <td>
+ <strong>Inner Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and an equality join predicate must be defined using a where or filter operator.</p>
{% highlight scala %}
@@ -435,7 +617,10 @@ val result = left.join(right).where('a === 'd).select('a, 'b, 'e);
</tr>
<tr>
- <td><strong>LeftOuterJoin</strong></td>
+ <td>
+ <strong>Left Outer Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL LEFT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
{% highlight scala %}
@@ -447,7 +632,10 @@ val result = left.leftOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
</tr>
<tr>
- <td><strong>RightOuterJoin</strong></td>
+ <td>
+ <strong>Right Outer Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL RIGHT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
{% highlight scala %}
@@ -459,7 +647,10 @@ val result = left.rightOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
</tr>
<tr>
- <td><strong>FullOuterJoin</strong></td>
+ <td>
+ <strong>Full Outer Join</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
{% highlight scala %}
@@ -470,12 +661,41 @@ val result = left.fullOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
</td>
</tr>
<tr>
- <td><strong>TableFunction CrossJoin</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>TableFunction Join</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
+ <td>
+ <p>Joins a table with a the results of a table function. Each row of the left (outer) table is joined with all rows produced by the corresponding call of the table function. A row of the left (outer) table is dropped, if its table function call returns an empty result.
+ </p>
+ {% highlight scala %}
+// instantiate function
+val split: TableFunction[_] = new MySplitUDTF()
+
+// join
+val result: Table = table
+ .join(split('c) as ('s, 't, 'v))
+ .select('a, 'b, 's, 't, 'v)
+{% endhighlight %}
+ </td>
</tr>
<tr>
- <td><strong>TableFunction LeftOuterJoin</strong></td>
- <td>TODO</td>
+ <td>
+ <strong>TableFunction Left Outer Join</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span></td>
+ <td>
+ <p>Joins a table with a the results of a table function. Each row of the left (outer) table is joined with all rows produced by the corresponding call of the table function. If a table function call returns an empty result, the corresponding outer row is preserved and the result padded with null values.
+ </p>
+{% highlight scala %}
+// instantiate function
+val split: TableFunction[_] = new MySplitUDTF()
+
+// join
+val result: Table = table
+ .leftOuterJoin(split('c) as ('s, 't, 'v))
+ .select('a, 'b, 's, 't, 'v)
+{% endhighlight %}
+ </td>
</tr>
</tbody>
@@ -499,7 +719,10 @@ val result = left.fullOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
</thead>
<tbody>
<tr>
- <td><strong>Union</strong></td>
+ <td>
+ <strong>Union</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL UNION clause. Unions two tables with duplicate records removed. Both tables must have identical field types.</p>
{% highlight java %}
@@ -511,7 +734,10 @@ Table result = left.union(right);
</tr>
<tr>
- <td><strong>UnionAll</strong></td>
+ <td>
+ <strong>UnionAll</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+ </td>
<td>
<p>Similar to a SQL UNION ALL clause. Unions two tables. Both tables must have identical field types.</p>
{% highlight java %}
@@ -523,7 +749,10 @@ Table result = left.unionAll(right);
</tr>
<tr>
- <td><strong>Intersect</strong></td>
+ <td>
+ <strong>Intersect</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL INTERSECT clause. Intersect returns records that exist in both tables. If a record is present one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Both tables must have identical field types.</p>
{% highlight java %}
@@ -535,7 +764,10 @@ Table result = left.intersect(right);
</tr>
<tr>
- <td><strong>IntersectAll</strong></td>
+ <td>
+ <strong>IntersectAll</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL INTERSECT ALL clause. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Both tables must have identical field types.</p>
{% highlight java %}
@@ -547,7 +779,10 @@ Table result = left.intersectAll(right);
</tr>
<tr>
- <td><strong>Minus</strong></td>
+ <td>
+ <strong>Minus</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.</p>
{% highlight java %}
@@ -559,7 +794,10 @@ Table result = left.minus(right);
</tr>
<tr>
- <td><strong>MinusAll</strong></td>
+ <td>
+ <strong>MinusAll</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.</p>
{% highlight java %}
@@ -584,7 +822,10 @@ Table result = left.minusAll(right);
</thead>
<tbody>
<tr>
- <td><strong>Union</strong></td>
+ <td>
+ <strong>Union</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL UNION clause. Unions two tables with duplicate records removed, both tables must have identical field types.</p>
{% highlight scala %}
@@ -596,7 +837,11 @@ val result = left.union(right);
</tr>
<tr>
- <td><strong>UnionAll</strong></td>
+ <td>
+ <strong>UnionAll</strong><br>
+ <span class="label label-primary">Batch</span> <span class="label label-primary">Streaming</span>
+
+ </td>
<td>
<p>Similar to a SQL UNION ALL clause. Unions two tables, both tables must have identical field types.</p>
{% highlight scala %}
@@ -608,7 +853,10 @@ val result = left.unionAll(right);
</tr>
<tr>
- <td><strong>Intersect</strong></td>
+ <td>
+ <strong>Intersect</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL INTERSECT clause. Intersect returns records that exist in both tables. If a record is present in one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Both tables must have identical field types.</p>
{% highlight scala %}
@@ -620,7 +868,10 @@ val result = left.intersect(right);
</tr>
<tr>
- <td><strong>IntersectAll</strong></td>
+ <td>
+ <strong>IntersectAll</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL INTERSECT ALL clause. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Both tables must have identical field types.</p>
{% highlight scala %}
@@ -632,7 +883,10 @@ val result = left.intersectAll(right);
</tr>
<tr>
- <td><strong>Minus</strong></td>
+ <td>
+ <strong>Minus</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.</p>
{% highlight scala %}
@@ -644,7 +898,10 @@ val result = left.minus(right);
</tr>
<tr>
- <td><strong>MinusAll</strong></td>
+ <td>
+ <strong>MinusAll</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.</p>
{% highlight scala %}
@@ -676,7 +933,10 @@ val result = left.minusAll(right);
</thead>
<tbody>
<tr>
- <td><strong>Order By</strong></td>
+ <td>
+ <strong>Order By</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL ORDER BY clause. Returns records globally sorted across all parallel partitions.</p>
{% highlight scala %}
@@ -687,7 +947,10 @@ val result = in.orderBy('a.asc);
</tr>
<tr>
- <td><strong>Limit</strong></td>
+ <td>
+ <strong>Limit</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL LIMIT clause. Limits a sorted result to a specified number of records from an offset position. Limit is technically part of the Order By operator and thus must be preceded by it.</p>
{% highlight scala %}
@@ -717,7 +980,10 @@ val result = in.orderBy('a.asc).limit(3, 5); // returns 5 records beginning with
</thead>
<tbody>
<tr>
- <td><strong>Order By</strong></td>
+ <td>
+ <strong>Order By</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL ORDER BY clause. Returns records globally sorted across all parallel partitions.</p>
{% highlight java %}
@@ -728,7 +994,10 @@ Table result = in.orderBy("a.asc");
</tr>
<tr>
- <td><strong>Limit</strong></td>
+ <td>
+ <strong>Limit</strong><br>
+ <span class="label label-primary">Batch</span>
+ </td>
<td>
<p>Similar to a SQL LIMIT clause. Limits a sorted result to a specified number of records from an offset position. Limit is technically part of the Order By operator and thus must be preceded by it.</p>
{% highlight java %}
@@ -748,11 +1017,9 @@ Table result = in.orderBy("a.asc").limit(3, 5); // returns 5 records beginning w
</div>
</div>
-### Windows
+### Group Windows
-**TODO: Figure out where to put this stuff. I think it would be good to have it in the "Operations" section for a brief overview. A more detailed discussion of windows should go somewhere else, maybe into the "Common Concepts"?**
-
-The Table API is a declarative API to define queries on batch and streaming tables. Projection, selection, and union operations can be applied both on streaming and batch tables without additional semantics. Aggregations on (possibly) infinite streaming tables, however, can only be computed on finite groups of records. Window aggregates group rows into finite groups based on time or row-count intervals and evaluate aggregation functions once per group. For batch tables, windows are a convenient shortcut to group records by time intervals.
+Group window aggregates group rows into finite groups based on time or row-count intervals and evaluate aggregation functions once per group. For batch tables, windows are a convenient shortcut to group records by time intervals.
Windows are defined using the `window(w: Window)` clause and require an alias, which is specified using the `as` clause. In order to group a table by a window, the window alias must be referenced in the `groupBy(...)` clause like a regular grouping attribute.
The following example shows how to define a window aggregation on a table.
@@ -800,7 +1067,7 @@ val table = input
</div>
</div>
-The `Window` parameter defines how rows are mapped to windows. `Window` is not an interface that users can implement. Instead, the Table API provides a set of predefined `Window` classes with specific semantics, which are translated into underlying `DataStream` or `DataSet` operations. The supported window definitions are listed below. Window properties such as the start and end timestamp of a time window can be added in the select statement as a property of the window alias as `w.start` and `w.end`, respectively.
+Window properties such as the start and end timestamp of a time window can be added in the select statement as a property of the window alias as `w.start` and `w.end`, respectively.
<div class="codetabs" markdown="1">
<div data-lang="java" markdown="1">
@@ -822,6 +1089,8 @@ val table = input
</div>
</div>
+The `Window` parameter defines how rows are mapped to windows. `Window` is not an interface that users can implement. Instead, the Table API provides a set of predefined `Window` classes with specific semantics, which are translated into underlying `DataStream` or `DataSet` operations. The supported window definitions are listed below.
+
#### Tumble (Tumbling Windows)
A tumbling window assigns rows to non-overlapping, continuous windows of fixed length. For example, a tumbling window of 5 minutes groups rows in 5 minutes intervals. Tumbling windows can be defined on event-time, processing-time, or on a row-count.
@@ -832,7 +1101,6 @@ Tumbling windows are defined by using the `Tumble` class as follows:
<thead>
<tr>
<th class="text-left" style="width: 20%">Method</th>
- <th class="text-left" style="width: 20%">Required?</th>
<th class="text-left">Description</th>
</tr>
</thead>
@@ -840,17 +1108,14 @@ Tumbling windows are defined by using the `Tumble` class as follows:
<tbody>
<tr>
<td><code>over</code></td>
- <td>Required.</td>
<td>Defines the length the window, either as time or row-count interval.</td>
</tr>
<tr>
<td><code>on</code></td>
- <td>Required for streaming event-time windows and windows on batch tables.</td>
- <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped.</td>
+ <td>The time attribute to group (time interval) or sort (row count) on. For batch queries this might be any Long or Timestamp attribute. For streaming queries this must be a <a href="streaming.html#time-attributes">declared event-time or processing-time time attribute</a>.</td>
</tr>
<tr>
<td><code>as</code></td>
- <td>Required.</td>
<td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
</tr>
</tbody>
@@ -862,11 +1127,11 @@ Tumbling windows are defined by using the `Tumble` class as follows:
// Tumbling Event-time Window
.window(Tumble.over("10.minutes").on("rowtime").as("w"));
-// Tumbling Processing-time Window
-.window(Tumble.over("10.minutes").as("w"));
+// Tumbling Processing-time Window (assuming a processing-time attribute "proctime")
+.window(Tumble.over("10.minutes").on("proctime").as("w"));
-// Tumbling Row-count Window
-.window(Tumble.over("10.rows").as("w"));
+// Tumbling Row-count Window (assuming a processing-time attribute "proctime")
+.window(Tumble.over("10.rows").on("proctime").as("w"));
{% endhighlight %}
</div>
@@ -875,11 +1140,11 @@ Tumbling windows are defined by using the `Tumble` class as follows:
// Tumbling Event-time Window
.window(Tumble over 10.minutes on 'rowtime as 'w)
-// Tumbling Processing-time Window
-.window(Tumble over 10.minutes as 'w)
+// Tumbling Processing-time Window (assuming a processing-time attribute "proctime")
+.window(Tumble over 10.minutes on 'proctime as 'w)
-// Tumbling Row-count Window
-.window(Tumble over 10.rows as 'w)
+// Tumbling Row-count Window (assuming a processing-time attribute "proctime")
+.window(Tumble over 10.rows on 'proctime as 'w)
{% endhighlight %}
</div>
</div>
@@ -894,7 +1159,6 @@ Sliding windows are defined by using the `Slide` class as follows:
<thead>
<tr>
<th class="text-left" style="width: 20%">Method</th>
- <th class="text-left" style="width: 20%">Required?</th>
<th class="text-left">Description</th>
</tr>
</thead>
@@ -902,22 +1166,18 @@ Sliding windows are defined by using the `Slide` class as follows:
<tbody>
<tr>
<td><code>over</code></td>
- <td>Required.</td>
<td>Defines the length of the window, either as time or row-count interval.</td>
</tr>
<tr>
<td><code>every</code></td>
- <td>Required.</td>
<td>Defines the slide interval, either as time or row-count interval. The slide interval must be of the same type as the size interval.</td>
</tr>
<tr>
<td><code>on</code></td>
- <td>Required for event-time windows and windows on batch tables.</td>
- <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped</td>
+ <td>The time attribute to group (time interval) or sort (row count) on. For batch queries this might be any Long or Timestamp attribute. For streaming queries this must be a <a href="streaming.html#time-attributes">declared event-time or processing-time time attribute</a>.</td>
</tr>
<tr>
<td><code>as</code></td>
- <td>Required.</td>
<td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
</tr>
</tbody>
@@ -929,11 +1189,11 @@ Sliding windows are defined by using the `Slide` class as follows:
// Sliding Event-time Window
.window(Slide.over("10.minutes").every("5.minutes").on("rowtime").as("w"));
-// Sliding Processing-time window
-.window(Slide.over("10.minutes").every("5.minutes").as("w"));
+// Sliding Processing-time window (assuming a processing-time attribute "proctime")
+.window(Slide.over("10.minutes").every("5.minutes").on("proctime").as("w"));
-// Sliding Row-count window
-.window(Slide.over("10.rows").every("5.rows").as("w"));
+// Sliding Row-count window (assuming a processing-time attribute "proctime")
+.window(Slide.over("10.rows").every("5.rows").on("proctime").as("w"));
{% endhighlight %}
</div>
@@ -942,11 +1202,11 @@ Sliding windows are defined by using the `Slide` class as follows:
// Sliding Event-time Window
.window(Slide over 10.minutes every 5.minutes on 'rowtime as 'w)
-// Sliding Processing-time window
-.window(Slide over 10.minutes every 5.minutes as 'w)
+// Sliding Processing-time window (assuming a processing-time attribute "proctime")
+.window(Slide over 10.minutes every 5.minutes on 'proctime as 'w)
-// Sliding Row-count window
-.window(Slide over 10.rows every 5.rows as 'w)
+// Sliding Row-count window (assuming a processing-time attribute "proctime")
+.window(Slide over 10.rows every 5.rows on 'proctime as 'w)
{% endhighlight %}
</div>
</div>
@@ -961,7 +1221,6 @@ A session window is defined by using the `Session` class as follows:
<thead>
<tr>
<th class="text-left" style="width: 20%">Method</th>
- <th class="text-left" style="width: 20%">Required?</th>
<th class="text-left">Description</th>
</tr>
</thead>
@@ -969,17 +1228,14 @@ A session window is defined by using the `Session` class as follows:
<tbody>
<tr>
<td><code>withGap</code></td>
- <td>Required.</td>
<td>Defines the gap between two windows as time interval.</td>
</tr>
<tr>
<td><code>on</code></td>
- <td>Required for event-time windows and windows on batch tables.</td>
- <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped</td>
+ <td>The time attribute to group (time interval) or sort (row count) on. For batch queries this might be any Long or Timestamp attribute. For streaming queries this must be a <a href="streaming.html#time-attributes">declared event-time or processing-time time attribute</a>.</td>
</tr>
<tr>
<td><code>as</code></td>
- <td>Required.</td>
<td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
</tr>
</tbody>
@@ -991,8 +1247,8 @@ A session window is defined by using the `Session` class as follows:
// Session Event-time Window
.window(Session.withGap("10.minutes").on("rowtime").as("w"));
-// Session Processing-time Window
-.window(Session.withGap("10.minutes").as("w"));
+// Session Processing-time Window (assuming a processing-time attribute "proctime")
+.window(Session.withGap("10.minutes").on("proctime").as("w"));
{% endhighlight %}
</div>
@@ -1001,14 +1257,18 @@ A session window is defined by using the `Session` class as follows:
// Session Event-time Window
.window(Session withGap 10.minutes on 'rowtime as 'w)
-// Session Processing-time Window
-.window(Session withGap 10.minutes as 'w)
+// Session Processing-time Window (assuming a processing-time attribute "proctime")
+.window(Session withGap 10.minutes on 'proctime as 'w)
{% endhighlight %}
</div>
</div>
{% top %}
+### Over Windows
+
+**TO BE DONE**
+
Data Types
----------
@@ -3546,14 +3806,14 @@ ANY.as(name [, name ]* )
</div>
-### Limitations
+### Unsupported Functions
The following operations are not supported yet:
- Binary string operators and functions
- System functions
- Collection functions
-- Aggregate functions like STDDEV_xxx, VAR_xxx, and REGR_xxx
+- Aggregate functions like REGR_xxx
- Distinct aggregate functions like COUNT DISTINCT
{% top %}
http://git-wip-us.apache.org/repos/asf/flink/blob/23248157/docs/dev/tableApi.md
----------------------------------------------------------------------
diff --git a/docs/dev/tableApi.md b/docs/dev/tableApi.md
deleted file mode 100644
index f7b13f0..0000000
--- a/docs/dev/tableApi.md
+++ /dev/null
@@ -1,82 +0,0 @@
----
-title: "Table API & SQL"
-nav-id: tableapi
-nav-parent_id: dev
-is_beta: true
-nav-show_overview: true
-nav-pos: 35
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-
-The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-
-**Please note that the Table API and SQL are not yet feature complete and are being active developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
-
-Setup
------
-
-The Table API and SQL are bundled in the `flink-table` Maven artifact.
-The following dependency must be added to your project in order to use the Table API and SQL:
-
-{% highlight xml %}
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-{% endhighlight %}
-
-In addition, you need to add a dependency for either Flink's Scala batch or streaming API. For a batch query you need to add:
-
-{% highlight xml %}
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-{% endhighlight %}
-
-For a streaming query you need to add:
-
-{% highlight xml %}
-<dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
- <version>{{site.version }}</version>
-</dependency>
-{% endhighlight %}
-
-**Note:** Due to an issue in Apache Calcite, which prevents the user classloaders from being garbage-collected, we do *not* recommend building a fat-jar that includes the `flink-table` dependency. Instead, we recommend configuring Flink to include the `flink-table` dependency in the system classloader. This can be done by copying the `flink-table.jar` file from the `./opt` folder to the `./lib` folder. See [these instructions]({{ site.baseurl }}/dev/linking.html) for further details.
-
-{% top %}
-
-Where to go next?
------------------
-
-* [Concepts & Common API]({{ site.baseurl }}/dev/table/common.html): Shared concepts and APIs of the Table API and SQL.
-* [Streaming Table API & SQL]({{ site.baseurl }}/dev/table/streaming.html): Streaming-specific documentation for the Table API or SQL such as configuration of time attributes and handling of updating results.
-* [Table API]({{ site.baseurl }}/dev/table/tableapi.html): Supported operations and API for the Table API.
-* [SQL]({{ site.baseurl }}/dev/table/sql.html): Supported operations and syntax for SQL
-* [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html): Reading tables from and emitting tables to external storage systems.
-* [User-Defined Functions]({{ site.baseurl }}/dev/table/udfs.html): Definition and usage of user-defined functions.
-
-{% top %}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/flink/blob/23248157/docs/redirects/table.md
----------------------------------------------------------------------
diff --git a/docs/redirects/table.md b/docs/redirects/table.md
index 0cc9a32..0284608 100644
--- a/docs/redirects/table.md
+++ b/docs/redirects/table.md
@@ -1,7 +1,7 @@
---
title: "Table API"
layout: redirect
-redirect: /dev/table_api.html
+redirect: /dev/table/index.html
permalink: /apis/table.html
---
<!--
http://git-wip-us.apache.org/repos/asf/flink/blob/23248157/docs/redirects/table_api.md
----------------------------------------------------------------------
diff --git a/docs/redirects/table_api.md b/docs/redirects/table_api.md
new file mode 100644
index 0000000..c1afaec
--- /dev/null
+++ b/docs/redirects/table_api.md
@@ -0,0 +1,24 @@
+---
+title: "Table API"
+layout: redirect
+redirect: /dev/table/index.html
+permalink: /dev/table_api.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
[02/10] flink git commit: [FLNK-5354] [docs] Restructured Table API /
SQL docs
Posted by fh...@apache.org.
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table/udfs.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/udfs.md b/docs/dev/table/udfs.md
new file mode 100644
index 0000000..55f58b6
--- /dev/null
+++ b/docs/dev/table/udfs.md
@@ -0,0 +1,362 @@
+---
+title: "User-defined Functions"
+nav-parent_id: tableapi
+nav-pos: 50
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+User-defined functions are an important feature, because they significantly extend the expressiveness of queries.
+
+**TODO**
+
+* This will be replaced by the TOC
+{:toc}
+
+Register User-Defined Functions
+-------------------------------
+
+**TODO**
+
+{% top %}
+
+Scalar Functions
+----------------
+
+If a required scalar function is not contained in the built-in functions, it is possible to define custom, user-defined scalar functions for both the Table API and SQL. A user-defined scalar functions maps zero, one, or multiple scalar values to a new scalar value.
+
+In order to define a scalar function one has to extend the base class `ScalarFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a scalar function is determined by the evaluation method. An evaluation method must be declared publicly and named `eval`. The parameter types and return type of the evaluation method also determine the parameter and return types of the scalar function. Evaluation methods can also be overloaded by implementing multiple methods named `eval`.
+
+The following example shows how to define your own hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+public class HashCode extends ScalarFunction {
+ private int factor = 12;
+
+ public HashCode(int factor) {
+ this.factor = factor;
+ }
+
+ public int eval(String s) {
+ return s.hashCode() * factor;
+ }
+}
+
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// register the function
+tableEnv.registerFunction("hashCode", new HashCode(10));
+
+// use the function in Java Table API
+myTable.select("string, string.hashCode(), hashCode(string)");
+
+// use the function in SQL API
+tableEnv.sql("SELECT string, HASHCODE(string) FROM MyTable");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// must be defined in static/object context
+class HashCode(factor: Int) extends ScalarFunction {
+ def eval(s: String): Int = {
+ s.hashCode() * factor
+ }
+}
+
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// use the function in Scala Table API
+val hashCode = new HashCode(10)
+myTable.select('string, hashCode('string))
+
+// register and use the function in SQL
+tableEnv.registerFunction("hashCode", new HashCode(10))
+tableEnv.sql("SELECT string, HASHCODE(string) FROM MyTable");
+{% endhighlight %}
+</div>
+</div>
+
+By default the result type of an evaluation method is determined by Flink's type extraction facilities. This is sufficient for basic types or simple POJOs but might be wrong for more complex, custom, or composite types. In these cases `TypeInformation` of the result type can be manually defined by overriding `ScalarFunction#getResultType()`.
+
+Internally, the Table API and SQL code generation works with primitive values as much as possible. If a user-defined scalar function should not introduce much overhead through object creation/casting during runtime, it is recommended to declare parameters and result types as primitive types instead of their boxed classes. `Types.DATE` and `Types.TIME` can also be represented as `int`. `Types.TIMESTAMP` can be represented as `long`.
+
+The following example shows an advanced example which takes the internal timestamp representation and also returns the internal timestamp representation as a long value. By overriding `ScalarFunction#getResultType()` we define that the returned long value should be interpreted as a `Types.TIMESTAMP` by the code generation.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+public static class TimestampModifier extends ScalarFunction {
+ public long eval(long t) {
+ return t % 1000;
+ }
+
+ public TypeInformation<?> getResultType(signature: Class<?>[]) {
+ return Types.TIMESTAMP;
+ }
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+object TimestampModifier extends ScalarFunction {
+ def eval(t: Long): Long = {
+ t % 1000
+ }
+
+ override def getResultType(signature: Array[Class[_]]): TypeInformation[_] = {
+ Types.TIMESTAMP
+ }
+}
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+Table Functions
+---------------
+
+Similar to a user-defined scalar function, a user-defined table function takes zero, one, or multiple scalar values as input parameters. However in contrast to a scalar function, it can return an arbitrary number of rows as output instead of a single value. The returned rows may consist of one or more columns.
+
+In order to define a table function one has to extend the base class `TableFunction` in `org.apache.flink.table.functions` and implement (one or more) evaluation methods. The behavior of a table function is determined by its evaluation methods. An evaluation method must be declared `public` and named `eval`. The `TableFunction` can be overloaded by implementing multiple methods named `eval`. The parameter types of the evaluation methods determine all valid parameters of the table function. The type of the returned table is determined by the generic type of `TableFunction`. Evaluation methods emit output rows using the protected `collect(T)` method.
+
+In the Table API, a table function is used with `.join(Expression)` or `.leftOuterJoin(Expression)` for Scala users and `.join(String)` or `.leftOuterJoin(String)` for Java users. The `join` operator (cross) joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator). The `leftOuterJoin` operator joins each row from the outer table (table on the left of the operator) with all rows produced by the table-valued function (which is on the right side of the operator) and preserves outer rows for which the table function returns an empty table. In SQL use `LATERAL TABLE(<TableFunction>)` with CROSS JOIN and LEFT JOIN with an ON TRUE join condition (see examples below).
+
+The following example shows how to define table-valued function, register it in the TableEnvironment, and call it in a query. Note that you can configure your table function via a constructor before it is registered:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// The generic type "Tuple2<String, Integer>" determines the schema of the returned table as (String, Integer).
+public class Split extends TableFunction<Tuple2<String, Integer>> {
+ private String separator = " ";
+
+ public Split(String separator) {
+ this.separator = separator;
+ }
+
+ public void eval(String str) {
+ for (String s : str.split(separator)) {
+ // use collect(...) to emit a row
+ collect(new Tuple2<String, Integer>(s, s.length()));
+ }
+ }
+}
+
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+Table myTable = ... // table schema: [a: String]
+
+// Register the function.
+tableEnv.registerFunction("split", new Split("#"));
+
+// Use the table function in the Java Table API. "as" specifies the field names of the table.
+myTable.join("split(a) as (word, length)").select("a, word, length");
+myTable.leftOuterJoin("split(a) as (word, length)").select("a, word, length");
+
+// Use the table function in SQL with LATERAL and TABLE keywords.
+// CROSS JOIN a table function (equivalent to "join" in Table API).
+tableEnv.sql("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)");
+// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API).
+tableEnv.sql("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// The generic type "(String, Int)" determines the schema of the returned table as (String, Integer).
+class Split(separator: String) extends TableFunction[(String, Int)] {
+ def eval(str: String): Unit = {
+ // use collect(...) to emit a row.
+ str.split(separator).foreach(x -> collect((x, x.length))
+ }
+}
+
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+val myTable = ... // table schema: [a: String]
+
+// Use the table function in the Scala Table API (Note: No registration required in Scala Table API).
+val split = new Split("#")
+// "as" specifies the field names of the generated table.
+myTable.join(split('a) as ('word, 'length)).select('a, 'word, 'length);
+myTable.leftOuterJoin(split('a) as ('word, 'length)).select('a, 'word, 'length);
+
+// Register the table function to use it in SQL queries.
+tableEnv.registerFunction("split", new Split("#"))
+
+// Use the table function in SQL with LATERAL and TABLE keywords.
+// CROSS JOIN a table function (equivalent to "join" in Table API)
+tableEnv.sql("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)");
+// LEFT JOIN a table function (equivalent to "leftOuterJoin" in Table API)
+tableEnv.sql("SELECT a, word, length FROM MyTable LEFT JOIN TABLE(split(a)) as T(word, length) ON TRUE");
+{% endhighlight %}
+**IMPORTANT:** Do not implement TableFunction as a Scala object. Scala object is a singleton and will cause concurrency issues.
+</div>
+</div>
+
+Please note that POJO types do not have a deterministic field order. Therefore, you cannot rename the fields of POJO returned by a table function using `AS`.
+
+By default the result type of a `TableFunction` is determined by Flink’s automatic type extraction facilities. This works well for basic types and simple POJOs but might be wrong for more complex, custom, or composite types. In such a case, the type of the result can be manually specified by overriding `TableFunction#getResultType()` which returns its `TypeInformation`.
+
+The following example shows an example of a `TableFunction` that returns a `Row` type which requires explicit type information. We define that the returned table type should be `RowTypeInfo(String, Integer)` by overriding `TableFunction#getResultType()`.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+public class CustomTypeSplit extends TableFunction<Row> {
+ public void eval(String str) {
+ for (String s : str.split(" ")) {
+ Row row = new Row(2);
+ row.setField(0, s);
+ row.setField(1, s.length);
+ collect(row);
+ }
+ }
+
+ @Override
+ public TypeInformation<Row> getResultType() {
+ return new RowTypeInfo(new TypeInformation[]{
+ BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.INT_TYPE_INFO});
+ }
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+class CustomTypeSplit extends TableFunction[Row] {
+ def eval(str: String): Unit = {
+ str.split(" ").foreach({ s =>
+ val row = new Row(2)
+ row.setField(0, s)
+ row.setField(1, s.length)
+ collect(row)
+ })
+ }
+
+ override def getResultType: TypeInformation[Row] = {
+ new RowTypeInfo(Seq(BasicTypeInfo.STRING_TYPE_INFO,
+ BasicTypeInfo.INT_TYPE_INFO))
+ }
+}
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+Aggregation Functions
+---------------------
+
+**TODO**
+
+{% top %}
+
+Integrating UDFs with the Runtime
+---------------------------------
+
+Sometimes it might be necessary for a user-defined function to get global runtime information or do some setup/clean-up work before the actual work. User-defined functions provide `open()` and `close()` methods that can be overriden and provide similar functionality as the methods in `RichFunction` of DataSet or DataStream API.
+
+The `open()` method is called once before the evaluation method. The `close()` method after the last call to the evaluation method.
+
+The `open()` method provides a `FunctionContext` that contains information about the context in which user-defined functions are executed, such as the metric group, the distributed cache files, or the global job parameters.
+
+The following information can be obtained by calling the corresponding methods of `FunctionContext`:
+
+| Method | Description |
+| :------------------------------------ | :----------------------------------------------------- |
+| `getMetricGroup()` | Metric group for this parallel subtask. |
+| `getCachedFile(name)` | Local temporary file copy of a distributed cache file. |
+| `getJobParameter(name, defaultValue)` | Global job parameter value associated with given key. |
+
+The following example snippet shows how to use `FunctionContext` in a scalar function for accessing a global job parameter:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+public class HashCode extends ScalarFunction {
+
+ private int factor = 0;
+
+ @Override
+ public void open(FunctionContext context) throws Exception {
+ // access "hashcode_factor" parameter
+ // "12" would be the default value if parameter does not exist
+ factor = Integer.valueOf(context.getJobParameter("hashcode_factor", "12"));
+ }
+
+ public int eval(String s) {
+ return s.hashCode() * factor;
+ }
+}
+
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// set job parameter
+Configuration conf = new Configuration();
+conf.setString("hashcode_factor", "31");
+env.getConfig().setGlobalJobParameters(conf);
+
+// register the function
+tableEnv.registerFunction("hashCode", new HashCode());
+
+// use the function in Java Table API
+myTable.select("string, string.hashCode(), hashCode(string)");
+
+// use the function in SQL
+tableEnv.sql("SELECT string, HASHCODE(string) FROM MyTable");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+object hashCode extends ScalarFunction {
+
+ var hashcode_factor = 12;
+
+ override def open(context: FunctionContext): Unit = {
+ // access "hashcode_factor" parameter
+ // "12" would be the default value if parameter does not exist
+ hashcode_factor = context.getJobParameter("hashcode_factor", "12").toInt
+ }
+
+ def eval(s: String): Int = {
+ s.hashCode() * hashcode_factor
+ }
+}
+
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// use the function in Scala Table API
+myTable.select('string, hashCode('string))
+
+// register and use the function in SQL
+tableEnv.registerFunction("hashCode", hashCode)
+tableEnv.sql("SELECT string, HASHCODE(string) FROM MyTable");
+{% endhighlight %}
+
+</div>
+</div>
+
+{% top %}
+
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/tableApi.md
----------------------------------------------------------------------
diff --git a/docs/dev/tableApi.md b/docs/dev/tableApi.md
new file mode 100644
index 0000000..c1c02f1
--- /dev/null
+++ b/docs/dev/tableApi.md
@@ -0,0 +1,81 @@
+---
+title: "Table API & SQL"
+nav-id: tableapi
+nav-parent_id: dev
+is_beta: true
+nav-show_overview: true
+nav-pos: 35
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+**Table API and SQL are experimental features**
+
+The Table API is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flink's DataSet and DataStream APIs (Java and Scala).
+The Table API and SQL interface operate on a relational `Table` abstraction, which can be created from external data sources, or existing DataSets and DataStreams. With the Table API, you can apply relational operators such as selection, aggregation, and joins on `Table`s.
+
+`Table`s can also be queried with regular SQL, as long as they are registered (see [Registering Tables](#registering-tables)). The Table API and SQL offer equivalent functionality and can be mixed in the same program. When a `Table` is converted back into a `DataSet` or `DataStream`, the logical plan, which was defined by relational operators and SQL queries, is optimized using [Apache Calcite](https://calcite.apache.org/) and transformed into a `DataSet` or `DataStream` program.
+
+**TODO: Check, update, and add**
+
+* What are the Table API / SQL
+ * Relational APIs
+ * Unified APIs for batch and streaming
+ * Semantics are the same
+ * But not all operations can be efficiently mapped to streams
+ * Table API: language-integrated queries (LINQ) in Scala and Java
+ * SQL: Standard SQL
+
+**Please notice: Not all operations are supported by all four combinations of Stream/Batch and TableAPI/SQL.**
+
+* This will be replaced by the TOC
+{:toc}
+
+Setup
+-----
+
+The Table API and SQL are part of the *flink-table* Maven project.
+The following dependency must be added to your project in order to use the Table API and SQL:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+*Note: The Table API is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/linking.html).*
+
+**TODO: Rework and add:**
+* Project dependencies (flink-table + flink-scala or flink-streaming-scala)
+* Copy `./opt/flink-table.jar` to `./lib`
+
+{% top %}
+
+Where to go next?
+-----------------
+
+* [Concepts & Common API]({{ site.baseurl }}/dev/table/common.html): Share concepts and API of the Table API and SQL.
+* [Table API]({{ site.baseurl }}/dev/table/tableapi.html): Supported Operations and API for the Table API
+* [SQL]({{ site.baseurl }}/dev/table/sql.html): Supported Operations and Syntax for SQL
+* [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html): Ingestion and emission of tables.
+* [User-Defined Functions]({{ site.baseurl }}/dev/table/udfs.html): Defintion and usage of user-defined functions.
+
+{% top %}
\ No newline at end of file
[04/10] flink git commit: [FLNK-5354] [docs] Restructured Table API /
SQL docs
Posted by fh...@apache.org.
[FLNK-5354] [docs] Restructured Table API / SQL docs
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a5d93a56
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a5d93a56
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a5d93a56
Branch: refs/heads/master
Commit: a5d93a56cb37e691ec9bb06d17c76151e7619267
Parents: 9141379
Author: Fabian Hueske <fh...@apache.org>
Authored: Tue May 23 16:19:49 2017 +0100
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:41:59 2017 +0200
----------------------------------------------------------------------
docs/dev/table/common.md | 548 ++++
docs/dev/table/sourceSinks.md | 286 ++
docs/dev/table/sql.md | 1778 +++++++++++
docs/dev/table/streaming.md | 67 +
docs/dev/table/tableApi.md | 3559 +++++++++++++++++++++
docs/dev/table/udfs.md | 362 +++
docs/dev/tableApi.md | 81 +
docs/dev/table_api.md | 6015 ------------------------------------
8 files changed, 6681 insertions(+), 6015 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table/common.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/common.md b/docs/dev/table/common.md
new file mode 100644
index 0000000..d0d4914
--- /dev/null
+++ b/docs/dev/table/common.md
@@ -0,0 +1,548 @@
+---
+title: "Concepts & Common API"
+nav-parent_id: tableapi
+nav-pos: 0
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The Table API and SQL are integrated API and share many concepts and much of their API.
+
+**TODO: Extend**
+
+* This will be replaced by the TOC
+{:toc}
+
+Structure of Table API and SQL Programs
+---------------------------------------
+
+All Table API and SQL programs for batch and streaming have the same structure.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+// Create a TableEnvironment
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// Register a Table
+tableEnv.registerTable("yourTable", ...) // or
+tableEnv.registerTableSource("yourTableSrc", ...); // or
+tableEnv.registerDataStream("yourTableStream", ...); // or
+tableEnv.registerDataSet("yourTableSet", ...); // or
+tableEnv.registerExternalCatalog("yourCatalog", ...);
+
+// Create a table from a Table API query
+Table tapiResult = tableEnv.scan("yourTableSrc").select(...);
+// Or create a table from a SQL query
+Table sqlResult = tableEnv.sql("SELECT ... FROM yourTableSrc ... ");
+
+// Emit a Table to a TableSink / DataStream / DataSet
+resultTable.writeToSink(...); // or
+resultTable.toAppendStream(...); // or
+resultTable.toRetractStream(...); // or
+resultTable.toDataSet(...);
+
+// Execute
+env.execute("Your Query");
+
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = ExecutionEnvironment.getExecutionEnvironment
+
+// Create a TableEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// Register a Table
+tableEnv.registerTable("yourTable", ...) // or
+tableEnv.registerTableSource("yourTableSrc", ...) // or
+tableEnv.registerDataStream("yourTableStream", ...) // or
+tableEnv.registerDataSet("yourTableSet", ...) // or
+tableEnv.registerExternalCatalog("yourCatalog", ...)
+
+// Create a table from a Table API query
+val tapiResult = tableEnv.scan("yourTableSrc").select(...)
+// Or create a table from a SQL query
+val sqlResult = tableEnv.sql("SELECT ... FROM yourTableSrc ...")
+
+// Emit a Table
+resultTable.writeToSink(...) // or
+resultTable.toAppendStream(...) // or
+resultTable.toRetractStream(...) // or
+resultTable.toDataSet(...)
+
+// Execute
+env.execute("Your Query")
+
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+Create a TableEnvironment
+-------------------------
+
+A `Table` is always bound to a specific `TableEnvironment`. It is not possible to combine Tables of different TableEnvironments.
+
+**TODO: Extend**
+
+{% top %}
+
+Register a Table in the Catalog
+-------------------------------
+
+`TableEnvironment`s have an internal table catalog to which tables can be registered with a unique name. After registration, a table can be accessed from the `TableEnvironment` by its name.
+
+*Note: `DataSet`s or `DataStream`s can be directly converted into `Table`s without registering them in the `TableEnvironment`. See [Create a Table from a DataStream or DataSet](#tbd) for details.
+
+### Register a Table
+
+A `Table` that originates from a Table API operation or a SQL query is registered in a `TableEnvironment` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// works for StreamExecutionEnvironment identically
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// convert a DataSet into a Table
+Table custT = tableEnv
+ .toTable(custDs, "name, zipcode")
+ .where("zipcode = '12345'")
+ .select("name");
+
+// register the Table custT as table "custNames"
+tableEnv.registerTable("custNames", custT);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// works for StreamExecutionEnvironment identically
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// convert a DataSet into a Table
+val custT = custDs
+ .toTable(tableEnv, 'name, 'zipcode)
+ .where('zipcode === "12345")
+ .select('name)
+
+// register the Table custT as table "custNames"
+tableEnv.registerTable("custNames", custT)
+{% endhighlight %}
+</div>
+</div>
+
+A registered `Table` that originates from a Table API operation or SQL query is treated similarly as a view as known from relational DBMS, i.e., it can be inlined when optimizing the query.
+
+{% top %}
+
+### Register a DataSet
+
+A `DataSet` is registered as a `Table` in a `BatchTableEnvironment` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// register the DataSet cust as table "Customers" with fields derived from the dataset
+tableEnv.registerDataSet("Customers", cust);
+
+// register the DataSet ord as table "Orders" with fields user, product, and amount
+tableEnv.registerDataSet("Orders", ord, "user, product, amount");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// register the DataSet cust as table "Customers" with fields derived from the dataset
+tableEnv.registerDataSet("Customers", cust)
+
+// register the DataSet ord as table "Orders" with fields user, product, and amount
+tableEnv.registerDataSet("Orders", ord, 'user, 'product, 'amount)
+{% endhighlight %}
+</div>
+</div>
+
+*Note: The name of a `DataSet` `Table` must not match the `^_DataSetTable_[0-9]+` pattern which is reserved for internal use only.*
+
+{% top %}
+
+### Register a DataStream
+
+A `DataStream` is registered as a `Table` in a `StreamTableEnvironment` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// register the DataStream cust as table "Customers" with fields derived from the datastream
+tableEnv.registerDataStream("Customers", cust);
+
+// register the DataStream ord as table "Orders" with fields user, product, and amount
+tableEnv.registerDataStream("Orders", ord, "user, product, amount");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// register the DataStream cust as table "Customers" with fields derived from the datastream
+tableEnv.registerDataStream("Customers", cust)
+
+// register the DataStream ord as table "Orders" with fields user, product, and amount
+tableEnv.registerDataStream("Orders", ord, 'user, 'product, 'amount)
+{% endhighlight %}
+</div>
+</div>
+
+*Note: The name of a `DataStream` `Table` must not match the `^_DataStreamTable_[0-9]+` pattern which is reserved for internal use only.*
+
+{% top %}
+
+### Register a TableSource
+
+TableSources provided access to data stored in various storage systems such as databases (MySQL, HBase, ...), file formats (CSV, Apache Parquet, Avro, ORC, ...), or messaging systems (Apache Kafka, RabbitMQ, ...). Flink provides a TableSources for common data formats and storage systems. Please have a look at the [Table Sources and Sinks page]({{ site.baseurl }}/dev/table/sourceSinks.html) for a list of provided TableSources and documentation for how to built your own.
+
+An external table is registered in a `TableEnvironment` using a `TableSource` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// works for StreamExecutionEnvironment identically
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+TableSource custTS = new CsvTableSource("/path/to/file", ...);
+
+// register a `TableSource` as external table "Customers"
+tableEnv.registerTableSource("Customers", custTS);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// works for StreamExecutionEnvironment identically
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+val custTS: TableSource = new CsvTableSource("/path/to/file", ...)
+
+// register a `TableSource` as external table "Customers"
+tableEnv.registerTableSource("Customers", custTS)
+
+{% endhighlight %}
+</div>
+</div>
+
+A `TableSource` can provide access to data stored in various storage systems such as databases (MySQL, HBase, ...), file formats (CSV, Apache Parquet, Avro, ORC, ...), or messaging systems (Apache Kafka, RabbitMQ, ...).
+
+{% top %}
+
+Register an External Catalog
+----------------------------
+
+An external catalog is defined by the `ExternalCatalog` interface and provides information about databases and tables such as their name, schema, statistics, and access information. An `ExternalCatalog` is registered in a `TableEnvironment` as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// works for StreamExecutionEnvironment identically
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+ExternalCatalog customerCatalog = new InMemoryExternalCatalog();
+
+// register the ExternalCatalog customerCatalog
+tableEnv.registerExternalCatalog("Customers", customerCatalog);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// works for StreamExecutionEnvironment identically
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+val customerCatalog: ExternalCatalog = new InMemoryExternalCatalog
+
+// register the ExternalCatalog customerCatalog
+tableEnv.registerExternalCatalog("Customers", customerCatalog)
+
+{% endhighlight %}
+</div>
+</div>
+
+Once registered in a `TableEnvironment`, all tables defined in a `ExternalCatalog` can be accessed from Table API or SQL queries by specifying their full path (`catalog`.`database`.`table`).
+
+Currently, Flink provides an `InMemoryExternalCatalog` for demo and testing purposes. However, the `ExternalCatalog` interface can also be used to connect catalogs like HCatalog or Metastore to the Table API.
+
+{% top %}
+
+Create a Table from a DataStream or DataSet
+-------------------------------------------
+
+Besides registering a Table in a catalog, it is also possible to directly create a `Table` from a `DataStream` or `DataSet`.
+
+### Create a Table from a DataStream
+
+**TODO**
+
+{% top %}
+
+### Create a Table from a DataSet
+
+**TODO**
+
+### Scala Implicit Conversion
+
+If you use the Scala API, A `DataSet` or `DataStream` can be implicitly converted into a `Table`.
+
+{% top %}
+
+Query a Table
+-------------
+
+### Table API
+
+**TODO**
+
+{% top %}
+
+### SQL
+
+**TODO**
+
+{% top %}
+
+### Interoperability
+
+**TODO**
+
+* Mix SQL and Table as you like
+* Table API to SQL requires registered tables, register Table
+* SQL to Table API just use resulting table
+
+{% top %}
+
+Emit a Table
+------------
+
+### Emit to a TableSink
+
+A `Table` can be written to a `TableSink`, which is a generic interface to support a wide variety of file formats (e.g. CSV, Apache Parquet, Apache Avro), storage systems (e.g., JDBC, Apache HBase, Apache Cassandra, Elasticsearch), or messaging systems (e.g., Apache Kafka, RabbitMQ). A batch `Table` can only be written to a `BatchTableSink`, a streaming table requires a `StreamTableSink`. A `TableSink` can implement both interfaces at the same time.
+
+Currently, Flink only provides a `CsvTableSink` that writes a batch or streaming `Table` to CSV-formatted files. A custom `TableSink` can be defined by implementing the `BatchTableSink` and/or `StreamTableSink` interface.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// compute the result Table using Table API operators and/or SQL queries
+Table result = ...
+
+// create a TableSink
+TableSink sink = new CsvTableSink("/path/to/file", fieldDelim = "|");
+// write the result Table to the TableSink
+result.writeToSink(sink);
+
+// execute the program
+env.execute();
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// compute the result Table using Table API operators and/or SQL queries
+val result: Table = ...
+
+// create a TableSink
+val sink: TableSink = new CsvTableSink("/path/to/file", fieldDelim = "|")
+// write the result Table to the TableSink
+result.writeToSink(sink)
+
+// execute the program
+env.execute()
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+### Convert to a DataStream
+
+**TODO**
+
+{% top %}
+
+### Convert to a DataSet
+
+**TODO**
+
+### Scala Implicit Conversion
+
+If you use the Scala API, A `Table` can be implicitly converted into a `DataSet` or `DataStream`.
+
+{% top %}
+
+Execute a Query
+---------------
+
+**TODO**
+
+{% top %}
+
+Mappings Types to Table Schema
+------------------------------
+
+* Explain how types are mapped to table schema
+ * Atomic Types
+ * Row
+ * Tuples (Java / Scala)
+ * Pojos
+ * Case Classes
+
+**TODO**
+
+{% top %}
+
+Integration with DataSet and DataStream API
+-------------------------------------------
+
+**TODO**
+
+* Create `Table` from `DataSet` and `DataStream` and back
+* Easy integration with more expressive APIs and libraries
+ * CEP / Gelly / ML
+ * Ingestion and projection
+
+{% top %}
+
+Query Optimization
+------------------
+
+* No join order yet
+* Filter / Projection push down
+* Custom rules
+
+### Explaining a Table
+
+The Table API provides a mechanism to explain the logical and optimized query plans to compute a `Table`.
+This is done through the `TableEnvironment#explain(table)` method. It returns a string describing three plans:
+
+1. the Abstract Syntax Tree of the relational query, i.e., the unoptimized logical query plan,
+2. the optimized logical query plan, and
+3. the physical execution plan.
+
+The following code shows an example and the corresponding output:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env);
+
+DataStream<Tuple2<Integer, String>> stream1 = env.fromElements(new Tuple2<>(1, "hello"));
+DataStream<Tuple2<Integer, String>> stream2 = env.fromElements(new Tuple2<>(1, "hello"));
+
+Table table1 = tEnv.fromDataStream(stream1, "count, word");
+Table table2 = tEnv.fromDataStream(stream2, "count, word");
+Table table = table1
+ .where("LIKE(word, 'F%')")
+ .unionAll(table2);
+
+String explanation = tEnv.explain(table);
+System.out.println(explanation);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tEnv = TableEnvironment.getTableEnvironment(env)
+
+val table1 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word)
+val table2 = env.fromElements((1, "hello")).toTable(tEnv, 'count, 'word)
+val table = table1
+ .where('word.like("F%"))
+ .unionAll(table2)
+
+val explanation: String = tEnv.explain(table)
+println(explanation)
+{% endhighlight %}
+</div>
+</div>
+
+{% highlight text %}
+== Abstract Syntax Tree ==
+LogicalUnion(all=[true])
+ LogicalFilter(condition=[LIKE($1, 'F%')])
+ LogicalTableScan(table=[[_DataStreamTable_0]])
+ LogicalTableScan(table=[[_DataStreamTable_1]])
+
+== Optimized Logical Plan ==
+DataStreamUnion(union=[count, word])
+ DataStreamCalc(select=[count, word], where=[LIKE(word, 'F%')])
+ DataStreamScan(table=[[_DataStreamTable_0]])
+ DataStreamScan(table=[[_DataStreamTable_1]])
+
+== Physical Execution Plan ==
+Stage 1 : Data Source
+ content : collect elements with CollectionInputFormat
+
+Stage 2 : Data Source
+ content : collect elements with CollectionInputFormat
+
+ Stage 3 : Operator
+ content : from: (count, word)
+ ship_strategy : REBALANCE
+
+ Stage 4 : Operator
+ content : where: (LIKE(word, 'F%')), select: (count, word)
+ ship_strategy : FORWARD
+
+ Stage 5 : Operator
+ content : from: (count, word)
+ ship_strategy : REBALANCE
+{% endhighlight %}
+
+{% top %}
+
+
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table/sourceSinks.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/sourceSinks.md b/docs/dev/table/sourceSinks.md
new file mode 100644
index 0000000..2d07254
--- /dev/null
+++ b/docs/dev/table/sourceSinks.md
@@ -0,0 +1,286 @@
+---
+title: "Table Sources & Sinks"
+nav-parent_id: tableapi
+nav-pos: 40
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* This will be replaced by the TOC
+{:toc}
+
+Provided TableSources
+---------------------
+
+**TODO: extend and complete**
+
+Currently, Flink provides the `CsvTableSource` to read CSV files and various `TableSources` to read JSON or Avro objects from Kafka.
+A custom `TableSource` can be defined by implementing the `BatchTableSource` or `StreamTableSource` interface.
+
+| **Class name** | **Maven dependency** | **Batch?** | **Streaming?** | **Description**
+| `CsvTableSouce` | `flink-table` | Y | Y | A simple source for CSV files.
+| `Kafka08JsonTableSource` | `flink-connector-kafka-0.8` | N | Y | A Kafka 0.8 source for JSON data.
+| `Kafka08AvroTableSource` | `flink-connector-kafka-0.8` | N | Y | A Kafka 0.8 source for Avro data.
+| `Kafka09JsonTableSource` | `flink-connector-kafka-0.9` | N | Y | A Kafka 0.9 source for JSON data.
+| `Kafka09AvroTableSource` | `flink-connector-kafka-0.9` | N | Y | A Kafka 0.9 source for Avro data.
+| `Kafka010JsonTableSource` | `flink-connector-kafka-0.10` | N | Y | A Kafka 0.10 source for JSON data.
+| `Kafka010AvroTableSource` | `flink-connector-kafka-0.10` | N | Y | A Kafka 0.10 source for Avro data.
+
+All sources that come with the `flink-table` dependency can be directly used by your Table programs. For all other table sources, you have to add the respective dependency in addition to the `flink-table` dependency.
+
+{% top %}
+
+### KafkaJsonTableSource
+
+To use the Kafka JSON source, you have to add the Kafka connector dependency to your project:
+
+ - `flink-connector-kafka-0.8` for Kafka 0.8,
+ - `flink-connector-kafka-0.9` for Kafka 0.9, or
+ - `flink-connector-kafka-0.10` for Kafka 0.10, respectively.
+
+You can then create the source as follows (example for Kafka 0.8):
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// specify JSON field names and types
+TypeInformation<Row> typeInfo = Types.ROW(
+ new String[] { "id", "name", "score" },
+ new TypeInformation<?>[] { Types.INT(), Types.STRING(), Types.DOUBLE() }
+);
+
+KafkaJsonTableSource kafkaTableSource = new Kafka08JsonTableSource(
+ kafkaTopic,
+ kafkaProperties,
+ typeInfo);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// specify JSON field names and types
+val typeInfo = Types.ROW(
+ Array("id", "name", "score"),
+ Array(Types.INT, Types.STRING, Types.DOUBLE)
+)
+
+val kafkaTableSource = new Kafka08JsonTableSource(
+ kafkaTopic,
+ kafkaProperties,
+ typeInfo)
+{% endhighlight %}
+</div>
+</div>
+
+By default, a missing JSON field does not fail the source. You can configure this via:
+
+```java
+// Fail on missing JSON field
+tableSource.setFailOnMissingField(true);
+```
+
+You can work with the Table as explained in the rest of the Table API guide:
+
+```java
+tableEnvironment.registerTableSource("kafka-source", kafkaTableSource);
+Table result = tableEnvironment.scan("kafka-source");
+```
+
+{% top %}
+
+### KafkaAvroTableSource
+
+The `KafkaAvroTableSource` allows you to read Avro's `SpecificRecord` objects from Kafka.
+
+To use the Kafka Avro source, you have to add the Kafka connector dependency to your project:
+
+ - `flink-connector-kafka-0.8` for Kafka 0.8,
+ - `flink-connector-kafka-0.9` for Kafka 0.9, or
+ - `flink-connector-kafka-0.10` for Kafka 0.10, respectively.
+
+You can then create the source as follows (example for Kafka 0.8):
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// pass the generated Avro class to the TableSource
+Class<? extends SpecificRecord> clazz = MyAvroType.class;
+
+KafkaAvroTableSource kafkaTableSource = new Kafka08AvroTableSource(
+ kafkaTopic,
+ kafkaProperties,
+ clazz);
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// pass the generated Avro class to the TableSource
+val clazz = classOf[MyAvroType]
+
+val kafkaTableSource = new Kafka08AvroTableSource(
+ kafkaTopic,
+ kafkaProperties,
+ clazz)
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+### CsvTableSource
+
+The `CsvTableSource` is already included in `flink-table` without additional dependecies.
+
+The easiest way to create a `CsvTableSource` is by using the enclosed builder `CsvTableSource.builder()`, the builder has the following methods to configure properties:
+
+ - `path(String path)` Sets the path to the CSV file, required.
+ - `field(String fieldName, TypeInformation<?> fieldType)` Adds a field with the field name and field type information, can be called multiple times, required. The call order of this method defines also the order of the fields in a row.
+ - `fieldDelimiter(String delim)` Sets the field delimiter, `","` by default.
+ - `lineDelimiter(String delim)` Sets the line delimiter, `"\n"` by default.
+ - `quoteCharacter(Character quote)` Sets the quote character for String values, `null` by default.
+ - `commentPrefix(String prefix)` Sets a prefix to indicate comments, `null` by default.
+ - `ignoreFirstLine()` Ignore the first line. Disabled by default.
+ - `ignoreParseErrors()` Skip records with parse error instead to fail. Throwing an exception by default.
+
+You can create the source as follows:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+CsvTableSource csvTableSource = CsvTableSource
+ .builder()
+ .path("/path/to/your/file.csv")
+ .field("name", Types.STRING())
+ .field("id", Types.INT())
+ .field("score", Types.DOUBLE())
+ .field("comments", Types.STRING())
+ .fieldDelimiter("#")
+ .lineDelimiter("$")
+ .ignoreFirstLine()
+ .ignoreParseErrors()
+ .commentPrefix("%");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val csvTableSource = CsvTableSource
+ .builder
+ .path("/path/to/your/file.csv")
+ .field("name", Types.STRING)
+ .field("id", Types.INT)
+ .field("score", Types.DOUBLE)
+ .field("comments", Types.STRING)
+ .fieldDelimiter("#")
+ .lineDelimiter("$")
+ .ignoreFirstLine
+ .ignoreParseErrors
+ .commentPrefix("%")
+{% endhighlight %}
+</div>
+</div>
+
+You can work with the Table as explained in the rest of the Table API guide in both stream and batch `TableEnvironment`s:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+tableEnvironment.registerTableSource("mycsv", csvTableSource);
+
+Table streamTable = streamTableEnvironment.scan("mycsv");
+
+Table batchTable = batchTableEnvironment.scan("mycsv");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+tableEnvironment.registerTableSource("mycsv", csvTableSource)
+
+val streamTable = streamTableEnvironment.scan("mycsv")
+
+val batchTable = batchTableEnvironment.scan("mycsv")
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+Provided TableSinks
+-------------------
+
+**TODO**
+
+{% top %}
+
+Define a TableSource
+--------------------
+
+### BatchTableSource
+
+**TODO**
+
+{% top %}
+
+### StreamTableSource
+* TimestampAssigner
+* DefinedRowtimeAttribute / DefinedProctimeAttribute
+
+**TODO**
+
+{% top %}
+
+### ProjectableTableSource
+
+**TODO**
+
+{% top %}
+
+### FilterableTableSource
+
+**TODO**
+
+{% top %}
+
+Define a TableSink
+------------------
+
+### BatchTableSink
+
+**TODO**
+
+{% top %}
+
+### AppendStreamTableSink
+
+**TODO**
+
+{% top %}
+
+### RetractStreamTableSink
+
+**TODO**
+
+{% top %}
+
+### UpsertStreamTableSInk
+
+**TODO**
+
+{% top %}
+
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table/sql.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/sql.md b/docs/dev/table/sql.md
new file mode 100644
index 0000000..b79f4f2
--- /dev/null
+++ b/docs/dev/table/sql.md
@@ -0,0 +1,1778 @@
+---
+title: "SQL"
+nav-parent_id: tableapi
+nav-pos: 30
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+SQL queries are specified using the `sql()` method of the `TableEnvironment`. The method returns the result of the SQL query as a `Table` which can be converted into a `DataSet` or `DataStream`, used in subsequent Table API queries, or written to a `TableSink` (see [Writing Tables to External Sinks](#writing-tables-to-external-sinks)). SQL and Table API queries can seamlessly mixed and are holistically optimized and translated into a single DataStream or DataSet program.
+
+A `Table`, `DataSet`, `DataStream`, or external `TableSource` must be registered in the `TableEnvironment` in order to be accessible by a SQL query (see [Registering Tables](#registering-tables)). For convenience `Table.toString()` will automatically register an unique table name under the `Table`'s `TableEnvironment` and return the table name. So it allows to call SQL directly on tables in a string concatenation (see examples below).
+
+*Note: Flink's SQL support is not feature complete, yet. Queries that include unsupported SQL features will cause a `TableException`. The limitations of SQL on batch and streaming tables are listed in the following sections.*
+
+**TODO: Rework intro. Move some parts below. **
+
+* This will be replaced by the TOC
+{:toc}
+
+Specifying a Query
+---------------
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// ingest a DataStream from an external source
+DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
+
+// call SQL on unregistered tables
+Table table = tableEnv.toTable(ds, "user, product, amount");
+Table result = tableEnv.sql(
+ "SELECT SUM(amount) FROM " + table + " WHERE product LIKE '%Rubber%'");
+
+// call SQL on registered tables
+// register the DataStream as table "Orders"
+tableEnv.registerDataStream("Orders", ds, "user, product, amount");
+// run a SQL query on the Table and retrieve the result as a new Table
+Table result2 = tableEnv.sql(
+ "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'");
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// read a DataStream from an external source
+val ds: DataStream[(Long, String, Integer)] = env.addSource(...)
+
+// call SQL on unregistered tables
+val table = ds.toTable(tableEnv, 'user, 'product, 'amount)
+val result = tableEnv.sql(
+ s"SELECT SUM(amount) FROM $table WHERE product LIKE '%Rubber%'")
+
+// call SQL on registered tables
+// register the DataStream under the name "Orders"
+tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)
+// run a SQL query on the Table and retrieve the result as a new Table
+val result2 = tableEnv.sql(
+ "SELECT product, amount FROM Orders WHERE product LIKE '%Rubber%'")
+{% endhighlight %}
+</div>
+</div>
+
+**TODO: Add some intro.**
+
+{% top %}
+
+Supported Syntax
+----------------
+
+Flink uses [Apache Calcite](https://calcite.apache.org/docs/reference.html) for SQL parsing. Currently, Flink SQL only supports query-related SQL syntax and only a subset of the comprehensive SQL standard. The following BNF-grammar describes the supported SQL features:
+
+```
+
+query:
+ values
+ | {
+ select
+ | selectWithoutFrom
+ | query UNION [ ALL ] query
+ | query EXCEPT query
+ | query INTERSECT query
+ }
+ [ ORDER BY orderItem [, orderItem ]* ]
+ [ LIMIT { count | ALL } ]
+ [ OFFSET start { ROW | ROWS } ]
+ [ FETCH { FIRST | NEXT } [ count ] { ROW | ROWS } ONLY]
+
+orderItem:
+ expression [ ASC | DESC ]
+
+select:
+ SELECT [ ALL | DISTINCT ]
+ { * | projectItem [, projectItem ]* }
+ FROM tableExpression
+ [ WHERE booleanExpression ]
+ [ GROUP BY { groupItem [, groupItem ]* } ]
+ [ HAVING booleanExpression ]
+
+selectWithoutFrom:
+ SELECT [ ALL | DISTINCT ]
+ { * | projectItem [, projectItem ]* }
+
+projectItem:
+ expression [ [ AS ] columnAlias ]
+ | tableAlias . *
+
+tableExpression:
+ tableReference [, tableReference ]*
+ | tableExpression [ NATURAL ] [ LEFT | RIGHT | FULL ] JOIN tableExpression [ joinCondition ]
+
+joinCondition:
+ ON booleanExpression
+ | USING '(' column [, column ]* ')'
+
+tableReference:
+ tablePrimary
+ [ [ AS ] alias [ '(' columnAlias [, columnAlias ]* ')' ] ]
+
+tablePrimary:
+ [ TABLE ] [ [ catalogName . ] schemaName . ] tableName
+ | LATERAL TABLE '(' functionName '(' expression [, expression ]* ')' ')'
+ | UNNEST '(' expression ')'
+
+values:
+ VALUES expression [, expression ]*
+
+groupItem:
+ expression
+ | '(' ')'
+ | '(' expression [, expression ]* ')'
+ | CUBE '(' expression [, expression ]* ')'
+ | ROLLUP '(' expression [, expression ]* ')'
+ | GROUPING SETS '(' groupItem [, groupItem ]* ')'
+```
+
+For a better definition of SQL queries within a Java String, Flink SQL uses a lexical policy similar to Java:
+
+- The case of identifiers is preserved whether or not they are quoted.
+- After which, identifiers are matched case-sensitively.
+- Unlike Java, back-ticks allow identifiers to contain non-alphanumeric characters (e.g. <code>"SELECT a AS `my field` FROM t"</code>).
+
+{% top %}
+
+Example Queries
+---------------
+
+**TODO: Add a examples for different operations with similar structure as for the Table API. Add highlighted tags if an operation is not supported by stream / batch.**
+
+* Scan & Values
+* Selection & Projection
+* Aggregations (distinct only Batch)
+ * GroupBy
+ * GroupBy Windows (TUMBLE, HOP, SESSION)
+ * OVER windows (Only Stream)
+ * Grouping sets, rollup, cube (only batch)
+ * Having (only batch?)
+* Joins
+ * Inner equi joins (only batch)
+ * Outer equi joins (only batch)
+ * TableFunction
+* Set operations (only batch, except Union ALL)
+* OrderBy + Limit + Offset
+
+{% top %}
+
+### GroupBy Windows
+
+**TODO: Integrate this with the examples**
+
+### Group Windows
+
+Group windows are defined in the `GROUP BY` clause of a SQL query. Just like queries with regular `GROUP BY` clauses, queries with a `GROUP BY` clause that includes a group window function compute a single result row per group. The following group windows functions are supported for SQL on batch and streaming tables.
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 30%">Group Window Function</th>
+ <th class="text-left">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td><code>TUMBLE(time_attr, interval)</code></td>
+ <td>Defines a tumbling time window. A tumbling time window assigns rows to non-overlapping, continuous windows with a fixed duration (<code>interval</code>). For example, a tumbling window of 5 minutes groups rows in 5 minutes intervals. Tumbling windows can be defined on event-time (stream + batch) or processing-time (stream).</td>
+ </tr>
+ <tr>
+ <td><code>HOP(time_attr, interval, interval)</code></td>
+ <td>Defines a hopping time window (called sliding window in the Table API). A hopping time window has a fixed duration (second <code>interval</code> parameter) and hops by a specified hop interval (first <code>interval</code> parameter). If the hop interval is smaller than the window size, hopping windows are overlapping. Thus, rows can be assigned to multiple windows. For example, a hopping window of 15 minutes size and 5 minute hop interval assigns each row to 3 different windows of 15 minute size, which are evaluated in an interval of 5 minutes. Hopping windows can be defined on event-time (stream + batch) or processing-time (stream).</td>
+ </tr>
+ <tr>
+ <td><code>SESSION(time_attr, interval)</code></td>
+ <td>Defines a session time window. Session time windows do not have a fixed duration but their bounds are defined by a time <code>interval</code> of inactivity, i.e., a session window is closed if no event appears for a defined gap period. For example a session window with a 30 minute gap starts when a row is observed after 30 minutes inactivity (otherwise the row would be added to an existing window) and is closed if no row is added within 30 minutes. Session windows can work on event-time (stream + batch) or processing-time (stream).</td>
+ </tr>
+ </tbody>
+</table>
+
+For SQL queries on streaming tables, the `time_attr` argument of the group window function must be one of the `rowtime()` or `proctime()` time-indicators, which distinguish between event or processing time, respectively. For SQL on batch tables, the `time_attr` argument of the group window function must be an attribute of type `TIMESTAMP`.
+
+#### Selecting Group Window Start and End Timestamps
+
+The start and end timestamps of group windows can be selected with the following auxiliary functions:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Auxiliary Function</th>
+ <th class="text-left">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ <code>TUMBLE_START(time_attr, interval)</code><br/>
+ <code>HOP_START(time_attr, interval, interval)</code><br/>
+ <code>SESSION_START(time_attr, interval)</code><br/>
+ </td>
+ <td>Returns the start timestamp of the corresponding tumbling, hopping, and session window.</td>
+ </tr>
+ <tr>
+ <td>
+ <code>TUMBLE_END(time_attr, interval)</code><br/>
+ <code>HOP_END(time_attr, interval, interval)</code><br/>
+ <code>SESSION_END(time_attr, interval)</code><br/>
+ </td>
+ <td>Returns the end timestamp of the corresponding tumbling, hopping, and session window.</td>
+ </tr>
+ </tbody>
+</table>
+
+Note that the auxiliary functions must be called with exactly same arguments as the group window function in the `GROUP BY` clause.
+
+The following examples show how to specify SQL queries with group windows on streaming tables.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// ingest a DataStream from an external source
+DataStream<Tuple3<Long, String, Integer>> ds = env.addSource(...);
+// register the DataStream as table "Orders"
+tableEnv.registerDataStream("Orders", ds, "user, product, amount");
+
+// compute SUM(amount) per day (in event-time)
+Table result1 = tableEnv.sql(
+ "SELECT user, " +
+ " TUMBLE_START(rowtime(), INTERVAL '1' DAY) as wStart, " +
+ " SUM(amount) FROM Orders " +
+ "GROUP BY TUMBLE(rowtime(), INTERVAL '1' DAY), user");
+
+// compute SUM(amount) per day (in processing-time)
+Table result2 = tableEnv.sql(
+ "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime(), INTERVAL '1' DAY), user");
+
+// compute every hour the SUM(amount) of the last 24 hours in event-time
+Table result3 = tableEnv.sql(
+ "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime(), INTERVAL '1' HOUR, INTERVAL '1' DAY), product");
+
+// compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
+Table result4 = tableEnv.sql(
+ "SELECT user, " +
+ " SESSION_START(rowtime(), INTERVAL '12' HOUR) AS sStart, " +
+ " SESSION_END(rowtime(), INTERVAL '12' HOUR) AS snd, " +
+ " SUM(amount) " +
+ "FROM Orders " +
+ "GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user");
+
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// read a DataStream from an external source
+val ds: DataStream[(Long, String, Int)] = env.addSource(...)
+// register the DataStream under the name "Orders"
+tableEnv.registerDataStream("Orders", ds, 'user, 'product, 'amount)
+
+// compute SUM(amount) per day (in event-time)
+val result1 = tableEnv.sql(
+ """
+ |SELECT
+ | user,
+ | TUMBLE_START(rowtime(), INTERVAL '1' DAY) as wStart,
+ | SUM(amount)
+ | FROM Orders
+ | GROUP BY TUMBLE(rowtime(), INTERVAL '1' DAY), user
+ """.stripMargin)
+
+// compute SUM(amount) per day (in processing-time)
+val result2 = tableEnv.sql(
+ "SELECT user, SUM(amount) FROM Orders GROUP BY TUMBLE(proctime(), INTERVAL '1' DAY), user")
+
+// compute every hour the SUM(amount) of the last 24 hours in event-time
+val result3 = tableEnv.sql(
+ "SELECT product, SUM(amount) FROM Orders GROUP BY HOP(rowtime(), INTERVAL '1' HOUR, INTERVAL '1' DAY), product")
+
+// compute SUM(amount) per session with 12 hour inactivity gap (in event-time)
+val result4 = tableEnv.sql(
+ """
+ |SELECT
+ | user,
+ | SESSION_START(rowtime(), INTERVAL '12' HOUR) AS sStart,
+ | SESSION_END(rowtime(), INTERVAL '12' HOUR) AS sEnd,
+ | SUM(amount)
+ | FROM Orders
+ | GROUP BY SESSION(rowtime(), INTERVAL '12' HOUR), user
+ """.stripMargin)
+
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+### Limitations
+
+**TODO: Integrate this with the examples**
+
+#### Batch
+
+The current version supports selection (filter), projection, inner equi-joins, grouping, aggregates, and sorting on batch tables.
+
+Among others, the following SQL features are not supported, yet:
+
+- Timestamps and intervals are limited to milliseconds precision
+- Interval arithmetic is currenly limited
+- Non-equi joins and Cartesian products
+- Efficient grouping sets
+
+*Note: Tables are joined in the order in which they are specified in the `FROM` clause. In some cases the table order must be manually tweaked to resolve Cartesian products.*
+
+#### Streaming
+
+Joins, set operations, and non-windowed aggregations are not supported yet.
+`UNNEST` supports only arrays and does not support `WITH ORDINALITY` yet.
+
+Data Types
+----------
+
+The SQL runtime is built on top of Flink's DataSet and DataStream APIs. Internally, it also uses Flink's `TypeInformation` to distinguish between types. The SQL support does not include all Flink types so far. All supported simple types are listed in `org.apache.flink.table.api.Types`. The following table summarizes the relation between SQL Types, Table API types, and the resulting Java class.
+
+| Table API | SQL | Java type |
+| :--------------------- | :-------------------------- | :--------------------- |
+| `Types.STRING` | `VARCHAR` | `java.lang.String` |
+| `Types.BOOLEAN` | `BOOLEAN` | `java.lang.Boolean` |
+| `Types.BYTE` | `TINYINT` | `java.lang.Byte` |
+| `Types.SHORT` | `SMALLINT` | `java.lang.Short` |
+| `Types.INT` | `INTEGER, INT` | `java.lang.Integer` |
+| `Types.LONG` | `BIGINT` | `java.lang.Long` |
+| `Types.FLOAT` | `REAL, FLOAT` | `java.lang.Float` |
+| `Types.DOUBLE` | `DOUBLE` | `java.lang.Double` |
+| `Types.DECIMAL` | `DECIMAL` | `java.math.BigDecimal` |
+| `Types.DATE` | `DATE` | `java.sql.Date` |
+| `Types.TIME` | `TIME` | `java.sql.Time` |
+| `Types.TIMESTAMP` | `TIMESTAMP(3)` | `java.sql.Timestamp` |
+| `Types.INTERVAL_MONTHS`| `INTERVAL YEAR TO MONTH` | `java.lang.Integer` |
+| `Types.INTERVAL_MILLIS`| `INTERVAL DAY TO SECOND(3)` | `java.lang.Long` |
+| `Types.PRIMITIVE_ARRAY`| `ARRAY` | e.g. `int[]` |
+| `Types.OBJECT_ARRAY` | `ARRAY` | e.g. `java.lang.Byte[]`|
+| `Types.MAP` | `MAP` | `java.util.HashMap` |
+
+
+Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and array types (object or primitive arrays) can be fields of a row.
+
+Generic types are treated as a black box within Table API and SQL yet.
+
+Composite types, however, are fully supported types where fields of a composite type can be accessed using the `.get()` operator in Table API and dot operator (e.g. `MyTable.pojoColumn.myField`) in SQL. Composite types can also be flattened using `.flatten()` in Table API or `MyTable.pojoColumn.*` in SQL.
+
+Array types can be accessed using the `myArray.at(1)` operator in Table API and `myArray[1]` operator in SQL. Array literals can be created using `array(1, 2, 3)` in Table API and `ARRAY[1, 2, 3]` in SQL.
+
+{% top %}
+
+Built-In Functions
+------------------
+
+Both the Table API and SQL come with a set of built-in functions for data transformations. This section gives a brief overview of the available functions so far.
+
+<!--
+This list of SQL functions should be kept in sync with SqlExpressionTest to reduce confusion due to the large amount of SQL functions.
+The documentation is split up and ordered like the tests in SqlExpressionTest.
+-->
+
+The Flink SQL functions (including their syntax) are a subset of Apache Calcite's built-in functions. Most of the documentation has been adopted from the [Calcite SQL reference](https://calcite.apache.org/docs/reference.html).
+
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Comparison functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+value1 = value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Equals.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 <> value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Not equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 > value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Greater than.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 >= value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Greater than or equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 < value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Less than.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 <= value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Less than or equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value IS NULL
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value</i> is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value IS NOT NULL
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value</i> is not null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 IS DISTINCT FROM value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if two values are not equal, treating null values as the same.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 IS NOT DISTINCT FROM value2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if two values are equal, treating null values as the same.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 BETWEEN [ASYMMETRIC | SYMMETRIC] value2 AND value3
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value1</i> is greater than or equal to <i>value2</i> and less than or equal to <i>value3</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value1 NOT BETWEEN value2 AND value3
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value1</i> is less than <i>value2</i> or greater than <i>value3</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+string1 LIKE string2 [ ESCAPE string3 ]
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>string1</i> matches pattern <i>string2</i>. An escape character can be defined if necessary.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+string1 NOT LIKE string2 [ ESCAPE string3 ]
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>string1</i> does not match pattern <i>string2</i>. An escape character can be defined if necessary.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+string1 SIMILAR TO string2 [ ESCAPE string3 ]
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>string1</i> matches regular expression <i>string2</i>. An escape character can be defined if necessary.</p>
+ </td>
+ </tr>
+
+
+ <tr>
+ <td>
+ {% highlight text %}
+string1 NOT SIMILAR TO string2 [ ESCAPE string3 ]
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>string1</i> does not match regular expression <i>string2</i>. An escape character can be defined if necessary.</p>
+ </td>
+ </tr>
+
+
+ <tr>
+ <td>
+ {% highlight text %}
+value IN (value [, value]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value</i> is equal to a value in a list.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value NOT IN (value [, value]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value</i> is not equal to every value in a list.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+EXISTS (sub-query)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>sub-query</i> returns at least one row. Only supported if the operation can be rewritten in a join and group operation.</p>
+ </td>
+ </tr>
+
+<!-- NOT SUPPORTED SO FAR
+ <tr>
+ <td>
+ {% highlight text %}
+value IN (sub-query)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value</i> is equal to a row returned by sub-query.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+value NOT IN (sub-query)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>value</i> is not equal to every row returned by sub-query.</p>
+ </td>
+ </tr>
+ -->
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Logical functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+boolean1 OR boolean2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean1</i> is TRUE or <i>boolean2</i> is TRUE. Supports three-valued logic.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean1 AND boolean2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean1</i> and <i>boolean2</i> are both TRUE. Supports three-valued logic.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+NOT boolean
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is not TRUE; returns UNKNOWN if <i>boolean</i> is UNKNOWN.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean IS FALSE
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is FALSE; returns FALSE if <i>boolean</i> is UNKNOWN.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean IS NOT FALSE
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is not FALSE; returns TRUE if <i>boolean</i> is UNKNOWN.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean IS TRUE
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is TRUE; returns FALSE if <i>boolean</i> is UNKNOWN.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean IS NOT TRUE
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is not TRUE; returns TRUE if <i>boolean</i> is UNKNOWN.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean IS UNKNOWN
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is UNKNOWN.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+boolean IS NOT UNKNOWN
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns TRUE if <i>boolean</i> is not UNKNOWN.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Arithmetic functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
++ numeric
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+- numeric
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns negative <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+numeric1 + numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> plus <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+numeric1 - numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> minus <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+numeric1 * numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> multiplied by <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+numeric1 / numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> divided by <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+POWER(numeric1, numeric2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> raised to the power of <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ABS(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the absolute value of <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+MOD(numeric1, numeric2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the remainder (modulus) of <i>numeric1</i> divided by <i>numeric2</i>. The result is negative only if <i>numeric1</i> is negative.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SQRT(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the square root of <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+LN(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the natural logarithm (base e) of <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+LOG10(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the base 10 logarithm of <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+EXP(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns e raised to the power of <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CEIL(numeric)
+CEILING(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds <i>numeric</i> up, and returns the smallest number that is greater than or equal to <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+FLOOR(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds <i>numeric</i> down, and returns the largest number that is less than or equal to <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SIN(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the sine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+COS(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the cosine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+TAN(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the tangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+COT(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the cotangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ASIN(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc sine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ACOS(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc cosine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ATAN(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc tangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+DEGREES(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts <i>numeric</i> from radians to degrees.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+RADIANS(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts <i>numeric</i> from degrees to radians.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SIGN(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the signum of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ROUND(numeric, int)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds the given number to <i>integer</i> places right to the decimal point.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+PI()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a value that is closer than any other value to pi.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">String functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+string || string
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Concatenates two character strings.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CHAR_LENGTH(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the number of characters in a character string.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CHARACTER_LENGTH(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>As CHAR_LENGTH(<i>string</i>).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+UPPER(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a character string converted to upper case.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+LOWER(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a character string converted to lower case.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+POSITION(string1 IN string2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the position of the first occurrence of <i>string1</i> in <i>string2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+TRIM( { BOTH | LEADING | TRAILING } string1 FROM string2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Removes leading and/or trailing characters from <i>string2</i>. By default, whitespaces at both sides are removed.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+OVERLAY(string1 PLACING string2 FROM integer [ FOR integer2 ])
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Replaces a substring of <i>string1</i> with <i>string2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SUBSTRING(string FROM integer)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a substring of a character string starting at a given point.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SUBSTRING(string FROM integer FOR integer)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a substring of a character string starting at a given point with a given length.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+INITCAP(string)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns string with the first letter of each word converter to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Conditional functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+CASE value
+WHEN value1 [, value11 ]* THEN result1
+[ WHEN valueN [, valueN1 ]* THEN resultN ]*
+[ ELSE resultZ ]
+END
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Simple case.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CASE
+WHEN condition1 THEN result1
+[ WHEN conditionN THEN resultN ]*
+[ ELSE resultZ ]
+END
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Searched case.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+NULLIF(value, value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns NULL if the values are the same. For example, <code>NULLIF(5, 5)</code> returns NULL; <code>NULLIF(5, 0)</code> returns 5.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+COALESCE(value, value [, value ]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Provides a value if the first value is null. For example, <code>COALESCE(NULL, 5)</code> returns 5.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Type conversion functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+CAST(value AS type)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts a value to a given type.</p>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Value constructor functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <!-- Disabled temporarily in favor of composite type support
+ <tr>
+ <td>
+ {% highlight text %}
+ROW (value [, value]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates a row from a list of values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+(value [, value]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates a row from a list of values.</p>
+ </td>
+ </tr>
+-->
+
+ <tr>
+ <td>
+ {% highlight text %}
+array ‘[’ index ‘]’
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the element at a particular position in an array. The index starts at 1.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ARRAY ‘[’ value [, value ]* ‘]’
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an array from a list of values.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Temporal functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+DATE string
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a date string in the form "yy-mm-dd" to a SQL date.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+TIME string
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a time <i>string</i> in the form "hh:mm:ss" to a SQL time.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+TIMESTAMP string
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a timestamp <i>string</i> in the form "yy-mm-dd hh:mm:ss.fff" to a SQL timestamp.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+INTERVAL string range
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses an interval <i>string</i> in the form "dd hh:mm:ss.fff" for SQL intervals of milliseconds or "yyyy-mm" for SQL intervals of months. An interval range might be e.g. <code>DAY</code>, <code>MINUTE</code>, <code>DAY TO HOUR</code>, or <code>DAY TO SECOND</code> for intervals of milliseconds; <code>YEAR</code> or <code>YEAR TO MONTH</code> for intervals of months. E.g. <code>INTERVAL '10 00:00:00.004' DAY TO SECOND</code>, <code>INTERVAL '10' DAY</code>, or <code>INTERVAL '2-10' YEAR TO MONTH</code> return intervals.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CURRENT_DATE
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL date in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CURRENT_TIME
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL time in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CURRENT_TIMESTAMP
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL timestamp in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+LOCALTIME
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL time in local time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+LOCALTIMESTAMP
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL timestamp in local time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+EXTRACT(timeintervalunit FROM temporal)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Extracts parts of a time point or time interval. Returns the part as a long value. E.g. <code>EXTRACT(DAY FROM DATE '2006-06-05')</code> leads to 5.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+FLOOR(timepoint TO timeintervalunit)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds a time point down to the given unit. E.g. <code>FLOOR(TIME '12:44:31' TO MINUTE)</code> leads to 12:44:00.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+CEIL(timepoint TO timeintervalunit)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds a time point up to the given unit. E.g. <code>CEIL(TIME '12:44:31' TO MINUTE)</code> leads to 12:45:00.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+QUARTER(date)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the quarter of a year from a SQL date. E.g. <code>QUARTER(DATE '1994-09-27')</code> leads to 3.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+(timepoint, temporal) OVERLAPS (timepoint, temporal)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Determines whether two anchored time intervals overlap. Time point and temporal are transformed into a range defined by two time points (start, end). The function evaluates <code>leftEnd >= rightStart && rightEnd >= leftStart</code>. E.g. <code>(TIME '2:55:00', INTERVAL '1' HOUR) OVERLAPS (TIME '3:30:00', INTERVAL '2' HOUR)</code> leads to true; <code>(TIME '9:00:00', TIME '10:00:00') OVERLAPS (TIME '10:15:00', INTERVAL '3' HOUR)</code> leads to false.</p>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Aggregate functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+COUNT(value [, value]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the number of input rows for which <i>value</i> is not null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+COUNT(*)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the number of input rows.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+AVG(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the average (arithmetic mean) of <i>numeric</i> across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+SUM(numeric)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sum of <i>numeric</i> across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+MAX(value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the maximum value of <i>value</i> across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+MIN(value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the minimum value of <i>value</i> across all input values.</p>
+ </td>
+ </tr>
+ <tr>
+ <td>
+ {% highlight text %}
+STDDEV_POP(value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the population standard deviation of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+<tr>
+ <td>
+ {% highlight text %}
+STDDEV_SAMP(value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sample standard deviation of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+VAR_POP(value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the population variance (square of the population standard deviation) of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+VAR_SAMP(value)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sample variance (square of the sample standard deviation) of the numeric field across all input values.</p>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Grouping functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+GROUP_ID()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns an integer that uniquely identifies the combination of grouping keys.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+GROUPING(expression)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns 1 if <i>expression</i> is rolled up in the current row’s grouping set, 0 otherwise.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+GROUPING_ID(expression [, expression]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a bit vector of the given grouping expressions.</p>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Value access functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+tableName.compositeType.field
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Accesses the field of a Flink composite type (such as Tuple, POJO, etc.) by name and returns it's value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+tableName.compositeType.*
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts a Flink composite type (such as Tuple, POJO, etc.) and all of its direct subtypes into a flat representation where every subtype is a separate field.</p>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Array functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td>
+ {% highlight text %}
+CARDINALITY(ARRAY)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the number of elements of an array.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight text %}
+ELEMENT(ARRAY)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sole element of an array with a single element. Returns <code>null</code> if the array is empty. Throws an exception if the array has more than one element.</p>
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+### Limitations
+
+The following operations are not supported yet:
+
+- Binary string operators and functions
+- System functions
+- Collection functions
+- Aggregate functions like STDDEV_xxx, VAR_xxx, and REGR_xxx
+- Distinct aggregate functions like COUNT DISTINCT
+
+{% top %}
+
+Reserved Keywords
+-----------------
+
+Although not every SQL feature is implemented yet, some string combinations are already reserved as keywords for future use. If you want to use one of the following strings as a field name, make sure to surround them with backticks (e.g. `` `value` ``, `` `count` ``).
+
+{% highlight sql %}
+
+A, ABS, ABSOLUTE, ACTION, ADA, ADD, ADMIN, AFTER, ALL, ALLOCATE, ALLOW, ALTER, ALWAYS, AND, ANY, ARE, ARRAY, AS, ASC, ASENSITIVE, ASSERTION, ASSIGNMENT, ASYMMETRIC, AT, ATOMIC, ATTRIBUTE, ATTRIBUTES, AUTHORIZATION, AVG, BEFORE, BEGIN, BERNOULLI, BETWEEN, BIGINT, BINARY, BIT, BLOB, BOOLEAN, BOTH, BREADTH, BY, C, CALL, CALLED, CARDINALITY, CASCADE, CASCADED, CASE, CAST, CATALOG, CATALOG_NAME, CEIL, CEILING, CENTURY, CHAIN, CHAR, CHARACTER, CHARACTERISTICTS, CHARACTERS, CHARACTER_LENGTH, CHARACTER_SET_CATALOG, CHARACTER_SET_NAME, CHARACTER_SET_SCHEMA, CHAR_LENGTH, CHECK, CLASS_ORIGIN, CLOB, CLOSE, COALESCE, COBOL, COLLATE, COLLATION, COLLATION_CATALOG, COLLATION_NAME, COLLATION_SCHEMA, COLLECT, COLUMN, COLUMN_NAME, COMMAND_FUNCTION, COMMAND_FUNCTION_CODE, COMMIT, COMMITTED, CONDITION, CONDITION_NUMBER, CONNECT, CONNECTION, CONNECTION_NAME, CONSTRAINT, CONSTRAINTS, CONSTRAINT_CATALOG, CONSTRAINT_NAME, CONSTRAINT_SCHEMA, CONSTRUCTOR, CONTAINS, CONTINUE, CONVERT, CORR, CORRESPONDING, COUN
T, COVAR_POP, COVAR_SAMP, CREATE, CROSS, CUBE, CUME_DIST, CURRENT, CURRENT_CATALOG, CURRENT_DATE, CURRENT_DEFAULT_TRANSFORM_GROUP, CURRENT_PATH, CURRENT_ROLE, CURRENT_SCHEMA, CURRENT_TIME, CURRENT_TIMESTAMP, CURRENT_TRANSFORM_GROUP_FOR_TYPE, CURRENT_USER, CURSOR, CURSOR_NAME, CYCLE, DATA, DATABASE, DATE, DATETIME_INTERVAL_CODE, DATETIME_INTERVAL_PRECISION, DAY, DEALLOCATE, DEC, DECADE, DECIMAL, DECLARE, DEFAULT, DEFAULTS, DEFERRABLE, DEFERRED, DEFINED, DEFINER, DEGREE, DELETE, DENSE_RANK, DEPTH, DEREF, DERIVED, DESC, DESCRIBE, DESCRIPTION, DESCRIPTOR, DETERMINISTIC, DIAGNOSTICS, DISALLOW, DISCONNECT, DISPATCH, DISTINCT, DOMAIN, DOUBLE, DOW, DOY, DROP, DYNAMIC, DYNAMIC_FUNCTION, DYNAMIC_FUNCTION_CODE, EACH, ELEMENT, ELSE, END, END-EXEC, EPOCH, EQUALS, ESCAPE, EVERY, EXCEPT, EXCEPTION, EXCLUDE, EXCLUDING, EXEC, EXECUTE, EXISTS, EXP, EXPLAIN, EXTEND, EXTERNAL, EXTRACT, FALSE, FETCH, FILTER, FINAL, FIRST, FIRST_VALUE, FLOAT, FLOOR, FOLLOWING, FOR, FOREIGN, FORTRAN, FOUND, FRAC_SECOND, F
REE, FROM, FULL, FUNCTION, FUSION, G, GENERAL, GENERATED, GET, GLOBAL, GO, GOTO, GRANT, GRANTED, GROUP, GROUPING, HAVING, HIERARCHY, HOLD, HOUR, IDENTITY, IMMEDIATE, IMPLEMENTATION, IMPORT, IN, INCLUDING, INCREMENT, INDICATOR, INITIALLY, INNER, INOUT, INPUT, INSENSITIVE, INSERT, INSTANCE, INSTANTIABLE, INT, INTEGER, INTERSECT, INTERSECTION, INTERVAL, INTO, INVOKER, IS, ISOLATION, JAVA, JOIN, K, KEY, KEY_MEMBER, KEY_TYPE, LABEL, LANGUAGE, LARGE, LAST, LAST_VALUE, LATERAL, LEADING, LEFT, LENGTH, LEVEL, LIBRARY, LIKE, LIMIT, LN, LOCAL, LOCALTIME, LOCALTIMESTAMP, LOCATOR, LOWER, M, MAP, MATCH, MATCHED, MAX, MAXVALUE, MEMBER, MERGE, MESSAGE_LENGTH, MESSAGE_OCTET_LENGTH, MESSAGE_TEXT, METHOD, MICROSECOND, MILLENNIUM, MIN, MINUTE, MINVALUE, MOD, MODIFIES, MODULE, MONTH, MORE, MULTISET, MUMPS, NAME, NAMES, NATIONAL, NATURAL, NCHAR, NCLOB, NESTING, NEW, NEXT, NO, NONE, NORMALIZE, NORMALIZED, NOT, NULL, NULLABLE, NULLIF, NULLS, NUMBER, NUMERIC, OBJECT, OCTETS, OCTET_LENGTH, OF, OFFSET, OLD, O
N, ONLY, OPEN, OPTION, OPTIONS, OR, ORDER, ORDERING, ORDINALITY, OTHERS, OUT, OUTER, OUTPUT, OVER, OVERLAPS, OVERLAY, OVERRIDING, PAD, PARAMETER, PARAMETER_MODE, PARAMETER_NAME, PARAMETER_ORDINAL_POSITION, PARAMETER_SPECIFIC_CATALOG, PARAMETER_SPECIFIC_NAME, PARAMETER_SPECIFIC_SCHEMA, PARTIAL, PARTITION, PASCAL, PASSTHROUGH, PATH, PERCENTILE_CONT, PERCENTILE_DISC, PERCENT_RANK, PLACING, PLAN, PLI, POSITION, POWER, PRECEDING, PRECISION, PREPARE, PRESERVE, PRIMARY, PRIOR, PRIVILEGES, PROCEDURE, PUBLIC, QUARTER, RANGE, RANK, READ, READS, REAL, RECURSIVE, REF, REFERENCES, REFERENCING, REGR_AVGX, REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE, REGR_SXX, REGR_SXY, REGR_SYY, RELATIVE, RELEASE, REPEATABLE, RESET, RESTART, RESTRICT, RESULT, RETURN, RETURNED_CARDINALITY, RETURNED_LENGTH, RETURNED_OCTET_LENGTH, RETURNED_SQLSTATE, RETURNS, REVOKE, RIGHT, ROLE, ROLLBACK, ROLLUP, ROUTINE, ROUTINE_CATALOG, ROUTINE_NAME, ROUTINE_SCHEMA, ROW, ROWS, ROW_COUNT, ROW_NUMBER, SAVEPOINT, SCALE
, SCHEMA, SCHEMA_NAME, SCOPE, SCOPE_CATALOGS, SCOPE_NAME, SCOPE_SCHEMA, SCROLL, SEARCH, SECOND, SECTION, SECURITY, SELECT, SELF, SENSITIVE, SEQUENCE, SERIALIZABLE, SERVER, SERVER_NAME, SESSION, SESSION_USER, SET, SETS, SIMILAR, SIMPLE, SIZE, SMALLINT, SOME, SOURCE, SPACE, SPECIFIC, SPECIFICTYPE, SPECIFIC_NAME, SQL, SQLEXCEPTION, SQLSTATE, SQLWARNING, SQL_TSI_DAY, SQL_TSI_FRAC_SECOND, SQL_TSI_HOUR, SQL_TSI_MICROSECOND, SQL_TSI_MINUTE, SQL_TSI_MONTH, SQL_TSI_QUARTER, SQL_TSI_SECOND, SQL_TSI_WEEK, SQL_TSI_YEAR, SQRT, START, STATE, STATEMENT, STATIC, STDDEV_POP, STDDEV_SAMP, STREAM, STRUCTURE, STYLE, SUBCLASS_ORIGIN, SUBMULTISET, SUBSTITUTE, SUBSTRING, SUM, SYMMETRIC, SYSTEM, SYSTEM_USER, TABLE, TABLESAMPLE, TABLE_NAME, TEMPORARY, THEN, TIES, TIME, TIMESTAMP, TIMESTAMPADD, TIMESTAMPDIFF, TIMEZONE_HOUR, TIMEZONE_MINUTE, TINYINT, TO, TOP_LEVEL_COUNT, TRAILING, TRANSACTION, TRANSACTIONS_ACTIVE, TRANSACTIONS_COMMITTED, TRANSACTIONS_ROLLED_BACK, TRANSFORM, TRANSFORMS, TRANSLATE, TRANSLATION,
TREAT, TRIGGER, TRIGGER_CATALOG, TRIGGER_NAME, TRIGGER_SCHEMA, TRIM, TRUE, TYPE, UESCAPE, UNBOUNDED, UNCOMMITTED, UNDER, UNION, UNIQUE, UNKNOWN, UNNAMED, UNNEST, UPDATE, UPPER, UPSERT, USAGE, USER, USER_DEFINED_TYPE_CATALOG, USER_DEFINED_TYPE_CODE, USER_DEFINED_TYPE_NAME, USER_DEFINED_TYPE_SCHEMA, USING, VALUE, VALUES, VARBINARY, VARCHAR, VARYING, VAR_POP, VAR_SAMP, VERSION, VIEW, WEEK, WHEN, WHENEVER, WHERE, WIDTH_BUCKET, WINDOW, WITH, WITHIN, WITHOUT, WORK, WRAPPER, WRITE, XML, YEAR, ZONE
+
+{% endhighlight %}
+
+{% top %}
+
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table/streaming.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/streaming.md b/docs/dev/table/streaming.md
new file mode 100644
index 0000000..9699507
--- /dev/null
+++ b/docs/dev/table/streaming.md
@@ -0,0 +1,67 @@
+---
+title: "Streaming Concepts"
+nav-parent_id: tableapi
+nav-pos: 10
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+**TODO: has to be completely written**
+
+* This will be replaced by the TOC
+{:toc}
+
+Dynamic table
+-------------
+
+* Stream -> Table
+* Table -> Stream
+* update changes / retraction
+
+{% top %}
+
+Time Attributes
+---------------
+
+### Event-time
+
+* DataStream: Timestamps & WMs required, `.rowtime` (replace attribute or extend schema)
+* TableSource: Timestamps & WMs & DefinedRowtimeAttribute
+
+{% top %}
+
+### Processing time
+
+* DataStream: `.proctime` (only extend schema)
+* TableSource: DefinedProctimeAttribute
+
+{% top %}
+
+Query Configuration
+-------------------
+
+In stream processing, compuations are constantly happening and there are many use cases that require to update previously emitted results. There are many ways in which a query can compute and emit updates. These do not affect the semantics of the query but might lead to approximated results.
+
+Flink's Table API and SQL interface use a `QueryConfig` to control the computation and emission of results and updates.
+
+### State Retention
+
+{% top %}
+
+
[08/10] flink git commit: [FLINK-6745] [table] [docs] Updated Table
API / SQL docs: Overview
Posted by fh...@apache.org.
[FLINK-6745] [table] [docs] Updated Table API / SQL docs: Overview
This closes #4013.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/4088409f
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/4088409f
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/4088409f
Branch: refs/heads/master
Commit: 4088409f21fa648143abbcb50c129aa63266679e
Parents: a29a9ad
Author: Fabian Hueske <fh...@apache.org>
Authored: Fri May 26 14:47:35 2017 +0100
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:42:19 2017 +0200
----------------------------------------------------------------------
docs/dev/tableApi.md | 59 ++++++++++++++++++++++++-----------------------
1 file changed, 30 insertions(+), 29 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/4088409f/docs/dev/tableApi.md
----------------------------------------------------------------------
diff --git a/docs/dev/tableApi.md b/docs/dev/tableApi.md
index c1c02f1..f7b13f0 100644
--- a/docs/dev/tableApi.md
+++ b/docs/dev/tableApi.md
@@ -25,32 +25,16 @@ specific language governing permissions and limitations
under the License.
-->
-**Table API and SQL are experimental features**
+Apache Flink features two relational APIs - the Table API and SQL - for unified stream and batch processing. The Table API is a language-integrated query API for Scala and Java that allows the composition of queries from relational operators such as selection, filter, and join in a very intuitive way. Flink's SQL support is based on [Apache Calcite](https://calcite.apache.org) which implements the SQL standard. Queries specified in either interface have the same semantics and specify the same result regardless whether the input is a batch input (DataSet) or a stream input (DataStream).
-The Table API is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flink's DataSet and DataStream APIs (Java and Scala).
-The Table API and SQL interface operate on a relational `Table` abstraction, which can be created from external data sources, or existing DataSets and DataStreams. With the Table API, you can apply relational operators such as selection, aggregation, and joins on `Table`s.
+The Table API and the SQL interfaces are tightly integrated with each other as well as Flink's DataStream and DataSet APIs. You can easily switch between all APIs and libraries which build upon the APIs. For instance, you can extract patterns from a DataStream using the [CEP library]({{ site.baseurl }}/dev/libs/cep.html) and later use the Table API to analyze the patterns, or you might scan, filter, and aggregate a batch table using a SQL query before running a [Gelly graph algorithm]({{ site.baseurl }}/dev/libs/gelly) on the preprocessed data.
-`Table`s can also be queried with regular SQL, as long as they are registered (see [Registering Tables](#registering-tables)). The Table API and SQL offer equivalent functionality and can be mixed in the same program. When a `Table` is converted back into a `DataSet` or `DataStream`, the logical plan, which was defined by relational operators and SQL queries, is optimized using [Apache Calcite](https://calcite.apache.org/) and transformed into a `DataSet` or `DataStream` program.
-
-**TODO: Check, update, and add**
-
-* What are the Table API / SQL
- * Relational APIs
- * Unified APIs for batch and streaming
- * Semantics are the same
- * But not all operations can be efficiently mapped to streams
- * Table API: language-integrated queries (LINQ) in Scala and Java
- * SQL: Standard SQL
-
-**Please notice: Not all operations are supported by all four combinations of Stream/Batch and TableAPI/SQL.**
-
-* This will be replaced by the TOC
-{:toc}
+**Please note that the Table API and SQL are not yet feature complete and are being active developed. Not all operations are supported by every combination of \[Table API, SQL\] and \[stream, batch\] input.**
Setup
-----
-The Table API and SQL are part of the *flink-table* Maven project.
+The Table API and SQL are bundled in the `flink-table` Maven artifact.
The following dependency must be added to your project in order to use the Table API and SQL:
{% highlight xml %}
@@ -61,21 +45,38 @@ The following dependency must be added to your project in order to use the Table
</dependency>
{% endhighlight %}
-*Note: The Table API is currently not part of the binary distribution. See linking with it for cluster execution [here]({{ site.baseurl }}/dev/linking.html).*
+In addition, you need to add a dependency for either Flink's Scala batch or streaming API. For a batch query you need to add:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
+
+For a streaming query you need to add:
+
+{% highlight xml %}
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-scala{{ site.scala_version_suffix }}</artifactId>
+ <version>{{site.version }}</version>
+</dependency>
+{% endhighlight %}
-**TODO: Rework and add:**
-* Project dependencies (flink-table + flink-scala or flink-streaming-scala)
-* Copy `./opt/flink-table.jar` to `./lib`
+**Note:** Due to an issue in Apache Calcite, which prevents the user classloaders from being garbage-collected, we do *not* recommend building a fat-jar that includes the `flink-table` dependency. Instead, we recommend configuring Flink to include the `flink-table` dependency in the system classloader. This can be done by copying the `flink-table.jar` file from the `./opt` folder to the `./lib` folder. See [these instructions]({{ site.baseurl }}/dev/linking.html) for further details.
{% top %}
Where to go next?
-----------------
-* [Concepts & Common API]({{ site.baseurl }}/dev/table/common.html): Share concepts and API of the Table API and SQL.
-* [Table API]({{ site.baseurl }}/dev/table/tableapi.html): Supported Operations and API for the Table API
-* [SQL]({{ site.baseurl }}/dev/table/sql.html): Supported Operations and Syntax for SQL
-* [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html): Ingestion and emission of tables.
-* [User-Defined Functions]({{ site.baseurl }}/dev/table/udfs.html): Defintion and usage of user-defined functions.
+* [Concepts & Common API]({{ site.baseurl }}/dev/table/common.html): Shared concepts and APIs of the Table API and SQL.
+* [Streaming Table API & SQL]({{ site.baseurl }}/dev/table/streaming.html): Streaming-specific documentation for the Table API or SQL such as configuration of time attributes and handling of updating results.
+* [Table API]({{ site.baseurl }}/dev/table/tableapi.html): Supported operations and API for the Table API.
+* [SQL]({{ site.baseurl }}/dev/table/sql.html): Supported operations and syntax for SQL
+* [Table Sources & Sinks]({{ site.baseurl }}/dev/table/sourceSinks.html): Reading tables from and emitting tables to external storage systems.
+* [User-Defined Functions]({{ site.baseurl }}/dev/table/udfs.html): Definition and usage of user-defined functions.
{% top %}
\ No newline at end of file
[03/10] flink git commit: [FLNK-5354] [docs] Restructured Table API /
SQL docs
Posted by fh...@apache.org.
http://git-wip-us.apache.org/repos/asf/flink/blob/a5d93a56/docs/dev/table/tableApi.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md
new file mode 100644
index 0000000..25810d2
--- /dev/null
+++ b/docs/dev/table/tableApi.md
@@ -0,0 +1,3559 @@
+---
+title: "Table API"
+nav-parent_id: tableapi
+nav-pos: 20
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The Table API is a language-integrated relational API for Scala and Java. The Table API is a unified API for stream and batch processing.
+
+Please have a look at the [Common Concepts & API]({{ site.baseurl }}/dev/table/common.html) and the [Streaming Concepts]({{ site.baseurl }}/dev/table/streaming.html) if you work with streaming data.
+
+The following examples assume a registered table called `Orders` with attributes `a, b, c, rowtime`.
+
+**TODO: Extend**
+
+* This will be replaced by the TOC
+{:toc}
+
+Table API Overview
+------------------
+
+The Table API is available for Scala and Java. The Scala Table API is based on Scala Expressions, the Java Table API on Strings which are parsed and converted into Expressions.
+
+The following example shows the differences between the Scala and Java Table API.
+
+**TODO: Extend**
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+The Java Table API is enabled by importing `org.apache.flink.table.api.java.*`. The following example shows how a Java Table API program is constructed.
+
+{% highlight java %}
+
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env);
+
+Table orders = tEnv.scan("Orders"); // schema (a, b, c, rowtime)
+
+Table counts = orders
+ .groupBy("a")
+ .select("a, b.count as cnt");
+
+DataSet<Row> result = tableEnv.toDataSet(wordCounts, Row.class);
+{% endhighlight %}
+
+With Java, expressions must be specified by Strings. The embedded expression DSL is not supported.
+
+{% highlight java %}
+ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
+BatchTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// register the DataSet cust as table "Customers" with fields derived from the dataset
+tableEnv.registerDataSet("Customers", cust)
+
+// register the DataSet ord as table "Orders" with fields user, product, and amount
+tableEnv.registerDataSet("Orders", ord, "user, product, amount");
+{% endhighlight %}
+
+</div>
+
+<div data-lang="scala" markdown="1">
+
+The Scala Table API is enabled by importing `org.apache.flink.table.api.scala._`. The following example shows how a Scala Table API program is constructed.
+
+{% highlight scala %}
+import org.apache.flink.api.scala._
+import org.apache.flink.table.api.scala._
+
+val env = ExecutionEnvironment.getExecutionEnvironment
+val tEnv = TableEnvironment.getTableEnvironment(env)
+
+val orders = tEnv.scan("Orders") // schema (a, b, c, rowtime)
+val result = orders
+ .groupBy('a)
+ .select('a, 'b.count as 'cnt)
+ .toDataSet[Row]
+{% endhighlight %}
+
+The expression DSL uses Scala symbols to refer to field names and code generation to
+transform expressions to efficient runtime code. Please note that the conversion to and from
+Tables only works when using Scala case classes or Java POJOs. Please refer to the [Type Extraction and Serialization]({{ site.baseurl }}/internals/types_serialization.html) section
+to learn the characteristics of a valid POJO.
+
+</div>
+</div>
+
+**TODO**
+
+{% top %}
+
+Operations
+----------
+
+**TODO: Add Tags for Batch and Streaming support**
+
+### Scan, Projection, and Filter
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Scan</strong></td>
+ <td>
+ <p>Similar to the FROM clause in a SQL query. Performs a scan of a registered table.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>Select</strong></td>
+ <td>
+ <p>Similar to a SQL SELECT statement. Performs a select operation.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders.select("a, c as d");
+{% endhighlight %}
+ <p>You can use star (<code>*</code>) to act as a wild card, selecting all of the columns in the table.</p>
+{% highlight java %}
+Table result = orders.select("*");
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>As</strong></td>
+ <td>
+ <p>Renames fields.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders.as("x, y, z, t");
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Where / Filter</strong></td>
+ <td>
+ <p>Similar to a SQL WHERE clause. Filters out rows that do not pass the filter predicate.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders.where("b = 'red'");
+{% endhighlight %}
+or
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders.filter("a % 2 = 0");
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+</div>
+<div data-lang="scala" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Scan</strong></td>
+ <td>
+ <p>Similar to the FROM clause in a SQL query. Performs a scan of a registered table.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>Select</strong></td>
+ <td>
+ <p>Similar to a SQL SELECT statement. Performs a select operation.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result = orders.select('a, 'c as 'd)
+{% endhighlight %}
+ <p>You can use star (<code>*</code>) to act as a wild card, selecting all of the columns in the table.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result = orders.select('*)
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>As</strong></td>
+ <td>
+ <p>Renames fields.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders").as('x, 'y, 'z, 't')
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Where / Filter</strong></td>
+ <td>
+ <p>Similar to a SQL WHERE clause. Filters out rows that do not pass the filter predicate.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result = orders.filter('a % 2 === 0)
+{% endhighlight %}
+or
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result = orders.where('b === "red")
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+</div>
+</div>
+
+{% top %}
+
+### Aggregations
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>GroupBy</strong></td>
+ <td>
+ <p>Similar to a SQL GROUPBY clause. Groups the rows on the grouping keys, with a following aggregation
+ operator to aggregate rows group-wise.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders.groupBy("a").select("a, b.sum as d");
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>GroupBy Window</strong></td>
+ <td>TODO</td>
+ </tr>
+ <tr>
+ <td><strong>Over Window</strong></td>
+ <td>TODO</td>
+ </tr>
+ <tr>
+ <td><strong>Distinct</strong></td>
+ <td>
+ <p>Similar to a SQL DISTINCT clause. Returns records with distinct value combinations.</p>
+{% highlight java %}
+Table orders = tableEnv.scan("Orders");
+Table result = orders.distinct();
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+</div>
+<div data-lang="scala" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+
+ <tr>
+ <td><strong>GroupBy</strong></td>
+ <td>
+ <p>Similar to a SQL GROUPBY clause. Groups rows on the grouping keys, with a following aggregation
+ operator to aggregate rows group-wise.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result = orders.groupBy('a).select('a, 'b.sum as 'd)
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>GroupBy Window</strong></td>
+ <td>TODO</td>
+ </tr>
+ <tr>
+ <td><strong>Over Window</strong></td>
+ <td>TODO</td>
+ </tr>
+ <tr>
+ <td><strong>Distinct</strong></td>
+ <td>
+ <p>Similar to a SQL DISTINCT clause. Returns records with distinct value combinations.</p>
+{% highlight scala %}
+val orders: Table = tableEnv.scan("Orders")
+val result = orders.distinct()
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+</div>
+</div>
+
+{% top %}
+
+### Joins
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Inner Join</strong></td>
+ <td>
+ <p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined through join operator or using a where or filter operator.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.join(right).where("a = d").select("a, b, e");
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>LeftOuterJoin</strong></td>
+ <td>
+ <p>Similar to a SQL LEFT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.leftOuterJoin(right, "a = d").select("a, b, e");
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>RightOuterJoin</strong></td>
+ <td>
+ <p>Similar to a SQL RIGHT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.rightOuterJoin(right, "a = d").select("a, b, e");
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>FullOuterJoin</strong></td>
+ <td>
+ <p>Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.fullOuterJoin(right, "a = d").select("a, b, e");
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>TableFunction CrossJoin</strong></td>
+ <td>TODO</td>
+ </tr>
+ <tr>
+ <td><strong>TableFunction LeftOuterJoin</strong></td>
+ <td>TODO</td>
+ </tr>
+
+ </tbody>
+</table>
+
+</div>
+<div data-lang="scala" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+
+ <tr>
+ <td><strong>Join</strong></td>
+ <td>
+ <p>Similar to a SQL JOIN clause. Joins two tables. Both tables must have distinct field names and an equality join predicate must be defined using a where or filter operator.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'd, 'e, 'f);
+val result = left.join(right).where('a === 'd).select('a, 'b, 'e);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>LeftOuterJoin</strong></td>
+ <td>
+ <p>Similar to a SQL LEFT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.leftOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>RightOuterJoin</strong></td>
+ <td>
+ <p>Similar to a SQL RIGHT OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.rightOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>FullOuterJoin</strong></td>
+ <td>
+ <p>Similar to a SQL FULL OUTER JOIN clause. Joins two tables. Both tables must have distinct field names and at least one equality join predicate must be defined.</p>
+{% highlight scala %}
+val left = tableEnv.fromDataSet(ds1, 'a, 'b, 'c)
+val right = tableEnv.fromDataSet(ds2, 'd, 'e, 'f)
+val result = left.fullOuterJoin(right, 'a === 'd).select('a, 'b, 'e)
+{% endhighlight %}
+ </td>
+ </tr>
+ <tr>
+ <td><strong>TableFunction CrossJoin</strong></td>
+ <td>TODO</td>
+ </tr>
+ <tr>
+ <td><strong>TableFunction LeftOuterJoin</strong></td>
+ <td>TODO</td>
+ </tr>
+
+ </tbody>
+</table>
+</div>
+</div>
+
+{% top %}
+
+### Set Operations
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Union</strong></td>
+ <td>
+ <p>Similar to a SQL UNION clause. Unions two tables with duplicate records removed. Both tables must have identical field types.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "a, b, c");
+Table result = left.union(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>UnionAll</strong></td>
+ <td>
+ <p>Similar to a SQL UNION ALL clause. Unions two tables. Both tables must have identical field types.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "a, b, c");
+Table result = left.unionAll(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Intersect</strong></td>
+ <td>
+ <p>Similar to a SQL INTERSECT clause. Intersect returns records that exist in both tables. If a record is present one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Both tables must have identical field types.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.intersect(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>IntersectAll</strong></td>
+ <td>
+ <p>Similar to a SQL INTERSECT ALL clause. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Both tables must have identical field types.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "d, e, f");
+Table result = left.intersectAll(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Minus</strong></td>
+ <td>
+ <p>Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "a, b, c");
+Table result = left.minus(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>MinusAll</strong></td>
+ <td>
+ <p>Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.</p>
+{% highlight java %}
+Table left = tableEnv.fromDataSet(ds1, "a, b, c");
+Table right = tableEnv.fromDataSet(ds2, "a, b, c");
+Table result = left.minusAll(right);
+{% endhighlight %}
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+</div>
+<div data-lang="scala" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Union</strong></td>
+ <td>
+ <p>Similar to a SQL UNION clause. Unions two tables with duplicate records removed, both tables must have identical field types.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
+val result = left.union(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>UnionAll</strong></td>
+ <td>
+ <p>Similar to a SQL UNION ALL clause. Unions two tables, both tables must have identical field types.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
+val result = left.unionAll(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Intersect</strong></td>
+ <td>
+ <p>Similar to a SQL INTERSECT clause. Intersect returns records that exist in both tables. If a record is present in one or both tables more than once, it is returned just once, i.e., the resulting table has no duplicate records. Both tables must have identical field types.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'e, 'f, 'g);
+val result = left.intersect(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>IntersectAll</strong></td>
+ <td>
+ <p>Similar to a SQL INTERSECT ALL clause. IntersectAll returns records that exist in both tables. If a record is present in both tables more than once, it is returned as many times as it is present in both tables, i.e., the resulting table might have duplicate records. Both tables must have identical field types.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'e, 'f, 'g);
+val result = left.intersectAll(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Minus</strong></td>
+ <td>
+ <p>Similar to a SQL EXCEPT clause. Minus returns records from the left table that do not exist in the right table. Duplicate records in the left table are returned exactly once, i.e., duplicates are removed. Both tables must have identical field types.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
+val result = left.minus(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>MinusAll</strong></td>
+ <td>
+ <p>Similar to a SQL EXCEPT ALL clause. MinusAll returns the records that do not exist in the right table. A record that is present n times in the left table and m times in the right table is returned (n - m) times, i.e., as many duplicates as are present in the right table are removed. Both tables must have identical field types.</p>
+{% highlight scala %}
+val left = ds1.toTable(tableEnv, 'a, 'b, 'c);
+val right = ds2.toTable(tableEnv, 'a, 'b, 'c);
+val result = left.minusAll(right);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+</div>
+</div>
+
+{% top %}
+
+### OrderBy & Limit
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Order By</strong></td>
+ <td>
+ <p>Similar to a SQL ORDER BY clause. Returns records globally sorted across all parallel partitions.</p>
+{% highlight scala %}
+val in = ds.toTable(tableEnv, 'a, 'b, 'c);
+val result = in.orderBy('a.asc);
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Limit</strong></td>
+ <td>
+ <p>Similar to a SQL LIMIT clause. Limits a sorted result to a specified number of records from an offset position. Limit is technically part of the Order By operator and thus must be preceded by it.</p>
+{% highlight scala %}
+val in = ds.toTable(tableEnv, 'a, 'b, 'c);
+val result = in.orderBy('a.asc).limit(3); // returns unlimited number of records beginning with the 4th record
+{% endhighlight %}
+or
+{% highlight scala %}
+val in = ds.toTable(tableEnv, 'a, 'b, 'c);
+val result = in.orderBy('a.asc).limit(3, 5); // returns 5 records beginning with the 4th record
+{% endhighlight %}
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+</div>
+<div data-lang="scala" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Operators</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><strong>Order By</strong></td>
+ <td>
+ <p>Similar to a SQL ORDER BY clause. Returns records globally sorted across all parallel partitions.</p>
+{% highlight java %}
+Table in = tableEnv.fromDataSet(ds, "a, b, c");
+Table result = in.orderBy("a.asc");
+{% endhighlight %}
+ </td>
+ </tr>
+
+ <tr>
+ <td><strong>Limit</strong></td>
+ <td>
+ <p>Similar to a SQL LIMIT clause. Limits a sorted result to a specified number of records from an offset position. Limit is technically part of the Order By operator and thus must be preceded by it.</p>
+{% highlight java %}
+Table in = tableEnv.fromDataSet(ds, "a, b, c");
+Table result = in.orderBy("a.asc").limit(3); // returns unlimited number of records beginning with the 4th record
+{% endhighlight %}
+or
+{% highlight java %}
+Table in = tableEnv.fromDataSet(ds, "a, b, c");
+Table result = in.orderBy("a.asc").limit(3, 5); // returns 5 records beginning with the 4th record
+{% endhighlight %}
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+</div>
+</div>
+
+### Windows
+
+**TODO: Figure out where to put this stuff. I think it would be good to have it in the "Operations" section for a brief overview. A more detailed discussion of windows should go somewhere else, maybe into the "Common Concepts"?**
+
+The Table API is a declarative API to define queries on batch and streaming tables. Projection, selection, and union operations can be applied both on streaming and batch tables without additional semantics. Aggregations on (possibly) infinite streaming tables, however, can only be computed on finite groups of records. Window aggregates group rows into finite groups based on time or row-count intervals and evaluate aggregation functions once per group. For batch tables, windows are a convenient shortcut to group records by time intervals.
+
+Windows are defined using the `window(w: Window)` clause and require an alias, which is specified using the `as` clause. In order to group a table by a window, the window alias must be referenced in the `groupBy(...)` clause like a regular grouping attribute.
+The following example shows how to define a window aggregation on a table.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+Table table = input
+ .window([Window w].as("w")) // define window with alias w
+ .groupBy("w") // group the table by window w
+ .select("b.sum"); // aggregate
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val table = input
+ .window([w: Window] as 'w) // define window with alias w
+ .groupBy('w) // group the table by window w
+ .select('b.sum) // aggregate
+{% endhighlight %}
+</div>
+</div>
+
+In streaming environments, window aggregates can only be computed in parallel if they group on one or more attributes in addition to the window, i.e., the `groupBy(...)` clause references a window alias and at least one additional attribute. A `groupBy(...)` clause that only references a window alias (such as in the example above) can only be evaluated by a single, non-parallel task.
+The following example shows how to define a window aggregation with additional grouping attributes.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+Table table = input
+ .window([Window w].as("w")) // define window with alias w
+ .groupBy("w, a") // group the table by attribute a and window w
+ .select("a, b.sum"); // aggregate
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val table = input
+ .window([w: Window] as 'w) // define window with alias w
+ .groupBy('w, 'a) // group the table by attribute a and window w
+ .select('a, 'b.sum) // aggregate
+{% endhighlight %}
+</div>
+</div>
+
+The `Window` parameter defines how rows are mapped to windows. `Window` is not an interface that users can implement. Instead, the Table API provides a set of predefined `Window` classes with specific semantics, which are translated into underlying `DataStream` or `DataSet` operations. The supported window definitions are listed below. Window properties such as the start and end timestamp of a time window can be added in the select statement as a property of the window alias as `w.start` and `w.end`, respectively.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+Table table = input
+ .window([Window w].as("w")) // define window with alias w
+ .groupBy("w, a") // group the table by attribute a and window w
+ .select("a, w.start, w.end, b.count"); // aggregate and add window start and end timestamps
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val table = input
+ .window([w: Window] as 'w) // define window with alias w
+ .groupBy('w, 'a) // group the table by attribute a and window w
+ .select('a, 'w.start, 'w.end, 'b.count) // aggregate and add window start and end timestamps
+{% endhighlight %}
+</div>
+</div>
+
+#### Tumble (Tumbling Windows)
+
+A tumbling window assigns rows to non-overlapping, continuous windows of fixed length. For example, a tumbling window of 5 minutes groups rows in 5 minutes intervals. Tumbling windows can be defined on event-time, processing-time, or on a row-count.
+
+Tumbling windows are defined by using the `Tumble` class as follows:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Method</th>
+ <th class="text-left" style="width: 20%">Required?</th>
+ <th class="text-left">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td><code>over</code></td>
+ <td>Required.</td>
+ <td>Defines the length the window, either as time or row-count interval.</td>
+ </tr>
+ <tr>
+ <td><code>on</code></td>
+ <td>Required for streaming event-time windows and windows on batch tables.</td>
+ <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped.</td>
+ </tr>
+ <tr>
+ <td><code>as</code></td>
+ <td>Required.</td>
+ <td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
+ </tr>
+ </tbody>
+</table>
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// Tumbling Event-time Window
+.window(Tumble.over("10.minutes").on("rowtime").as("w"));
+
+// Tumbling Processing-time Window
+.window(Tumble.over("10.minutes").as("w"));
+
+// Tumbling Row-count Window
+.window(Tumble.over("10.rows").as("w"));
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// Tumbling Event-time Window
+.window(Tumble over 10.minutes on 'rowtime as 'w)
+
+// Tumbling Processing-time Window
+.window(Tumble over 10.minutes as 'w)
+
+// Tumbling Row-count Window
+.window(Tumble over 10.rows as 'w)
+{% endhighlight %}
+</div>
+</div>
+
+#### Slide (Sliding Windows)
+
+A sliding window has a fixed size and slides by a specified slide interval. If the slide interval is smaller than the window size, sliding windows are overlapping. Thus, rows can be assigned to multiple windows. For example, a sliding window of 15 minutes size and 5 minute slide interval assigns each row to 3 different windows of 15 minute size, which are evaluated in an interval of 5 minutes. Sliding windows can be defined on event-time, processing-time, or on a row-count.
+
+Sliding windows are defined by using the `Slide` class as follows:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Method</th>
+ <th class="text-left" style="width: 20%">Required?</th>
+ <th class="text-left">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td><code>over</code></td>
+ <td>Required.</td>
+ <td>Defines the length of the window, either as time or row-count interval.</td>
+ </tr>
+ <tr>
+ <td><code>every</code></td>
+ <td>Required.</td>
+ <td>Defines the slide interval, either as time or row-count interval. The slide interval must be of the same type as the size interval.</td>
+ </tr>
+ <tr>
+ <td><code>on</code></td>
+ <td>Required for event-time windows and windows on batch tables.</td>
+ <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped</td>
+ </tr>
+ <tr>
+ <td><code>as</code></td>
+ <td>Required.</td>
+ <td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
+ </tr>
+ </tbody>
+</table>
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// Sliding Event-time Window
+.window(Slide.over("10.minutes").every("5.minutes").on("rowtime").as("w"));
+
+// Sliding Processing-time window
+.window(Slide.over("10.minutes").every("5.minutes").as("w"));
+
+// Sliding Row-count window
+.window(Slide.over("10.rows").every("5.rows").as("w"));
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// Sliding Event-time Window
+.window(Slide over 10.minutes every 5.minutes on 'rowtime as 'w)
+
+// Sliding Processing-time window
+.window(Slide over 10.minutes every 5.minutes as 'w)
+
+// Sliding Row-count window
+.window(Slide over 10.rows every 5.rows as 'w)
+{% endhighlight %}
+</div>
+</div>
+
+#### Session (Session Windows)
+
+Session windows do not have a fixed size but their bounds are defined by an interval of inactivity, i.e., a session window is closes if no event appears for a defined gap period. For example a session window with a 30 minute gap starts when a row is observed after 30 minutes inactivity (otherwise the row would be added to an existing window) and is closed if no row is added within 30 minutes. Session windows can work on event-time or processing-time.
+
+A session window is defined by using the `Session` class as follows:
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 20%">Method</th>
+ <th class="text-left" style="width: 20%">Required?</th>
+ <th class="text-left">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+ <tr>
+ <td><code>withGap</code></td>
+ <td>Required.</td>
+ <td>Defines the gap between two windows as time interval.</td>
+ </tr>
+ <tr>
+ <td><code>on</code></td>
+ <td>Required for event-time windows and windows on batch tables.</td>
+ <td>Defines the time mode for streaming tables (<code>rowtime</code> is a logical system attribute); for batch tables, the time attribute on which records are grouped</td>
+ </tr>
+ <tr>
+ <td><code>as</code></td>
+ <td>Required.</td>
+ <td>Assigns an alias to the window. The alias is used to reference the window in the following <code>groupBy()</code> clause and optionally to select window properties such as window start or end time in the <code>select()</code> clause.</td>
+ </tr>
+ </tbody>
+</table>
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// Session Event-time Window
+.window(Session.withGap("10.minutes").on("rowtime").as("w"));
+
+// Session Processing-time Window
+.window(Session.withGap("10.minutes").as("w"));
+{% endhighlight %}
+</div>
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// Session Event-time Window
+.window(Session withGap 10.minutes on 'rowtime as 'w)
+
+// Session Processing-time Window
+.window(Session withGap 10.minutes as 'w)
+{% endhighlight %}
+</div>
+</div>
+
+{% top %}
+
+Data Types
+----------
+
+The Table API is built on top of Flink's DataSet and DataStream API. Internally, it also uses Flink's `TypeInformation` to distinguish between types. The Table API does not support all Flink types so far. All supported simple types are listed in `org.apache.flink.table.api.Types`. The following table summarizes the relation between Table API types, SQL types, and the resulting Java class.
+
+| Table API | SQL | Java type |
+| :--------------------- | :-------------------------- | :--------------------- |
+| `Types.STRING` | `VARCHAR` | `java.lang.String` |
+| `Types.BOOLEAN` | `BOOLEAN` | `java.lang.Boolean` |
+| `Types.BYTE` | `TINYINT` | `java.lang.Byte` |
+| `Types.SHORT` | `SMALLINT` | `java.lang.Short` |
+| `Types.INT` | `INTEGER, INT` | `java.lang.Integer` |
+| `Types.LONG` | `BIGINT` | `java.lang.Long` |
+| `Types.FLOAT` | `REAL, FLOAT` | `java.lang.Float` |
+| `Types.DOUBLE` | `DOUBLE` | `java.lang.Double` |
+| `Types.DECIMAL` | `DECIMAL` | `java.math.BigDecimal` |
+| `Types.DATE` | `DATE` | `java.sql.Date` |
+| `Types.TIME` | `TIME` | `java.sql.Time` |
+| `Types.TIMESTAMP` | `TIMESTAMP(3)` | `java.sql.Timestamp` |
+| `Types.INTERVAL_MONTHS`| `INTERVAL YEAR TO MONTH` | `java.lang.Integer` |
+| `Types.INTERVAL_MILLIS`| `INTERVAL DAY TO SECOND(3)` | `java.lang.Long` |
+| `Types.PRIMITIVE_ARRAY`| `ARRAY` | e.g. `int[]` |
+| `Types.OBJECT_ARRAY` | `ARRAY` | e.g. `java.lang.Byte[]`|
+| `Types.MAP` | `MAP` | `java.util.HashMap` |
+
+
+Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and array types (object or primitive arrays) can be fields of a row.
+
+Generic types are treated as a black box within Table API and SQL yet.
+
+Composite types, however, are fully supported types where fields of a composite type can be accessed using the `.get()` operator in Table API and dot operator (e.g. `MyTable.pojoColumn.myField`) in SQL. Composite types can also be flattened using `.flatten()` in Table API or `MyTable.pojoColumn.*` in SQL.
+
+Array types can be accessed using the `myArray.at(1)` operator in Table API and `myArray[1]` operator in SQL. Array literals can be created using `array(1, 2, 3)` in Table API and `ARRAY[1, 2, 3]` in SQL.
+
+**TODO: Clean-up and move relevant parts to the "Mappings Types to Table Schema" section of the Common Concepts & API page.**
+
+{% top %}
+
+Expression Syntax
+-----------------
+
+Some of the operators in previous sections expect one or more expressions. Expressions can be specified using an embedded Scala DSL or as Strings. Please refer to the examples above to learn how expressions can be specified.
+
+This is the EBNF grammar for expressions:
+
+{% highlight ebnf %}
+
+expressionList = expression , { "," , expression } ;
+
+expression = timeIndicator | overConstant | alias ;
+
+alias = logic | ( logic , "as" , fieldReference ) | ( logic , "as" , "(" , fieldReference , { "," , fieldReference } , ")" ) ;
+
+logic = comparison , [ ( "&&" | "||" ) , comparison ] ;
+
+comparison = term , [ ( "=" | "==" | "===" | "!=" | "!==" | ">" | ">=" | "<" | "<=" ) , term ] ;
+
+term = product , [ ( "+" | "-" ) , product ] ;
+
+product = unary , [ ( "*" | "/" | "%") , unary ] ;
+
+unary = [ "!" | "-" ] , composite ;
+
+composite = over | nullLiteral | suffixed | atom ;
+
+suffixed = interval | cast | as | if | functionCall ;
+
+interval = timeInterval | rowInterval ;
+
+timeInterval = composite , "." , ("year" | "years" | "month" | "months" | "day" | "days" | "hour" | "hours" | "minute" | "minutes" | "second" | "seconds" | "milli" | "millis") ;
+
+rowInterval = composite , "." , "rows" ;
+
+cast = composite , ".cast(" , dataType , ")" ;
+
+dataType = "BYTE" | "SHORT" | "INT" | "LONG" | "FLOAT" | "DOUBLE" | "BOOLEAN" | "STRING" | "DECIMAL" | "SQL_DATE" | "SQL_TIME" | "SQL_TIMESTAMP" | "INTERVAL_MONTHS" | "INTERVAL_MILLIS" | ( "PRIMITIVE_ARRAY" , "(" , dataType , ")" ) | ( "OBJECT_ARRAY" , "(" , dataType , ")" ) ;
+
+as = composite , ".as(" , fieldReference , ")" ;
+
+if = composite , ".?(" , expression , "," , expression , ")" ;
+
+functionCall = composite , "." , functionIdentifier , [ "(" , [ expression , { "," , expression } ] , ")" ] ;
+
+atom = ( "(" , expression , ")" ) | literal | fieldReference ;
+
+fieldReference = "*" | identifier ;
+
+nullLiteral = "Null(" , dataType , ")" ;
+
+timeIntervalUnit = "YEAR" | "YEAR_TO_MONTH" | "MONTH" | "DAY" | "DAY_TO_HOUR" | "DAY_TO_MINUTE" | "DAY_TO_SECOND" | "HOUR" | "HOUR_TO_MINUTE" | "HOUR_TO_SECOND" | "MINUTE" | "MINUTE_TO_SECOND" | "SECOND" ;
+
+timePointUnit = "YEAR" | "MONTH" | "DAY" | "HOUR" | "MINUTE" | "SECOND" | "QUARTER" | "WEEK" | "MILLISECOND" | "MICROSECOND" ;
+
+over = composite , "over" , fieldReference ;
+
+overConstant = "current_row" | "current_range" | "unbounded_row" | "unbounded_row" ;
+
+timeIndicator = fieldReference , "." , ( "proctime" | "rowtime" ) ;
+
+{% endhighlight %}
+
+Here, `literal` is a valid Java literal, `fieldReference` specifies a column in the data (or all columns if `*` is used), and `functionIdentifier` specifies a supported scalar function. The
+column names and function names follow Java identifier syntax. Expressions specified as Strings can also use prefix notation instead of suffix notation to call operators and functions.
+
+If working with exact numeric values or large decimals is required, the Table API also supports Java's BigDecimal type. In the Scala Table API decimals can be defined by `BigDecimal("123456")` and in Java by appending a "p" for precise e.g. `123456p`.
+
+In order to work with temporal values the Table API supports Java SQL's Date, Time, and Timestamp types. In the Scala Table API literals can be defined by using `java.sql.Date.valueOf("2016-06-27")`, `java.sql.Time.valueOf("10:10:42")`, or `java.sql.Timestamp.valueOf("2016-06-27 10:10:42.123")`. The Java and Scala Table API also support calling `"2016-06-27".toDate()`, `"10:10:42".toTime()`, and `"2016-06-27 10:10:42.123".toTimestamp()` for converting Strings into temporal types. *Note:* Since Java's temporal SQL types are time zone dependent, please make sure that the Flink Client and all TaskManagers use the same time zone.
+
+Temporal intervals can be represented as number of months (`Types.INTERVAL_MONTHS`) or number of milliseconds (`Types.INTERVAL_MILLIS`). Intervals of same type can be added or subtracted (e.g. `1.hour + 10.minutes`). Intervals of milliseconds can be added to time points (e.g. `"2016-08-10".toDate + 5.days`).
+
+**TODO: needs to be reworked, IMO. Grammar might be complete but is hard to understand without concrete examples**
+
+{% top %}
+
+Built-In Functions
+------------------
+
+Both the Table API and SQL come with a set of built-in functions for data transformations. This section gives a brief overview of the available functions so far.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Comparison functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY === ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Equals.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY !== ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Not equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY > ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Greater than.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY >= ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Greater than or equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY < ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Less than.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY <= ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Less than or equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.isNull
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.isNotNull
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is not null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.like(STRING)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true, if a string matches the specified LIKE pattern. E.g. "Jo_n%" matches all strings that start with "Jo(arbitrary letter)n".</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.similar(STRING)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true, if a string matches the specified SQL regex pattern. E.g. "A+" matches all strings that consist of at least one "A".</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Logical functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+boolean1 || boolean2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if <i>boolean1</i> is true or <i>boolean2</i> is true. Supports three-valued logic.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+boolean1 && boolean2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if <i>boolean1</i> and <i>boolean2</i> are both true. Supports three-valued logic.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+!BOOLEAN
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if boolean expression is not true; returns null if boolean is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+BOOLEAN.isTrue
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given boolean expression is true. False otherwise (for null and false).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+BOOLEAN.isFalse
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if given boolean expression is false. False otherwise (for null and true).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+BOOLEAN.isNotTrue
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given boolean expression is not true (for null and false). False otherwise.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+BOOLEAN.isNotFalse
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if given boolean expression is not false (for null and true). False otherwise.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Arithmetic functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
++ numeric
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+- numeric
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns negative <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+numeric1 + numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> plus <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+numeric1 - numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> minus <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+numeric1 * numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> multiplied by <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+numeric1 / numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> divided by <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+numeric1.power(numeric2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> raised to the power of <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.abs()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the absolute value of given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+numeric1 % numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the remainder (modulus) of <i>numeric1</i> divided by <i>numeric2</i>. The result is negative only if <i>numeric1</i> is negative.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.sqrt()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the square root of a given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.ln()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the natural logarithm of given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.log10()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the base 10 logarithm of given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.exp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the Euler's number raised to the given power.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.ceil()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the smallest integer greater than or equal to a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.floor()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the largest integer less than or equal to a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.sin()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the sine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.cos()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the cosine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.tan()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the tangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.cot()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the cotangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.asin()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc sine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.acos()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc cosine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.atan()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc tangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.degrees()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts <i>numeric</i> from radians to degrees.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.radians()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts <i>numeric</i> from degrees to radians.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.sign()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the signum of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.round(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds the given number to <i>integer</i> places right to the decimal point.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+pi()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a value that is closer than any other value to pi.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">String functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING + STRING
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Concatenates two character strings.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.charLength()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the length of a String.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.upperCase()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns all of the characters in a string in upper case using the rules of the default locale.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.lowerCase()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns all of the characters in a string in lower case using the rules of the default locale.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.position(STRING)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the position of string in an other string starting at 1. Returns 0 if string could not be found. E.g. <code>'a'.position('bbbbba')</code> leads to 6.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.trim(LEADING, STRING)
+STRING.trim(TRAILING, STRING)
+STRING.trim(BOTH, STRING)
+STRING.trim(BOTH)
+STRING.trim()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Removes leading and/or trailing characters from the given string. By default, whitespaces at both sides are removed.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.overlay(STRING, INT)
+STRING.overlay(STRING, INT, INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Replaces a substring of string with a string starting at a position (starting at 1). An optional length specifies how many characters should be removed. E.g. <code>'xxxxxtest'.overlay('xxxx', 6)</code> leads to "xxxxxxxxx", <code>'xxxxxtest'.overlay('xxxx', 6, 2)</code> leads to "xxxxxxxxxst".</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.substring(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates a substring of the given string beginning at the given index to the end. The start index starts at 1 and is inclusive.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.substring(INT, INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates a substring of the given string at the given index for the given length. The index starts at 1 and is inclusive, i.e., the character at the index is included in the substring. The substring has the specified length or less.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.initCap()
+{% endhighlight %}
+ </td>
+
+ <td>
+ <p>Converts the initial letter of each word in a string to uppercase. Assumes a string containing only [A-Za-z0-9], everything else is treated as whitespace.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Conditional functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+BOOLEAN.?(value1, value2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Ternary conditional operator that decides which of two other expressions should be evaluated based on a evaluated boolean condition. E.g. <code>(42 > 5).?("A", "B")</code> leads to "A".</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Type conversion functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.cast(TYPE)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts a value to a given type. E.g. <code>"42".cast(INT)</code> leads to 42.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Value constructor functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ARRAY.at(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the element at a particular position in an array. The index starts at 1.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+array(ANY [, ANY ]*)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an array from a list of values. The array will be an array of objects (not primitives).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.rows
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of rows.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Temporal functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.toDate()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a date string in the form "yy-mm-dd" to a SQL date.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.toTime()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a time string in the form "hh:mm:ss" to a SQL time.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+STRING.toTimestamp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a timestamp string in the form "yy-mm-dd hh:mm:ss.fff" to a SQL timestamp.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.year
+NUMERIC.years
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of months for a given number of years.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.month
+NUMERIC.months
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of months for a given number of months.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.day
+NUMERIC.days
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of days.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.hour
+NUMERIC.hours
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of hours.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.minute
+NUMERIC.minutes
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of minutes.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.second
+NUMERIC.seconds
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of seconds.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+NUMERIC.milli
+NUMERIC.millis
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+currentDate()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL date in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+currentTime()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL time in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+currentTimestamp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL timestamp in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+localTime()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL time in local time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+localTimestamp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL timestamp in local time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+TEMPORAL.extract(TIMEINTERVALUNIT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Extracts parts of a time point or time interval. Returns the part as a long value. E.g. <code>'2006-06-05'.toDate.extract(DAY)</code> leads to 5.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+TIMEPOINT.floor(TIMEINTERVALUNIT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds a time point down to the given unit. E.g. <code>'12:44:31'.toDate.floor(MINUTE)</code> leads to 12:44:00.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+TIMEPOINT.ceil(TIMEINTERVALUNIT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds a time point up to the given unit. E.g. <code>'12:44:31'.toTime.floor(MINUTE)</code> leads to 12:45:00.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+DATE.quarter()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the quarter of a year from a SQL date. E.g. <code>'1994-09-27'.toDate.quarter()</code> leads to 3.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+temporalOverlaps(TIMEPOINT, TEMPORAL, TIMEPOINT, TEMPORAL)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Determines whether two anchored time intervals overlap. Time point and temporal are transformed into a range defined by two time points (start, end). The function evaluates <code>leftEnd >= rightStart && rightEnd >= leftStart</code>. E.g. <code>temporalOverlaps("2:55:00".toTime, 1.hour, "3:30:00".toTime, 2.hour)</code> leads to true.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Aggregate functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.count
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the number of input rows for which the field is not null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.avg
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the average (arithmetic mean) of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.sum
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sum of the numeric field across all input values. If all values are null, null is returned.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.sum0
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sum of the numeric field across all input values. If all values are null, 0 is returned.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.max
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the maximum value of field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.min
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the minimum value of field across all input values.</p>
+ </td>
+ </tr>
+
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.stddevPop
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the population standard deviation of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.stddevSamp
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sample standard deviation of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.varPop
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the population variance (square of the population standard deviation) of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+FIELD.varSamp
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sample variance (square of the sample standard deviation) of the numeric field across all input values.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Value access functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+COMPOSITE.get(STRING)
+COMPOSITE.get(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Accesses the field of a Flink composite type (such as Tuple, POJO, etc.) by index or name and returns it's value. E.g. <code>pojo.get('myField')</code> or <code>tuple.get(0)</code>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.flatten()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts a Flink composite type (such as Tuple, POJO, etc.) and all of its direct subtypes into a flat representation where every subtype is a separate field. In most cases the fields of the flat representation are named similarly to the original fields but with a dollar separator (e.g. <code>mypojo$mytuple$f0</code>).</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Array functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ARRAY.cardinality()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the number of elements of an array.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ARRAY.element()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the sole element of an array with a single element. Returns <code>null</code> if the array is empty. Throws an exception if the array has more than one element.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Auxiliary functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+ANY.as(name [, name ]* )
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Specifies a name for an expression i.e. a field. Additional names can be specified if the expression expands to multiple fields.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+</div>
+<div data-lang="scala" markdown="1">
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Comparison functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY === ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Equals.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY !== ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Not equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY > ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Greater than.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY >= ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Greater than or equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY < ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Less than.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY <= ANY
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Less than or equal.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY.isNull
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY.isNotNull
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given expression is not null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.like(STRING)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true, if a string matches the specified LIKE pattern. E.g. "Jo_n%" matches all strings that start with "Jo(arbitrary letter)n".</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.similar(STRING)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true, if a string matches the specified SQL regex pattern. E.g. "A+" matches all strings that consist of at least one "A".</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Logical functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+boolean1 || boolean2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if <i>boolean1</i> is true or <i>boolean2</i> is true. Supports three-valued logic.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+boolean1 && boolean2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if <i>boolean1</i> and <i>boolean2</i> are both true. Supports three-valued logic.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+!BOOLEAN
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if boolean expression is not true; returns null if boolean is null.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+BOOLEAN.isTrue
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given boolean expression is true. False otherwise (for null and false).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+BOOLEAN.isFalse
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if given boolean expression is false. False otherwise (for null and true).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+BOOLEAN.isNotTrue
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if the given boolean expression is not true (for null and false). False otherwise.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+BOOLEAN.isNotFalse
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns true if given boolean expression is not false (for null and true). False otherwise.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Arithmetic functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
++ numeric
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+- numeric
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns negative <i>numeric</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+numeric1 + numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> plus <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+numeric1 - numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> minus <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+numeric1 * numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> multiplied by <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+numeric1 / numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> divided by <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+numeric1.power(numeric2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns <i>numeric1</i> raised to the power of <i>numeric2</i>.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.abs()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the absolute value of given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+numeric1 % numeric2
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the remainder (modulus) of <i>numeric1</i> divided by <i>numeric2</i>. The result is negative only if <i>numeric1</i> is negative.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.sqrt()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the square root of a given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.ln()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the natural logarithm of given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.log10()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the base 10 logarithm of given value.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.exp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the Euler's number raised to the given power.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.ceil()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the smallest integer greater than or equal to a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.floor()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the largest integer less than or equal to a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.sin()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the sine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.cos()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the cosine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.tan()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the cotangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.cot()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc sine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.asin()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc cosine of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.acos()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the arc tangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.atan()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the tangent of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.degrees()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts <i>numeric</i> from radians to degrees.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.radians()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts <i>numeric</i> from degrees to radians.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.sign()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Calculates the signum of a given number.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.round(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds the given number to <i>integer</i> places right to the decimal point.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+pi()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns a value that is closer than any other value to pi.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Arithmetic functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING + STRING
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Concatenates two character strings.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.charLength()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the length of a String.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.upperCase()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns all of the characters in a string in upper case using the rules of the default locale.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.lowerCase()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns all of the characters in a string in lower case using the rules of the default locale.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.position(STRING)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the position of string in an other string starting at 1. Returns 0 if string could not be found. E.g. <code>"a".position("bbbbba")</code> leads to 6.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.trim(
+ leading = true,
+ trailing = true,
+ character = " ")
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Removes leading and/or trailing characters from the given string.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.overlay(STRING, INT)
+STRING.overlay(STRING, INT, INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Replaces a substring of string with a string starting at a position (starting at 1). An optional length specifies how many characters should be removed. E.g. <code>"xxxxxtest".overlay("xxxx", 6)</code> leads to "xxxxxxxxx", <code>"xxxxxtest".overlay('xxxx', 6, 2)</code> leads to "xxxxxxxxxst".</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.substring(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates a substring of the given string beginning at the given index to the end. The start index starts at 1 and is inclusive.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.substring(INT, INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates a substring of the given string at the given index for the given length. The index starts at 1 and is inclusive, i.e., the character at the index is included in the substring. The substring has the specified length or less.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.initCap()
+{% endhighlight %}
+ </td>
+
+ <td>
+ <p>Converts the initial letter of each word in a string to uppercase. Assumes a string containing only [A-Za-z0-9], everything else is treated as whitespace.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Conditional functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight java %}
+BOOLEAN.?(value1, value2)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Ternary conditional operator that decides which of two other expressions should be evaluated based on a evaluated boolean condition. E.g. <code>(42 > 5).?("A", "B")</code> leads to "A".</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Type conversion functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ANY.cast(TYPE)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Converts a value to a given type. E.g. <code>"42".cast(Types.INT)</code> leads to 42.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Value constructor functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+ARRAY.at(INT)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the element at a particular position in an array. The index starts at 1.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+array(ANY [, ANY ]*)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an array from a list of values. The array will be an array of objects (not primitives).</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.rows
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of rows.</p>
+ </td>
+ </tr>
+
+ </tbody>
+</table>
+
+<table class="table table-bordered">
+ <thead>
+ <tr>
+ <th class="text-left" style="width: 40%">Temporal functions</th>
+ <th class="text-center">Description</th>
+ </tr>
+ </thead>
+
+ <tbody>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.toDate
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a date string in the form "yy-mm-dd" to a SQL date.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.toTime
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a time string in the form "hh:mm:ss" to a SQL time.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+STRING.toTimestamp
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Parses a timestamp string in the form "yy-mm-dd hh:mm:ss.fff" to a SQL timestamp.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.year
+NUMERIC.years
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of months for a given number of years.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.month
+NUMERIC.months
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of months for a given number of months.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.day
+NUMERIC.days
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of days.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.hour
+NUMERIC.hours
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of hours.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.minute
+NUMERIC.minutes
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of minutes.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.second
+NUMERIC.seconds
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds for a given number of seconds.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+NUMERIC.milli
+NUMERIC.millis
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Creates an interval of milliseconds.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+currentDate()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL date in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+currentTime()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL time in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+currentTimestamp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL timestamp in UTC time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+localTime()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL time in local time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+localTimestamp()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the current SQL timestamp in local time zone.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+TEMPORAL.extract(TimeIntervalUnit)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Extracts parts of a time point or time interval. Returns the part as a long value. E.g. <code>"2006-06-05".toDate.extract(TimeIntervalUnit.DAY)</code> leads to 5.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+TIMEPOINT.floor(TimeIntervalUnit)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds a time point down to the given unit. E.g. <code>"12:44:31".toTime.floor(TimeIntervalUnit.MINUTE)</code> leads to 12:44:00.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+TIMEPOINT.ceil(TimeIntervalUnit)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Rounds a time point up to the given unit. E.g. <code>"12:44:31".toTime.floor(TimeIntervalUnit.MINUTE)</code> leads to 12:45:00.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+DATE.quarter()
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Returns the quarter of a year from a SQL date. E.g. <code>"1994-09-27".toDate.quarter()</code> leads to 3.</p>
+ </td>
+ </tr>
+
+ <tr>
+ <td>
+ {% highlight scala %}
+temporalOverlaps(TIMEPOINT, TEMPORAL, TIMEPOINT, TEMPORAL)
+{% endhighlight %}
+ </td>
+ <td>
+ <p>Determines whether two anchored time intervals overlap. Time point and te
<TRUNCATED>
[05/10] flink git commit: [FLINK-6747] [table] [docs] Time attributes
section added
Posted by fh...@apache.org.
[FLINK-6747] [table] [docs] Time attributes section added
This closes #4020.
Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/a29a9adb
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/a29a9adb
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/a29a9adb
Branch: refs/heads/master
Commit: a29a9adb8958cbb283a7cdfa24829cc66699e7e4
Parents: a5d93a5
Author: twalthr <tw...@apache.org>
Authored: Tue May 30 17:26:07 2017 +0200
Committer: Fabian Hueske <fh...@apache.org>
Committed: Thu Jun 15 11:42:18 2017 +0200
----------------------------------------------------------------------
docs/dev/table/streaming.md | 309 ++++++++++++++++++++++++++++++++++++++-
1 file changed, 302 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flink/blob/a29a9adb/docs/dev/table/streaming.md
----------------------------------------------------------------------
diff --git a/docs/dev/table/streaming.md b/docs/dev/table/streaming.md
index 9699507..d7d97fa 100644
--- a/docs/dev/table/streaming.md
+++ b/docs/dev/table/streaming.md
@@ -22,7 +22,7 @@ specific language governing permissions and limitations
under the License.
-->
-**TODO: has to be completely written**
+**TO BE DONE:** Intro
* This will be replaced by the TOC
{:toc}
@@ -30,6 +30,8 @@ under the License.
Dynamic table
-------------
+**TO BE DONE**
+
* Stream -> Table
* Table -> Stream
* update changes / retraction
@@ -39,17 +41,308 @@ Dynamic table
Time Attributes
---------------
-### Event-time
+Flink is able to process streaming data based on different notions of *time*.
-* DataStream: Timestamps & WMs required, `.rowtime` (replace attribute or extend schema)
-* TableSource: Timestamps & WMs & DefinedRowtimeAttribute
+- *Processing time* refers to the system time of the machine (also known as "wall-clock time") that is executing the respective operation.
+- *Event time* refers to the processing of streaming data based on timestamps which are attached to each row. The timestamps can encode when an event happened.
+- *Ingestion time* is the time that events enter Flink; internally, it is treated similarly to event time.
-{% top %}
+For more information about time handling in Flink, see the introduction about [Event Time and Watermarks]({{ site.baseurl }}/dev/event_time.html).
+
+Table programs require that the corresponding time characteristic has been specified for the streaming environment:
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+
+env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime); // default
+
+// alternatively:
+// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
+// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+
+env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime) // default
+
+// alternatively:
+// env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime)
+// env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
+{% endhighlight %}
+</div>
+</div>
+
+Time-based operations such as windows in both the [Table API]({{ site.baseurl }}/dev/table/tableApi.html#windows) and [SQL]({{ site.baseurl }}/dev/table/sql.html#group-windows) require information about the notion of time and its origin. Therefore, tables can offer *logical time attributes* for indicating time and accessing corresponding timestamps in table programs.
+
+Time attributes can be part of every table schema. They are defined when creating a table from a `DataStream` or are pre-defined when using a `TableSource`. Once a time attribute has been defined at the beginning, it can be referenced as a field and can used in time-based operations.
+
+As long as a time attribute is not modified and is simply forwarded from one part of the query to another, it remains a valid time attribute. Time attributes behave like regular timestamps and can be accessed for calculations. If a time attribute is used in a calculation, it will be materialized and becomes a regular timestamp. Regular timestamps do not cooperate with Flink's time and watermarking system and thus can not be used for time-based operations anymore.
### Processing time
-* DataStream: `.proctime` (only extend schema)
-* TableSource: DefinedProctimeAttribute
+Processing time allows a table program to produce results based on the time of the local machine. It is the simplest notion of time but does not provide determinism. It neither requires timestamp extraction nor watermark generation.
+
+There are two ways to define a processing time attribute.
+
+#### During DataStream-to-Table Conversion
+
+The processing time attribute is defined with the `.proctime` property during schema definition. The time attribute must only extend the physical schema by an additional logical field. Thus, it can only be defined at the end of the schema definition.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+DataStream<Tuple2<String, String>> stream = ...;
+
+// declare an additional logical field as a processing time attribute
+Table table = tEnv.fromDataStream(stream, "Username, Data, UserActionTime.proctime");
+
+WindowedTable windowedTable = table.window(Tumble.over("10.minutes").on("UserActionTime").as("userActionWindow"));
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val stream: DataStream[(String, String)] = ...
+
+// declare an additional logical field as a processing time attribute
+val table = tEnv.fromDataStream(stream, 'UserActionTimestamp, 'Username, 'Data, 'UserActionTime.proctime)
+
+val windowedTable = table.window(Tumble over 10.minutes on 'UserActionTime as 'userActionWindow)
+{% endhighlight %}
+</div>
+</div>
+
+#### Using a TableSource
+
+The processing time attribute is defined by a `TableSource` that implements the `DefinedProctimeAttribute` interface. The logical time attribute is appended to the physical schema defined by the return type of the `TableSource`.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// define a table source with a processing attribute
+public class UserActionSource implements StreamTableSource<Row>, DefinedProctimeAttribute {
+
+ @Override
+ public TypeInformation<Row> getReturnType() {
+ String[] names = new String[] {"Username" , "Data"};
+ TypeInformation[] types = new TypeInformation[] {Types.STRING(), Types.STRING()};
+ return Types.ROW(names, types);
+ }
+
+ @Override
+ public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {
+ // create stream
+ DataStream<Row> stream = ...;
+ return stream;
+ }
+
+ @Override
+ public String getProctimeAttribute() {
+ // field with this name will be appended as a third field
+ return "UserActionTime";
+ }
+}
+
+// register table source
+tEnv.registerTableSource("UserActions", new UserActionSource());
+
+WindowedTable windowedTable = tEnv
+ .scan("UserActions")
+ .window(Tumble.over("10.minutes").on("UserActionTime").as("userActionWindow"));
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// define a table source with a processing attribute
+class UserActionSource extends StreamTableSource[Row] with DefinedProctimeAttribute {
+
+ override def getReturnType = {
+ val names = Array[String]("Username" , "Data")
+ val types = Array[TypeInformation[_]](Types.STRING, Types.STRING)
+ Types.ROW(names, types)
+ }
+
+ override def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {
+ // create stream
+ val stream = ...
+ stream
+ }
+
+ override def getProctimeAttribute = {
+ // field with this name will be appended as a third field
+ "UserActionTime"
+ }
+}
+
+// register table source
+tEnv.registerTableSource("UserActions", new UserActionSource)
+
+val windowedTable = tEnv
+ .scan("UserActions")
+ .window(Tumble over 10.minutes on 'UserActionTime as 'userActionWindow)
+{% endhighlight %}
+</div>
+</div>
+
+### Event time
+
+Event time allows a table program to produce results based on the time that is contained in every record. This allows for consistent results even in case of out-of-order events or late events. It also ensures replayable results of the table program when reading records from persistent storage.
+
+Additionally, event time allows for unified syntax for table programs in both batch and streaming environments. A time attribute in a streaming environment can be a regular field of a record in a batch environment.
+
+In order to handle out-of-order events and distinguish between on-time and late events in streaming, Flink needs to extract timestamps from events and make some kind of progress in time (so-called [watermarks]({{ site.baseurl }}/dev/event_time.html)).
+
+An event time attribute can be defined either during DataStream-to-Table conversion or by using a TableSource.
+
+The Table API & SQL assume that in both cases timestamps and watermarks have been generated in the [underlying DataStream API]({{ site.baseurl }}/dev/event_timestamps_watermarks.html) before. Ideally, this happens within a `TableSource` with knowledge about the incoming data's characteristics and is hidden from the end user of the API.
+
+
+#### During DataStream-to-Table Conversion
+
+The event time attribute is defined with the `.rowtime` property during schema definition.
+
+Timestamps and watermarks must have been assigned in the `DataStream` that is converted.
+
+There are two ways of defining the time attribute when converting a `DataStream` into a `Table`:
+
+- Extending the physical schema by an additional logical field
+- Replacing a physical field by a logical field (e.g. because it is no longer needed after timestamp extraction).
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+
+// Option 1:
+
+// extract timestamp and assign watermarks based on knowledge of the stream
+DataStream<Tuple3<String, String>> stream = inputStream.assignTimestampsAndWatermarks(...);
+
+// declare an additional logical field as an event time attribute
+Table table = tEnv.fromDataStream(stream, "Username, Data, UserActionTime.rowtime");
+
+
+// Option 2:
+
+// extract timestamp from first field, and assign watermarks based on knowledge of the stream
+DataStream<Tuple3<Long, String, String>> stream = inputStream.assignTimestampsAndWatermarks(...);
+
+// the first field has been used for timestamp extraction, and is no longer necessary
+// replace first field with a logical event time attribute
+Table table = tEnv.fromDataStream(stream, "UserActionTime.rowtime, Username, Data");
+
+// Usage:
+
+WindowedTable windowedTable = table.window(Tumble.over("10.minutes").on("UserActionTime").as("userActionWindow"));
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+
+// Option 1:
+
+// extract timestamp and assign watermarks based on knowledge of the stream
+val stream: DataStream[(String, String)] = inputStream.assignTimestampsAndWatermarks(...)
+
+// declare an additional logical field as an event time attribute
+val table = tEnv.fromDataStream(stream, 'Username, 'Data, 'UserActionTime.rowtime)
+
+
+// Option 2:
+
+// extract timestamp from first field, and assign watermarks based on knowledge of the stream
+val stream: DataStream[(Long, String, String)] = inputStream.assignTimestampsAndWatermarks(...)
+
+// the first field has been used for timestamp extraction, and is no longer necessary
+// replace first field with a logical event time attribute
+val table = tEnv.fromDataStream(stream, 'UserActionTime.rowtime, 'Username, 'Data)
+
+// Usage:
+
+val windowedTable = table.window(Tumble over 10.minutes on 'UserActionTime as 'userActionWindow)
+{% endhighlight %}
+</div>
+</div>
+
+#### Using a TableSource
+
+The event time attribute is defined by a `TableSource` that implements the `DefinedRowtimeAttribute` interface. The logical time attribute is appended to the physical schema defined by the return type of the `TableSource`.
+
+Timestamps and watermarks must be assigned in the stream that is returned by the `getDataStream()` method.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+// define a table source with a rowtime attribute
+public class UserActionSource implements StreamTableSource<Row>, DefinedRowtimeAttribute {
+
+ @Override
+ public TypeInformation<Row> getReturnType() {
+ String[] names = new String[] {"Username" , "Data"};
+ TypeInformation[] types = new TypeInformation[] {Types.STRING(), Types.STRING()};
+ return Types.ROW(names, types);
+ }
+
+ @Override
+ public DataStream<Row> getDataStream(StreamExecutionEnvironment execEnv) {
+ // create stream
+ // ...
+ // extract timestamp and assign watermarks based on knowledge of the stream
+ DataStream<Row> stream = inputStream.assignTimestampsAndWatermarks(...);
+ return stream;
+ }
+
+ @Override
+ public String getRowtimeAttribute() {
+ // field with this name will be appended as a third field
+ return "UserActionTime";
+ }
+}
+
+// register the table source
+tEnv.registerTableSource("UserActions", new UserActionSource());
+
+WindowedTable windowedTable = tEnv
+ .scan("UserActions")
+ .window(Tumble.over("10.minutes").on("UserActionTime").as("userActionWindow"));
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+// define a table source with a rowtime attribute
+class UserActionSource extends StreamTableSource[Row] with DefinedRowtimeAttribute {
+
+ override def getReturnType = {
+ val names = Array[String]("Username" , "Data")
+ val types = Array[TypeInformation[_]](Types.STRING, Types.STRING)
+ Types.ROW(names, types)
+ }
+
+ override def getDataStream(execEnv: StreamExecutionEnvironment): DataStream[Row] = {
+ // create stream
+ // ...
+ // extract timestamp and assign watermarks based on knowledge of the stream
+ val stream = inputStream.assignTimestampsAndWatermarks(...)
+ stream
+ }
+
+ override def getRowtimeAttribute = {
+ // field with this name will be appended as a third field
+ "UserActionTime"
+ }
+}
+
+// register the table source
+tEnv.registerTableSource("UserActions", new UserActionSource)
+
+val windowedTable = tEnv
+ .scan("UserActions")
+ .window(Tumble over 10.minutes on 'UserActionTime as 'userActionWindow)
+{% endhighlight %}
+</div>
+</div>
{% top %}
@@ -62,6 +355,8 @@ Flink's Table API and SQL interface use a `QueryConfig` to control the computati
### State Retention
+**TO BE DONE**
+
{% top %}