You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flink.apache.org by yu...@apache.org on 2023/09/01 06:16:29 UTC

[flink] branch master updated: [FLINK-32475][docs] Add doc for time travel (#23109)

This is an automated email from the ASF dual-hosted git repository.

yuxia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git


The following commit(s) were added to refs/heads/master by this push:
     new 9c6b34d089a [FLINK-32475][docs] Add doc for time travel (#23109)
9c6b34d089a is described below

commit 9c6b34d089a291be08b51c23634627e451f5936a
Author: Feng Jin <ji...@gmail.com>
AuthorDate: Fri Sep 1 14:16:22 2023 +0800

    [FLINK-32475][docs] Add doc for time travel (#23109)
    
    Co-authored-by: luoyuxia <lu...@alumni.sjtu.edu.cn>
---
 docs/content.zh/docs/dev/table/catalogs.md         | 38 +++++++++++
 .../docs/dev/table/sql/queries/overview.md         |  1 +
 .../docs/dev/table/sql/queries/time-travel.md      | 72 +++++++++++++++++++++
 docs/content/docs/dev/table/catalogs.md            | 38 +++++++++++
 .../content/docs/dev/table/sql/queries/overview.md |  1 +
 .../docs/dev/table/sql/queries/time-travel.md      | 75 ++++++++++++++++++++++
 6 files changed, 225 insertions(+)

diff --git a/docs/content.zh/docs/dev/table/catalogs.md b/docs/content.zh/docs/dev/table/catalogs.md
index 78daf499d20..2933d35a93e 100644
--- a/docs/content.zh/docs/dev/table/catalogs.md
+++ b/docs/content.zh/docs/dev/table/catalogs.md
@@ -65,6 +65,44 @@ Catalog 是可扩展的,用户可以通过实现 `Catalog` 接口来开发自
  在用户自定义 catalog 中,应该将 `Thread.currentThread().getContextClassLoader()` 替换成该用户类加载器去加载类。否则,可能会发生 `ClassNotFoundException` 的异常。该用户类加载器可以通过 `CatalogFactory.Context#getClassLoader` 获得。
 {{< /hint >}}
 
+#### Catalog 中支持时间旅行的接口
+
+从 1.18 开始, Flink 框架开始支持[时间旅行]({{< ref "docs/dev/table/sql/queries/time-travel" >}})查询表的历史数据。如果要查询表的历史数据,需要这张表所属于的 `catalog` 实现 `getTable(ObjectPath tablePath, long timestamp)` 方法,如下所示:
+
+```java
+public class MyCatalogSupportTimeTravel implements Catalog {
+    
+    @Override
+    public CatalogBaseTable getTable(ObjectPath tablePath, long timestamp)
+            throws TableNotExistException {
+        // Build a schema corresponding to the specific time point.
+        Schema schema = buildSchema(timestamp);
+        // Set parameters to read data at the corresponding time point.
+        Map<String, String> options = buildOptions(timestamp);
+        // Build CatalogTable
+        CatalogTable catalogTable =
+                CatalogTable.of(schema, "", Collections.emptyList(), options, timestamp);
+        return catalogTable;
+    }
+}
+
+public class MyDynamicTableFactory implements DynamicTableSourceFactory {
+    @Override
+    public DynamicTableSource createDynamicTableSource(Context context) {
+        final ReadableConfig configuration =
+                Configuration.fromMap(context.getCatalogTable().getOptions());
+
+        // Get snapshot from CatalogTable
+        final Optional<Long> snapshot = context.getCatalogTable().getSnapshot();
+        
+        // Build DynamicTableSource using snapshot options.
+        final DynamicTableSource dynamicTableSource = buildDynamicSource(configuration, snapshot);
+
+        return dynamicTableSource;
+    }
+}
+```
+
 ## 如何创建 Flink 表并将其注册到 Catalog
 
 ### 使用 SQL DDL
diff --git a/docs/content.zh/docs/dev/table/sql/queries/overview.md b/docs/content.zh/docs/dev/table/sql/queries/overview.md
index bf3e8d88ed0..90f13d30452 100644
--- a/docs/content.zh/docs/dev/table/sql/queries/overview.md
+++ b/docs/content.zh/docs/dev/table/sql/queries/overview.md
@@ -444,5 +444,6 @@ Flink SQL> SELECT 'Hello World', 'It''s me';
 - [窗口 Top-N]({{< ref "docs/dev/table/sql/queries/window-topn" >}})
 - [去重]({{< ref "docs/dev/table/sql/queries/deduplication" >}})
 - [模式识别]({{< ref "docs/dev/table/sql/queries/match_recognize" >}})
+- [时间旅行]({{< ref "docs/dev/table/sql/queries/time-travel" >}})
 
 {{< top >}}
diff --git a/docs/content.zh/docs/dev/table/sql/queries/time-travel.md b/docs/content.zh/docs/dev/table/sql/queries/time-travel.md
new file mode 100644
index 00000000000..3a34ed2a628
--- /dev/null
+++ b/docs/content.zh/docs/dev/table/sql/queries/time-travel.md
@@ -0,0 +1,72 @@
+---
+title: 时间旅行
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 时间旅行
+
+{{< label Batch >}} {{< label Streaming >}}
+
+`时间旅行`语法主要用于查询历史数据。它允许用户指定一个时间点,查询对应时间点 table 的数据。
+
+<span class="label label-danger">注意</span> 目前, `时间旅行`语法需要查询 table 所属的 catalog 实现了 {{< gh_link file="flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Catalog.java" name="getTable(ObjectPath tablePath, long timestamp)" >}} 接口。
+更多详细信息,请参考 [Catalog]({{< ref "docs/dev/table/catalogs" >}}#catalog-中支持时间旅行的接口)。
+
+带有时间旅行子句的语法如下所示:
+```sql
+SELECT select_list FROM table_name FOR SYSTEM_TIME AS OF timestamp_expression
+```
+
+**参数说明:**
+
+- `FOR SYSTEM_TIME AS OF timestamp_expression`:用于特定的时间表达式,用于查询该时间点之前的数据。`timestamp_expression` 用于表示需要查询的时间点。`timestamp_expression` 可以是一个具体的 `TIMESTAMP` 常量或者是可以简化为 `TIMESTAMP` 常量的时间表达式,该表达式只能作用于物理表不能是视图或者子查询。
+
+## 示例
+ 
+```sql
+--使用时间常量
+SELECT select_list FROM paimon_tb FOR SYSTEM_TIME AS OF TIMESTAMP '2023-07-31 00:00:00'
+
+--使用可以转换为时间常量的时间表达式
+SELECT select_list FROM paimon_tb FOR SYSTEM_TIME AS OF TIMESTAMP '2023-07-31 00:00:00' - INTERVAL '1' DAY
+```
+
+## 限制
+
+<span class="label label-danger">注意</span> `时间旅行` 中使用的时间表达式只支持 `TIMESTAMP` 常量表达式, `TIMESTAMP` 加减运算, 以及部分`内置函数`和 `UDF`
+
+当时间表达式中使用 `UDF` 或者`函数`时, 由于当前框架的限制,部分表达式无法在 SQL 解析时直接换为一个 `TIMESTAMP` 常量,会直接抛出异常。
+
+```sql
+--使用无法在 SQL 解析时直接转换为时间常量的时间表达式
+SELECT select_list FROM paimon_tb FOR SYSTEM_TIME AS OF TO_TIMESTAMP_LTZ(0, 3)
+```
+
+对应的异常如下: 
+
+```bash
+Unsupported time travel expression: TO_TIMESTAMP_LTZ(0, 3) for the expression can not be reduced to a constant by Flink.
+```
+
+## 时区的处理
+
+`TIMESTAMP` 表达式生成的数据类型是 `TIMESTAMP` 类型,但在时间旅行语句中有一个特殊情况。执行时间旅行语句时,框架会根据`本地时区`将 `TIMESTAMP` 类型转换为 `LONG` 类型。
+因此,在不同的时区执行相同的时间旅行查询语句可能会得到不同的结果。
diff --git a/docs/content/docs/dev/table/catalogs.md b/docs/content/docs/dev/table/catalogs.md
index 2a1994ac685..4d691b61655 100644
--- a/docs/content/docs/dev/table/catalogs.md
+++ b/docs/content/docs/dev/table/catalogs.md
@@ -71,6 +71,44 @@ The provided factory identifier will be used for matching against the required `
 User-defined catalogs should replace `Thread.currentThread().getContextClassLoader()` with the user class loader to load classes. Otherwise, `ClassNotFoundException` maybe thrown. The user class loader can be accessed via `CatalogFactory.Context#getClassLoader`.
 {{< /hint >}}
 
+#### Interface in Catalog for supporting time travel
+
+Starting from version 1.18, the Flink framework supports [time travel]({{< ref "docs/dev/table/sql/queries/time-travel" >}}) to query historical data of a table. To query the historical data of a table, users should implement `getTable(ObjectPath tablePath, long timestamp)` method for the catalog that the table belongs to.
+
+```java
+public class MyCatalogSupportTimeTravel implements Catalog {
+
+    @Override
+    public CatalogBaseTable getTable(ObjectPath tablePath, long timestamp)
+            throws TableNotExistException {
+        // Build a schema corresponding to the specific time point.
+        Schema schema = buildSchema(timestamp);
+        // Set parameters to read data at the corresponding time point.
+        Map<String, String> options = buildOptions(timestamp);
+        // Build CatalogTable
+        CatalogTable catalogTable =
+                CatalogTable.of(schema, "", Collections.emptyList(), options, timestamp);
+        return catalogTable;
+    }
+}
+
+public class MyDynamicTableFactory implements DynamicTableSourceFactory {
+    @Override
+    public DynamicTableSource createDynamicTableSource(Context context) {
+        final ReadableConfig configuration =
+                Configuration.fromMap(context.getCatalogTable().getOptions());
+
+        // Get snapshot from CatalogTable
+        final Optional<Long> snapshot = context.getCatalogTable().getSnapshot();
+
+        // Build DynamicTableSource using snapshot options.
+        final DynamicTableSource dynamicTableSource = buildDynamicSource(configuration, snapshot);
+
+        return dynamicTableSource;
+    }
+}
+```
+
 ## How to Create and Register Flink Tables to Catalog
 
 ### Using SQL DDL
diff --git a/docs/content/docs/dev/table/sql/queries/overview.md b/docs/content/docs/dev/table/sql/queries/overview.md
index 6ff8b101db8..681fdb31f37 100644
--- a/docs/content/docs/dev/table/sql/queries/overview.md
+++ b/docs/content/docs/dev/table/sql/queries/overview.md
@@ -444,5 +444,6 @@ Unicode characters are supported in string literals. If explicit unicode code po
 - [Window Top-N]({{< ref "docs/dev/table/sql/queries/window-topn" >}})
 - [Deduplication]({{< ref "docs/dev/table/sql/queries/deduplication" >}})
 - [Pattern Recognition]({{< ref "docs/dev/table/sql/queries/match_recognize" >}})
+- [Time Travel]({{< ref "docs/dev/table/sql/queries/time-travel" >}})
 
 {{< top >}}
diff --git a/docs/content/docs/dev/table/sql/queries/time-travel.md b/docs/content/docs/dev/table/sql/queries/time-travel.md
new file mode 100644
index 00000000000..43a0a38ef84
--- /dev/null
+++ b/docs/content/docs/dev/table/sql/queries/time-travel.md
@@ -0,0 +1,75 @@
+---
+title: Time Travel
+weight: 18
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Time Travel
+
+{{< label Batch >}} {{< label Streaming >}}
+
+The syntax of `time travel` is used for querying historical data. It allows users to specify a point in time and query the corresponding table data.
+
+<span class="label label-danger">Attention</span> Currently, `time travel` requires the corresponding catalog that the table belongs to implementing the {{< gh_link file="flink-table/flink-table-common/src/main/java/org/apache/flink/table/catalog/Catalog.java" name="getTable(ObjectPath tablePath, long timestamp)" >}} method.
+See more details in [Catalog]({{< ref "/docs/dev/table/catalogs" >}}#interface-in-catalog-for-supporting-time-travel).
+
+The syntax with time travel clause is:
+```sql
+SELECT select_list FROM table_name FOR SYSTEM_TIME AS OF timestamp_expression
+```
+
+**Parameter Specification:**
+
+- `FOR SYSTEM_TIME AS OF timestamp_expression`:Used to query data at a specific point in time, the `timestamp_expression` represents the historical time point you want to query. The `timestamp_expression` can be a specific timestamp or a time-related expression that can be reduced to a constant, and this expression can only apply to physical tables and not to views or sub-queries.
+
+## Example
+
+```sql
+--use timestamp constant expression
+SELECT select_list FROM paimon_tb FOR SYSTEM_TIME AS OF TIMESTAMP '2023-07-31 00:00:00'
+
+--use expression with functions that can be reduced to a timestamp constant
+SELECT select_list FROM paimon_tb FOR SYSTEM_TIME AS OF TIMESTAMP '2023-07-31 00:00:00' - INTERVAL '1' DAY
+```
+
+## Limitation
+
+<span class="label label-danger">Attention</span> The `timestamp_expression` used in `time travel` only supports certain types of expressions that can be reduced to TIMESTAMP constants, including constant expressions of type `TIMESTAMP`, adding and subtracting operations on timestamps, as well as some partial built-in functions and UDFs.
+
+When `UDFs` are used in a `timestamp_expression`, due to the limitations of the current framework,
+some expressions cannot be directly reduced to a `TIMESTAMP` constant during SQL parsing and an exception will be thrown.
+
+```sql
+--use expression with functions that can not be reduced to a timestamp constant
+SELECT select_list FROM paimon_tb FOR SYSTEM_TIME AS OF TO_TIMESTAMP_LTZ(0, 3)
+```
+
+The corresponding exceptions are as follows:
+
+```bash
+Unsupported time travel expression: TO_TIMESTAMP_LTZ(0, 3) for the expression can not be reduced to a constant by Flink.
+```
+
+## Time Zone Handling
+
+The data type generated by the TIMESTAMP expression is TIMESTAMP type, but there's a special case in the time travel clause.
+When encountering the time travel clause, the framework will convert the TIMESTAMP type to the LONG type based on the local time zone.
+Therefore, the results of the same time travel query statement may vary in different time zones.