You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2023/03/23 05:00:33 UTC
[kyuubi] branch branch-1.7 updated: [KYUUBI #4583] [DOCS] Rename Flink Table Store to Apache Paimon (Incubating)
This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.7
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.7 by this push:
new 258e741f8 [KYUUBI #4583] [DOCS] Rename Flink Table Store to Apache Paimon (Incubating)
258e741f8 is described below
commit 258e741f8549865fafffeb0c312f1a4fb283152a
Author: guanhua.lgh <gu...@alibaba-inc.com>
AuthorDate: Thu Mar 23 13:00:08 2023 +0800
[KYUUBI #4583] [DOCS] Rename Flink Table Store to Apache Paimon (Incubating)
… `Connectors for Spark SQL Query Engine `
### _Why are the changes needed?_
To update docs.
Update Flink Table Store to Apache Paimon (Incubating) in docs `Connectors for Spark SQL Query Engine `
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #4583 from huage1994/docss-3067.
Closes #4583
5eaf76b84 [guanhua.lgh] [DOC] Update Flink Table Store to Apache Paimon (Incubating) in docs `Connectors for Spark SQL Query Engine `
Authored-by: guanhua.lgh <gu...@alibaba-inc.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
(cherry picked from commit 3771dc9bf221513fe37103e58aa27fbb52263481)
Signed-off-by: Cheng Pan <ch...@apache.org>
---
docs/connector/spark/flink_table_store.rst | 90 -----------------------
docs/connector/spark/index.rst | 4 +-
docs/connector/spark/paimon.rst | 110 +++++++++++++++++++++++++++++
3 files changed, 112 insertions(+), 92 deletions(-)
diff --git a/docs/connector/spark/flink_table_store.rst b/docs/connector/spark/flink_table_store.rst
deleted file mode 100644
index ee4c2b352..000000000
--- a/docs/connector/spark/flink_table_store.rst
+++ /dev/null
@@ -1,90 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-`Flink Table Store`_
-==========
-
-Flink Table Store is a unified storage to build dynamic tables for both streaming and batch processing in Flink,
-supporting high-speed data ingestion and timely data query.
-
-.. tip::
- This article assumes that you have mastered the basic knowledge and operation of `Flink Table Store`_.
- For the knowledge about Flink Table Store not mentioned in this article,
- you can obtain it from its `Official Documentation`_.
-
-By using kyuubi, we can run SQL queries towards Flink Table Store which is more
-convenient, easy to understand, and easy to expand than directly using
-spark to manipulate Flink Table Store.
-
-Flink Table Store Integration
--------------------
-
-To enable the integration of kyuubi spark sql engine and Flink Table Store through
-Apache Spark Datasource V2 and Catalog APIs, you need to:
-
-- Referencing the Flink Table Store :ref:`dependencies<spark-flink-table-store-deps>`
-- Setting the spark extension and catalog :ref:`configurations<spark-flink-table-store-conf>`
-
-.. _spark-flink-table-store-deps:
-
-Dependencies
-************
-
-The **classpath** of kyuubi spark sql engine with Flink Table Store supported consists of
-
-1. kyuubi-spark-sql-engine-\ |release|\ _2.12.jar, the engine jar deployed with Kyuubi distributions
-2. a copy of spark distribution
-3. flink-table-store-spark-<version>.jar (example: flink-table-store-spark-0.2.jar), which can be found in the `Maven Central`_
-
-In order to make the Flink Table Store packages visible for the runtime classpath of engines, we can use one of these methods:
-
-1. Put the Flink Table Store packages into ``$SPARK_HOME/jars`` directly
-2. Set ``spark.jars=/path/to/flink-table-store-spark``
-
-.. warning::
- Please mind the compatibility of different Flink Table Store and Spark versions, which can be confirmed on the page of `Flink Table Store multi engine support`_.
-
-.. _spark-flink-table-store-conf:
-
-Configurations
-**************
-
-To activate functionality of Flink Table Store, we can set the following configurations:
-
-.. code-block:: properties
-
- spark.sql.catalog.tablestore=org.apache.flink.table.store.spark.SparkCatalog
- spark.sql.catalog.tablestore.warehouse=file:/tmp/warehouse
-
-Flink Table Store Operations
-------------------
-
-Flink Table Store supports reading table store tables through Spark.
-A common scenario is to write data with Flink and read data with Spark.
-You can follow this document `Flink Table Store Quick Start`_ to write data to a table store table
-and then use kyuubi spark sql engine to query the table with the following SQL ``SELECT`` statement.
-
-
-.. code-block:: sql
-
- select * from table_store.default.word_count;
-
-
-
-.. _Flink Table Store: https://nightlies.apache.org/flink/flink-table-store-docs-stable/
-.. _Flink Table Store Quick Start: https://nightlies.apache.org/flink/flink-table-store-docs-stable/docs/try-table-store/quick-start/
-.. _Official Documentation: https://nightlies.apache.org/flink/flink-table-store-docs-stable/
-.. _Maven Central: https://mvnrepository.com/artifact/org.apache.flink
-.. _Flink Table Store multi engine support: https://nightlies.apache.org/flink/flink-table-store-docs-stable/docs/engines/overview/
diff --git a/docs/connector/spark/index.rst b/docs/connector/spark/index.rst
index 790e804f2..d1503443c 100644
--- a/docs/connector/spark/index.rst
+++ b/docs/connector/spark/index.rst
@@ -23,7 +23,7 @@ By default, it provides accessibility to hive warehouses with various file forma
supported, such as parquet, orc, json, etc.
Also，it can easily integrate with other third-party libraries, such as Hudi,
-Iceberg, Delta Lake, Kudu, Flink Table Store, HBase，Cassandra, etc.
+Iceberg, Delta Lake, Kudu, Apache Paimon (Incubating), HBase，Cassandra, etc.
We also provide sample data sources like TDC-DS, TPC-H for testing and benchmarking
purpose.
@@ -37,7 +37,7 @@ purpose.
iceberg
kudu
hive
- flink_table_store
+ paimon
tidb
tpcds
tpch
diff --git a/docs/connector/spark/paimon.rst b/docs/connector/spark/paimon.rst
new file mode 100644
index 000000000..14e741955
--- /dev/null
+++ b/docs/connector/spark/paimon.rst
@@ -0,0 +1,110 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+
+`Apache Paimon (Incubating)`_
+==========
+
+Apache Paimon(incubating) is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics.
+
+.. tip::
+ This article assumes that you have mastered the basic knowledge and operation of `Apache Paimon (Incubating)`_.
+ For the knowledge about Apache Paimon (Incubating) not mentioned in this article,
+ you can obtain it from its `Official Documentation`_.
+
+By using kyuubi, we can run SQL queries towards Apache Paimon (Incubating) which is more
+convenient, easy to understand, and easy to expand than directly using
+spark to manipulate Apache Paimon (Incubating).
+
+Apache Paimon (Incubating) Integration
+-------------------
+
+To enable the integration of kyuubi spark sql engine and Apache Paimon (Incubating), you need to set the following configurations:
+
+- Referencing the Apache Paimon (Incubating) :ref:`dependencies<spark-paimon-deps>`
+- Setting the spark extension and catalog :ref:`configurations<spark-paimon-conf>`
+
+.. _spark-paimon-deps:
+
+Dependencies
+************
+
+The **classpath** of kyuubi spark sql engine with Apache Paimon (Incubating) consists of
+
+1. kyuubi-spark-sql-engine-\ |release|\ _2.12.jar, the engine jar deployed with Kyuubi distributions
+2. a copy of spark distribution
+3. paimon-spark-<version>.jar (example: paimon-spark-3.3-0.4-20230323.002035-5.jar), which can be found in the `Apache Paimon (Incubating) Supported Engines Spark3`_
+
+In order to make the Apache Paimon (Incubating) packages visible for the runtime classpath of engines, we can use one of these methods:
+
+1. Put the Apache Paimon (Incubating) packages into ``$SPARK_HOME/jars`` directly
+2. Set ``spark.jars=/path/to/paimon-spark-<version>.jar``
+
+.. warning::
+ Please mind the compatibility of different Apache Paimon (Incubating) and Spark versions, which can be confirmed on the page of `Apache Paimon (Incubating) multi engine support`_.
+
+.. _spark-paimon-conf:
+
+Configurations
+**************
+
+To activate functionality of Apache Paimon (Incubating), we can set the following configurations:
+
+.. code-block:: properties
+
+ spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog
+ spark.sql.catalog.paimon.warehouse=file:/tmp/paimon
+
+Apache Paimon (Incubating) Operations
+------------------
+
+
+Taking ``CREATE NAMESPACE`` as a example,
+
+.. code-block:: sql
+
+ CREATE DATABASE paimon.default;
+ USE paimon.default;
+
+Taking ``CREATE TABLE`` as a example,
+
+.. code-block:: sql
+
+ create table my_table (
+ k int,
+ v string
+ ) tblproperties (
+ 'primary-key' = 'k'
+ );
+
+Taking ``SELECT`` as a example,
+
+.. code-block:: sql
+
+ SELECT * FROM my_table;
+
+
+Taking ``INSERT`` as a example,
+
+.. code-block:: sql
+
+ INSERT INTO my_table VALUES (1, 'Hi Again'), (3, 'Test');
+
+
+
+
+.. _Apache Paimon (Incubating): https://paimon.apache.org/
+.. _Official Documentation: https://paimon.apache.org/docs/master/
+.. _Apache Paimon (Incubating) Supported Engines Spark3: https://paimon.apache.org/docs/master/engines/spark3/
+.. _Apache Paimon (Incubating) multi engine support: https://paimon.apache.org/docs/master/engines/overview/