You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2023/12/04 02:04:58 UTC
(incubator-paimon) 01/02: fix
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
commit bbbe01f31033364b1f186527f6cf65691d3542f9
Author: Jingsong <ji...@gmail.com>
AuthorDate: Mon Dec 4 10:02:43 2023 +0800
fix
---
docs/content/_index.md | 2 +-
docs/content/engines/{spark3.md => spark.md} | 52 +++++++++++++++-
docs/content/engines/spark2.md | 93 ----------------------------
docs/content/how-to/creating-catalogs.md | 2 +-
4 files changed, 52 insertions(+), 97 deletions(-)
diff --git a/docs/content/_index.md b/docs/content/_index.md
index f86466258..59818f371 100644
--- a/docs/content/_index.md
+++ b/docs/content/_index.md
@@ -38,7 +38,7 @@ Paimon offers the following core capabilities:
## Try Paimon
If you’re interested in playing around with Paimon, check out our
-quick start guide with [Flink]({{< ref "engines/flink" >}}), [Spark]({{< ref "engines/spark3" >}}) or [Hive]({{< ref "engines/hive" >}}). It provides a step by
+quick start guide with [Flink]({{< ref "engines/flink" >}}), [Spark]({{< ref "engines/spark" >}}) or [Hive]({{< ref "engines/hive" >}}). It provides a step by
step introduction to the APIs and guides you through real applications.
<--->
diff --git a/docs/content/engines/spark3.md b/docs/content/engines/spark.md
similarity index 92%
rename from docs/content/engines/spark3.md
rename to docs/content/engines/spark.md
index ade2ebc74..8c75e3efb 100644
--- a/docs/content/engines/spark3.md
+++ b/docs/content/engines/spark.md
@@ -1,9 +1,9 @@
---
-title: "Spark3"
+title: "Spark"
weight: 3
type: docs
aliases:
-- /engines/spark3.html
+- /engines/spark.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -602,3 +602,51 @@ All Spark's data types are available in package `org.apache.spark.sql.types`.
- Conversion between Spark's `UserDefinedType` and Paimon's `UserDefinedType` is not supported.
{{< /hint >}}
+
+## Spark 2
+
+Paimon supports Spark 2.4+.
+
+{{< stable >}}
+
+Download [paimon-spark-2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-2/{{< version >}}/paimon-spark-2-{{< version >}}.jar).
+
+{{< /stable >}}
+
+{{< unstable >}}
+
+Download [paimon-spark-2-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-2/{{< version >}}/).
+
+{{< /unstable >}}
+
+{{< hint info >}}
+
+If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set.
+
+{{< /hint >}}
+
+**Step 1: Prepare Test Data**
+
+Paimon currently only supports reading tables through Spark2. To create a Paimon table with records, please follow our [Flink quick start guide]({{< ref "engines/flink#quick-start" >}}).
+
+After the guide, all table files should be stored under the path `/tmp/paimon`, or the warehouse path you've specified.
+
+**Step 2: Specify Paimon Jar File**
+
+You can append path to paimon jar file to the `--jars` argument when starting `spark-shell`.
+
+```bash
+spark-shell ... --jars /path/to/paimon-spark-2-{{< version >}}.jar
+```
+
+Alternatively, you can copy `paimon-spark-2-{{< version >}}.jar` under `spark/jars` in your Spark installation directory.
+
+**Step 3: Query Table**
+
+Paimon with Spark 2.4 does not support DDL. You can use the `Dataset` reader and register the `Dataset` as a temporary table. In spark shell:
+
+```scala
+val dataset = spark.read.format("paimon").load("file:/tmp/paimon/default.db/word_count")
+dataset.createOrReplaceTempView("word_count")
+spark.sql("SELECT * FROM word_count").show()
+```
diff --git a/docs/content/engines/spark2.md b/docs/content/engines/spark2.md
deleted file mode 100644
index 46e736a7f..000000000
--- a/docs/content/engines/spark2.md
+++ /dev/null
@@ -1,93 +0,0 @@
----
-title: "Spark2"
-weight: 4
-type: docs
-aliases:
-- /engines/spark2.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Spark2
-
-This documentation is a guide for using Paimon in Spark2.
-
-## Version
-
-Paimon supports Spark 2.4+. It is highly recommended to use Spark 2.4+ version with much improvement.
-
-## Preparing Paimon Jar File
-
-{{< stable >}}
-
-Download [paimon-spark-2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-2/{{< version >}}/paimon-spark-2-{{< version >}}.jar).
-
-{{< /stable >}}
-
-{{< unstable >}}
-
-Download [paimon-spark-2-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-2/{{< version >}}/).
-
-{{< /unstable >}}
-
-You can also manually build bundled jar from the source code.
-
-To build from source code, [clone the git repository]({{< github_repo >}}).
-
-Build bundled jar with the following command.
-
-```bash
-mvn clean install -DskipTests
-```
-
-You can find the bundled jar in `./paimon-spark/paimon-spark-2/target/paimon-spark-2-{{< version >}}.jar`.
-
-## Quick Start
-
-{{< hint info >}}
-
-If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set.
-
-{{< /hint >}}
-
-**Step 1: Prepare Test Data**
-
-Paimon currently only supports reading tables through Spark2. To create a Paimon table with records, please follow our [Flink quick start guide]({{< ref "engines/flink#quick-start" >}}).
-
-After the guide, all table files should be stored under the path `/tmp/paimon`, or the warehouse path you've specified.
-
-**Step 2: Specify Paimon Jar File**
-
-You can append path to paimon jar file to the `--jars` argument when starting `spark-shell`.
-
-```bash
-spark-shell ... --jars /path/to/paimon-spark-2-{{< version >}}.jar
-```
-
-Alternatively, you can copy `paimon-spark-2-{{< version >}}.jar` under `spark/jars` in your Spark installation directory.
-
-**Step 3: Query Table**
-
-Paimon with Spark 2.4 does not support DDL. You can use the `Dataset` reader and register the `Dataset` as a temporary table. In spark shell:
-
-```scala
-val dataset = spark.read.format("paimon").load("file:/tmp/paimon/default.db/word_count")
-dataset.createOrReplaceTempView("word_count")
-spark.sql("SELECT * FROM word_count").show()
-```
diff --git a/docs/content/how-to/creating-catalogs.md b/docs/content/how-to/creating-catalogs.md
index 19f11a2a9..a3de5df31 100644
--- a/docs/content/how-to/creating-catalogs.md
+++ b/docs/content/how-to/creating-catalogs.md
@@ -140,7 +140,7 @@ After `spark-sql` is started, you can switch to the `default` database of the `p
USE paimon.default;
```
-Also, you can create [SparkGenericCatalog]({{< ref "engines/spark3" >}}).
+Also, you can create [SparkGenericCatalog]({{< ref "engines/spark" >}}).
{{< /tab >}}