You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@paimon.apache.org by lz...@apache.org on 2023/12/04 02:04:57 UTC
(incubator-paimon) branch master updated (a1c03229d -> 631a24198)
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
from a1c03229d [docs] Fix docs change "-" to "_" (#2439)
new bbbe01f31 fix
new 631a24198 [doc] Merge Spark2 into Spark3
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/content/_index.md | 3 +-
docs/content/engines/{spark3.md => spark.md} | 52 +++++++++++++++-
docs/content/engines/spark2.md | 93 ----------------------------
docs/content/how-to/creating-catalogs.md | 2 +-
4 files changed, 53 insertions(+), 97 deletions(-)
rename docs/content/engines/{spark3.md => spark.md} (92%)
delete mode 100644 docs/content/engines/spark2.md
(incubator-paimon) 02/02: [doc] Merge Spark2 into Spark3
Posted by lz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
commit 631a241987ac9d548ff491d99f292c891349c865
Author: Jingsong <ji...@gmail.com>
AuthorDate: Mon Dec 4 10:04:47 2023 +0800
[doc] Merge Spark2 into Spark3
---
docs/content/_index.md | 1 +
docs/content/engines/spark.md | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/content/_index.md b/docs/content/_index.md
index 59818f371..d00771e0a 100644
--- a/docs/content/_index.md
+++ b/docs/content/_index.md
@@ -35,6 +35,7 @@ Paimon offers the following core capabilities:
- Append Only Tables: Paimon supports Append Only tables, automatically compact small files, and provides orderly stream reading. You can use this to replace message queues.
{{< columns >}}
+
## Try Paimon
If you’re interested in playing around with Paimon, check out our
diff --git a/docs/content/engines/spark.md b/docs/content/engines/spark.md
index 8c75e3efb..c6f5bbf08 100644
--- a/docs/content/engines/spark.md
+++ b/docs/content/engines/spark.md
@@ -605,7 +605,7 @@ All Spark's data types are available in package `org.apache.spark.sql.types`.
## Spark 2
-Paimon supports Spark 2.4+.
+Paimon supports Spark 2.4+. We highly recommend using versions above Spark3, as Spark2 only provides reading capabilities.
{{< stable >}}
(incubator-paimon) 01/02: fix
Posted by lz...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-paimon.git
commit bbbe01f31033364b1f186527f6cf65691d3542f9
Author: Jingsong <ji...@gmail.com>
AuthorDate: Mon Dec 4 10:02:43 2023 +0800
fix
---
docs/content/_index.md | 2 +-
docs/content/engines/{spark3.md => spark.md} | 52 +++++++++++++++-
docs/content/engines/spark2.md | 93 ----------------------------
docs/content/how-to/creating-catalogs.md | 2 +-
4 files changed, 52 insertions(+), 97 deletions(-)
diff --git a/docs/content/_index.md b/docs/content/_index.md
index f86466258..59818f371 100644
--- a/docs/content/_index.md
+++ b/docs/content/_index.md
@@ -38,7 +38,7 @@ Paimon offers the following core capabilities:
## Try Paimon
If you’re interested in playing around with Paimon, check out our
-quick start guide with [Flink]({{< ref "engines/flink" >}}), [Spark]({{< ref "engines/spark3" >}}) or [Hive]({{< ref "engines/hive" >}}). It provides a step by
+quick start guide with [Flink]({{< ref "engines/flink" >}}), [Spark]({{< ref "engines/spark" >}}) or [Hive]({{< ref "engines/hive" >}}). It provides a step by
step introduction to the APIs and guides you through real applications.
<--->
diff --git a/docs/content/engines/spark3.md b/docs/content/engines/spark.md
similarity index 92%
rename from docs/content/engines/spark3.md
rename to docs/content/engines/spark.md
index ade2ebc74..8c75e3efb 100644
--- a/docs/content/engines/spark3.md
+++ b/docs/content/engines/spark.md
@@ -1,9 +1,9 @@
---
-title: "Spark3"
+title: "Spark"
weight: 3
type: docs
aliases:
-- /engines/spark3.html
+- /engines/spark.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -602,3 +602,51 @@ All Spark's data types are available in package `org.apache.spark.sql.types`.
- Conversion between Spark's `UserDefinedType` and Paimon's `UserDefinedType` is not supported.
{{< /hint >}}
+
+## Spark 2
+
+Paimon supports Spark 2.4+.
+
+{{< stable >}}
+
+Download [paimon-spark-2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-2/{{< version >}}/paimon-spark-2-{{< version >}}.jar).
+
+{{< /stable >}}
+
+{{< unstable >}}
+
+Download [paimon-spark-2-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-2/{{< version >}}/).
+
+{{< /unstable >}}
+
+{{< hint info >}}
+
+If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set.
+
+{{< /hint >}}
+
+**Step 1: Prepare Test Data**
+
+Paimon currently only supports reading tables through Spark2. To create a Paimon table with records, please follow our [Flink quick start guide]({{< ref "engines/flink#quick-start" >}}).
+
+After the guide, all table files should be stored under the path `/tmp/paimon`, or the warehouse path you've specified.
+
+**Step 2: Specify Paimon Jar File**
+
+You can append path to paimon jar file to the `--jars` argument when starting `spark-shell`.
+
+```bash
+spark-shell ... --jars /path/to/paimon-spark-2-{{< version >}}.jar
+```
+
+Alternatively, you can copy `paimon-spark-2-{{< version >}}.jar` under `spark/jars` in your Spark installation directory.
+
+**Step 3: Query Table**
+
+Paimon with Spark 2.4 does not support DDL. You can use the `Dataset` reader and register the `Dataset` as a temporary table. In spark shell:
+
+```scala
+val dataset = spark.read.format("paimon").load("file:/tmp/paimon/default.db/word_count")
+dataset.createOrReplaceTempView("word_count")
+spark.sql("SELECT * FROM word_count").show()
+```
diff --git a/docs/content/engines/spark2.md b/docs/content/engines/spark2.md
deleted file mode 100644
index 46e736a7f..000000000
--- a/docs/content/engines/spark2.md
+++ /dev/null
@@ -1,93 +0,0 @@
----
-title: "Spark2"
-weight: 4
-type: docs
-aliases:
-- /engines/spark2.html
----
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Spark2
-
-This documentation is a guide for using Paimon in Spark2.
-
-## Version
-
-Paimon supports Spark 2.4+. It is highly recommended to use Spark 2.4+ version with much improvement.
-
-## Preparing Paimon Jar File
-
-{{< stable >}}
-
-Download [paimon-spark-2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-2/{{< version >}}/paimon-spark-2-{{< version >}}.jar).
-
-{{< /stable >}}
-
-{{< unstable >}}
-
-Download [paimon-spark-2-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-2/{{< version >}}/).
-
-{{< /unstable >}}
-
-You can also manually build bundled jar from the source code.
-
-To build from source code, [clone the git repository]({{< github_repo >}}).
-
-Build bundled jar with the following command.
-
-```bash
-mvn clean install -DskipTests
-```
-
-You can find the bundled jar in `./paimon-spark/paimon-spark-2/target/paimon-spark-2-{{< version >}}.jar`.
-
-## Quick Start
-
-{{< hint info >}}
-
-If you are using HDFS, make sure that the environment variable `HADOOP_HOME` or `HADOOP_CONF_DIR` is set.
-
-{{< /hint >}}
-
-**Step 1: Prepare Test Data**
-
-Paimon currently only supports reading tables through Spark2. To create a Paimon table with records, please follow our [Flink quick start guide]({{< ref "engines/flink#quick-start" >}}).
-
-After the guide, all table files should be stored under the path `/tmp/paimon`, or the warehouse path you've specified.
-
-**Step 2: Specify Paimon Jar File**
-
-You can append path to paimon jar file to the `--jars` argument when starting `spark-shell`.
-
-```bash
-spark-shell ... --jars /path/to/paimon-spark-2-{{< version >}}.jar
-```
-
-Alternatively, you can copy `paimon-spark-2-{{< version >}}.jar` under `spark/jars` in your Spark installation directory.
-
-**Step 3: Query Table**
-
-Paimon with Spark 2.4 does not support DDL. You can use the `Dataset` reader and register the `Dataset` as a temporary table. In spark shell:
-
-```scala
-val dataset = spark.read.format("paimon").load("file:/tmp/paimon/default.db/word_count")
-dataset.createOrReplaceTempView("word_count")
-spark.sql("SELECT * FROM word_count").show()
-```
diff --git a/docs/content/how-to/creating-catalogs.md b/docs/content/how-to/creating-catalogs.md
index 19f11a2a9..a3de5df31 100644
--- a/docs/content/how-to/creating-catalogs.md
+++ b/docs/content/how-to/creating-catalogs.md
@@ -140,7 +140,7 @@ After `spark-sql` is started, you can switch to the `default` database of the `p
USE paimon.default;
```
-Also, you can create [SparkGenericCatalog]({{< ref "engines/spark3" >}}).
+Also, you can create [SparkGenericCatalog]({{< ref "engines/spark" >}}).
{{< /tab >}}