You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by zy...@apache.org on 2023/06/06 15:33:34 UTC
[doris] branch master updated: [typo](doc) update spark connnector version compatibility instructions (#20477)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 880e2d8373 [typo](doc) update spark connnector version compatibility instructions (#20477)
880e2d8373 is described below
commit 880e2d8373ef5b51ae44a0aad65430eb23bba6f1
Author: gnehil <ad...@gmail.com>
AuthorDate: Tue Jun 6 23:33:27 2023 +0800
[typo](doc) update spark connnector version compatibility instructions (#20477)
---
docs/en/docs/ecosystem/spark-doris-connector.md | 21 ++++++------
docs/zh-CN/docs/ecosystem/spark-doris-connector.md | 39 +++++++++++-----------
2 files changed, 29 insertions(+), 31 deletions(-)
diff --git a/docs/en/docs/ecosystem/spark-doris-connector.md b/docs/en/docs/ecosystem/spark-doris-connector.md
index 0c86b9ddce..2d9a116006 100644
--- a/docs/en/docs/ecosystem/spark-doris-connector.md
+++ b/docs/en/docs/ecosystem/spark-doris-connector.md
@@ -37,11 +37,10 @@ Github: https://github.com/apache/doris-spark-connector
## Version Compatibility
-| Connector | Spark | Doris | Java | Scala |
-|---------------| ----- | ------ | ---- | ----- |
-| 2.3.4-2.11.xx | 2.x | 0.12+ | 8 | 2.11 |
-| 3.1.2-2.12.xx | 3.x | 0.12.+ | 8 | 2.12 |
-| 3.2.0-2.12.xx | 3.2.x | 0.12.+ | 8 | 2.12 |
+| Connector | Spark | Doris | Java | Scala |
+| --------- | ------------- |-------------| ---- | ---------- |
+| 1.1.0 | 3.2, 3.1, 2.3 | 1.0 + | 8 | 2.12, 2.11 |
+| 1.0.1 | 3.1, 2.3 | 0.12 - 0.15 | 8 | 2.12, 2.11 |
## Build and Install
@@ -58,14 +57,14 @@ Copy this file to `ClassPath` in `Spark` to use `Spark-Doris-Connector`. For exa
For example upload `spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar` to hdfs and add hdfs file path in spark.yarn.jars.
-1. Upload spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar Jar to hdfs.
+1. Upload `spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar` Jar to hdfs.
```
hdfs dfs -mkdir /spark-jars/
hdfs dfs -put /your_local_path/spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar /spark-jars/
```
-2. Add spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar depence in Cluster.
+2. Add `spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar` dependency in Cluster.
```
spark.yarn.jars=hdfs:///spark-jars/spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar
@@ -109,7 +108,7 @@ SELECT * FROM spark_doris;
```scala
val dorisSparkDF = spark.read.format("doris")
.option("doris.table.identifier", "$YOUR_DORIS_DATABASE_NAME.$YOUR_DORIS_TABLE_NAME")
- .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
+ .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
.load()
@@ -141,7 +140,7 @@ dorisSparkDF = spark.read.format("doris")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
.load()
-# show 5 lines data
+// show 5 lines data
dorisSparkDF.show(5)
```
@@ -176,7 +175,7 @@ mockDataDF.show(5)
mockDataDF.write.format("doris")
.option("doris.table.identifier", "$YOUR_DORIS_DATABASE_NAME.$YOUR_DORIS_TABLE_NAME")
- .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
+ .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
//other options
@@ -196,7 +195,7 @@ kafkaSource.selectExpr("CAST(key AS STRING)", "CAST(value as STRING)")
.format("doris")
.option("checkpointLocation", "$YOUR_CHECKPOINT_LOCATION")
.option("doris.table.identifier", "$YOUR_DORIS_DATABASE_NAME.$YOUR_DORIS_TABLE_NAME")
- .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
+ .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
//other options
diff --git a/docs/zh-CN/docs/ecosystem/spark-doris-connector.md b/docs/zh-CN/docs/ecosystem/spark-doris-connector.md
index c64c23cb9c..8c8d044f56 100644
--- a/docs/zh-CN/docs/ecosystem/spark-doris-connector.md
+++ b/docs/zh-CN/docs/ecosystem/spark-doris-connector.md
@@ -37,11 +37,10 @@ Spark Doris Connector 可以支持通过 Spark 读取 Doris 中存储的数据
## 版本兼容
-| Connector | Spark | Doris | Java | Scala |
-|---------------| ----- | ------ | ---- | ----- |
-| 2.3.4-2.11.xx | 2.x | 0.12+ | 8 | 2.11 |
-| 3.1.2-2.12.xx | 3.x | 0.12.+ | 8 | 2.12 |
-| 3.2.0-2.12.xx | 3.2.x | 0.12.+ | 8 | 2.12 |
+| Connector | Spark | Doris | Java | Scala |
+| --------- | ------------- |-------------| ---- | ---------- |
+| 1.1.0 | 3.2, 3.1, 2.3 | 1.0 + | 8 | 2.12, 2.11 |
+| 1.0.1 | 3.1, 2.3 | 0.12 - 0.15 | 8 | 2.12, 2.11 |
## 编译与安装
@@ -53,22 +52,24 @@ Spark Doris Connector 可以支持通过 Spark 读取 Doris 中存储的数据
`sh build.sh`
根据提示输入你需要的 Scala 与 Spark 版本进行编译。
-编译成功后，会在 `dist` 目录生成目标jar包，如：`spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar`。
-将此文件复制到 `Spark` 的 `ClassPath` 中即可使用 `Spark-Doris-Connector`。例如，`Local` 模式运行的 `Spark`，将此文件放入 `jars/` 文件夹下。`Yarn`集群模式运行的`Spark`，则将此文件放入预部署包中。
+编译成功后，会在 `dist` 目录生成目标jar包，如：`spark-doris-connector-3.2_2.12-1.1.0-SNAPSHOT.jar`。
+将此文件复制到 `Spark` 的 `ClassPath` 中即可使用 `Spark-Doris-Connector`。
-例如将 `spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar` 上传到 hdfs并在spark.yarn.jars参数上添加 hdfs上的Jar包路径
+例如，`Local` 模式运行的 `Spark`，将此文件放入 `jars/` 文件夹下。`Yarn`集群模式运行的`Spark`，则将此文件放入预部署包中。
-1. 上传spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar 到hdfs。
+例如将 `spark-doris-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` 上传到 hdfs 并在 `spark.yarn.jars` 参数上添加 hdfs 上的 Jar 包路径
+
+1. 上传 `spark-doris-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` 到hdfs。
```
hdfs dfs -mkdir /spark-jars/
-hdfs dfs -put /your_local_path/spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar /spark-jars/
+hdfs dfs -put /your_local_path/spark-doris-connector-3.2_2.12-1.1.0-SNAPSHOT.jar /spark-jars/
```
-2. 在集群中添加spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar 依赖。
+2. 在集群中添加 `spark-doris-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` 依赖。
```
-spark.yarn.jars=hdfs:///spark-jars/spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT.jar
+spark.yarn.jars=hdfs:///spark-jars/spark-doris-connector-3.2_2.12-1.1.0-SNAPSHOT.jar
```
## 使用Maven管理
@@ -76,7 +77,7 @@ spark.yarn.jars=hdfs:///spark-jars/spark-doris-connector-3.1_2.12-1.1.0-SNAPSHOT
```
<dependency>
<groupId>org.apache.doris</groupId>
- <artifactId>spark-doris-connector-3.1_2.12</artifactId>
+ <artifactId>spark-doris-connector-3.2_2.12</artifactId>
<version>1.1.0</version>
</dependency>
```
@@ -108,7 +109,7 @@ SELECT * FROM spark_doris;
```scala
val dorisSparkDF = spark.read.format("doris")
.option("doris.table.identifier", "$YOUR_DORIS_DATABASE_NAME.$YOUR_DORIS_TABLE_NAME")
- .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
+ .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
.load()
@@ -141,12 +142,10 @@ dorisSparkDF = spark.read.format("doris")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
.load()
-# show 5 lines data
+// show 5 lines data
dorisSparkDF.show(5)
```
-
-
### 写入
#### SQL
@@ -179,7 +178,7 @@ mockDataDF.show(5)
mockDataDF.write.format("doris")
.option("doris.table.identifier", "$YOUR_DORIS_DATABASE_NAME.$YOUR_DORIS_TABLE_NAME")
- .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
+ .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
//其它选项
@@ -199,7 +198,7 @@ kafkaSource.selectExpr("CAST(key AS STRING)", "CAST(value as STRING)")
.format("doris")
.option("checkpointLocation", "$YOUR_CHECKPOINT_LOCATION")
.option("doris.table.identifier", "$YOUR_DORIS_DATABASE_NAME.$YOUR_DORIS_TABLE_NAME")
- .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
+ .option("doris.fenodes", "$YOUR_DORIS_FE_HOSTNAME:$YOUR_DORIS_FE_RESFUL_PORT")
.option("user", "$YOUR_DORIS_USERNAME")
.option("password", "$YOUR_DORIS_PASSWORD")
//其它选项
@@ -209,7 +208,7 @@ kafkaSource.selectExpr("CAST(key AS STRING)", "CAST(value as STRING)")
.awaitTermination()
```
-### java示例
+### Java示例
`samples/doris-demo/spark-demo/` 下提供了 Java 版本的示例，可供参考，[这里](https://github.com/apache/incubator-doris/tree/master/samples/doris-demo/spark-demo)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org