You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@linkis.apache.org by ca...@apache.org on 2023/06/21 04:41:44 UTC
[linkis-website] branch dev updated: Provide deployment instructions for S3 and PostgreSQL (#711)
This is an automated email from the ASF dual-hosted git repository.
casion pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/linkis-website.git
The following commit(s) were added to refs/heads/dev by this push:
new ca12cda6c9 Provide deployment instructions for S3 and PostgreSQL (#711)
ca12cda6c9 is described below
commit ca12cda6c9dcfb91b054f063211c61c2da1aec7a
Author: sjgllgh <12...@users.noreply.github.com>
AuthorDate: Wed Jun 21 12:41:40 2023 +0800
Provide deployment instructions for S3 and PostgreSQL (#711)
* Provide deployment instructions for S3 and PostgreSQL
* increasing version cue content
---
docs/deployment/deploy-quick.md | 69 +++++++++++++++++++---
.../current/deployment/deploy-quick.md | 63 +++++++++++++++++---
2 files changed, 115 insertions(+), 17 deletions(-)
diff --git a/docs/deployment/deploy-quick.md b/docs/deployment/deploy-quick.md
index aa9f497db6..19e2f036da 100644
--- a/docs/deployment/deploy-quick.md
+++ b/docs/deployment/deploy-quick.md
@@ -68,18 +68,36 @@ drwxrwxr-x 7 hadoop hadoop 4096 Jun 20 09:56 linkis-package // The actual p
````
-### 2.2 Configure database information
+### 2.2 Configure database
+
+`vim deploy-config/linkis-env.sh`
+
+```shell script
+# Select the type of Linkis business database, default is mysql.
+# If using PostgreSQL, please change it to postgresql.
+# Note: The configuration is only applicable to Linkis version 1.4.0 or higher.
+dbType=mysql
+```
`vim deploy-config/db.sh`
```shell script
-# Database information of Linkis' own business
+# Database information of Linkis' own business - mysql
MYSQL_HOST=xx.xx.xx.xx
MYSQL_PORT=3306
MYSQL_DB=linkis_test
MYSQL_USER=test
MYSQL_PASSWORD=xxxxx
+# Database information of Linkis' own business - postgresql
+# Note: The configurations is only applicable to Linkis version 1.4.0 or higher.
+PG_HOST=xx.xx.xx.xx
+PG_PORT=5432
+PG_DB=linkis_test
+PG_SCHEMA=linkis_test
+PG_USER=test
+PG_PASSWORD=123456
+
# Provide the DB information of the Hive metadata database. If the hive engine is not involved (or just a simple trial), you can not configure it
#Mainly used with scripts, if not configured, it will try to obtain it through the configuration file in $HIVE_CONF_DIR by default
HIVE_META_URL="jdbc:mysql://10.10.10.10:3306/hive_meta_demo?useUnicode=true&characterEncoding=UTF-8"
@@ -212,6 +230,27 @@ HADOOP_KERBEROS_ENABLE=true
HADOOP_KEYTAB_PATH=/appcom/keytab/
```
+#### S3 mode (optional)
+> Currently, it is possible to store engine execution logs and results to S3 in Linkis.
+>
+> Note: Linkis has not adapted permissions for S3, so it is not possible to grant authorization for it.
+
+`vim linkis.properties`
+```shell script
+# s3 file system
+linkis.storage.s3.access.key=xxx
+linkis.storage.s3.secret.key=xxx
+linkis.storage.s3.endpoint=http://xxx.xxx.xxx.xxx:xxx
+linkis.storage.s3.region=xxx
+linkis.storage.s3.bucket=xxx
+```
+
+`vim linkis-cg-entrance.properties`
+```shell script
+wds.linkis.entrance.config.log.path=s3:///linkis/logs
+wds.linkis.resultSet.store.path=s3:///linkis/results
+```
+
### 2.4 Configure Token
The original default Token of Linkis is fixed and the length is too short, which has security risks. Therefore, Linkis 1.3.2 changes the original fixed Token to random generation and increases the Token length.
@@ -308,38 +347,50 @@ cp mysql-connector-java-5.1.49.jar ${LINKIS_HOME}/lib/linkis-spring-cloud-servic
cp mysql-connector-java-5.1.49.jar ${LINKIS_HOME}/lib/linkis-commons/public-module/
````
-### 3.3 Configuration Adjustment (Optional)
+### 3.3 Add postgresql driver package (Optional)
+
+If you choose to use postgresql as the business database, you need to manually add the postgresql driver.
+
+To download the postgresql driver, take version 42.5.4 as an example: [download link](https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.4/postgresql-42.5.4.jar)
+
+Copy the postgresql driver package to the lib package
+````
+cp postgresql-42.5.4.jar ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis-mg-gateway/
+cp postgresql-42.5.4.jar ${LINKIS_HOME}/lib/linkis-commons/public-module/
+````
+
+### 3.4 Configuration Adjustment (Optional)
> The following operations are related to the dependent environment. According to the actual situation, determine whether the operation is required
-#### 3.3.1 kerberos authentication
+#### 3.4.1 kerberos authentication
If the hive cluster used has kerberos mode authentication enabled, modify the configuration `${LINKIS_HOME}/conf/linkis.properties` (<=1.1.3) file
```shell script
#Append the following configuration
echo "wds.linkis.keytab.enable=true" >> linkis.properties
````
-#### 3.3.2 Yarn Authentication
+#### 3.4.2 Yarn Authentication
When executing spark tasks, you need to use the ResourceManager of yarn, which is controlled by the configuration item `YARN_RESTFUL_URL=http://xx.xx.xx.xx:8088 `.
During installation and deployment, the `YARN_RESTFUL_URL=http://xx.xx.xx.xx:8088` information will be updated to the database table `linkis_cg_rm_external_resource_provider`. By default, access to yarn resources does not require permission verification.
If password authentication is enabled in yarn's ResourceManager, please modify the yarn data information generated in the database table `linkis_cg_rm_external_resource_provider` after installation and deployment.
For details, please refer to [Check whether the yarn address is configured correctly] (#811-Check whether the yarn address is configured correctly)
-#### 3.3.2 session
+#### 3.4.2 session
If you are upgrading to Linkis. Deploy DSS or other projects at the same time, but the dependent linkis version introduced in other software is <1.1.1 (mainly in the lib package, the linkis-module-x.x.x.jar package of the dependent Linkis is <1.1.1), you need to modify the linkis located in ` ${LINKIS_HOME}/conf/linkis.properties` file
```shell
echo "wds.linkis.session.ticket.key=bdp-user-ticket-id" >> linkis.properties
````
-### 3.4 Start the service
+### 3.5 Start the service
```shell script
sh sbin/linkis-start-all.sh
````
-### 3.5 Modification of post-installation configuration
+### 3.6 Modification of post-installation configuration
After the installation is complete, if you need to modify the configuration (because of port conflicts or some configuration problems, you need to adjust the configuration), you can re-execute the installation, or modify the configuration `${LINKIS_HOME}/conf/*properties` file of the corresponding service, Restart the corresponding service, such as: `sh sbin/linkis-daemon.sh start ps-publicservice`
-### 3.6 Check whether the service starts normally
+### 3.7 Check whether the service starts normally
Visit the eureka service page (http://eurekaip:20303),
The Linkis will start 6 microservices by default, and the linkis-cg-engineconn service in the figure below will be started only for running tasks
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/deploy-quick.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/deploy-quick.md
index e5f0a55edf..383d805cb8 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/deploy-quick.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/deployment/deploy-quick.md
@@ -70,16 +70,34 @@ drwxrwxr-x 7 hadoop hadoop 4096 Jun 20 09:56 linkis-package // 实际的软
### 2.2 配置数据库信息
+`vim deploy-config/linkis-env.sh`
+
+```shell script
+# 选择linkis业务数据库类型，默认mysql
+# 如果使用postgresql，请改为postgresql
+# 注意: 当前配置只适用于linkis>=1.4.0
+dbType=mysql
+```
+
`vim deploy-config/db.sh`
```shell script
-# Linkis自身业务的数据库信息
+# Linkis自身业务的数据库信息 - mysql
MYSQL_HOST=xx.xx.xx.xx
MYSQL_PORT=3306
MYSQL_DB=linkis_test
MYSQL_USER=test
MYSQL_PASSWORD=xxxxx
+# Linkis自身业务的数据库信息 - postgresql
+# 注意: 以下配置只适用于linkis>=1.4.0
+PG_HOST=xx.xx.xx.xx
+PG_PORT=5432
+PG_DB=linkis_test
+PG_SCHEMA=linkis_test
+PG_USER=test
+PG_PASSWORD=123456
+
# 提供 Hive 元数据数据库的 DB 信息，如果不涉及hive引擎（或则只是简单试用），可以不配置
#主要是配合scriptis一起使用，如果不配置，会默认尝试通过$HIVE_CONF_DIR 中的配置文件获取
HIVE_META_URL="jdbc:mysql://10.10.10.10:3306/hive_meta_demo?useUnicode=true&characterEncoding=UTF-8"
@@ -211,6 +229,28 @@ HADOOP_KERBEROS_ENABLE=true
HADOOP_KEYTAB_PATH=/appcom/keytab/
```
+
+#### S3模式（可选）
+> 目前支持将引擎执行日志和结果存储到S3
+>
+> 注意: linkis没有对S3做权限适配，所以无法对其做赋权操作
+
+`vim linkis.properties`
+```shell script
+# s3 file system
+linkis.storage.s3.access.key=xxx
+linkis.storage.s3.secret.key=xxx
+linkis.storage.s3.endpoint=http://xxx.xxx.xxx.xxx:xxx
+linkis.storage.s3.region=xxx
+linkis.storage.s3.bucket=xxx
+```
+
+`vim linkis-cg-entrance.properties`
+```shell script
+wds.linkis.entrance.config.log.path=s3:///linkis/logs
+wds.linkis.resultSet.store.path=s3:///linkis/results
+```
+
### 2.4 配置 Token
Linkis 原有默认 Token 固定且长度太短存在安全隐患。因此 Linkis 1.3.2 将原有固定 Token 改为随机生成，并增加 Token 长度。
@@ -306,33 +346,40 @@ Your default account password is [hadoop/5e8e312b4]`
cp mysql-connector-java-5.1.49.jar ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis-mg-gateway/
cp mysql-connector-java-5.1.49.jar ${LINKIS_HOME}/lib/linkis-commons/public-module/
```
-
-### 3.3 配置调整（可选）
+### 3.3 添加postgresql驱动包 (可选)
+如果选择使用postgresql作为业务数据库，需要手动添加postgresql驱动
+下载postgresql驱动 以42.5.4版本为例：[下载链接](https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.4/postgresql-42.5.4.jar)
+拷贝postgresql驱动包至lib包下
+```
+cp postgresql-42.5.4.jar ${LINKIS_HOME}/lib/linkis-spring-cloud-services/linkis-mg-gateway/
+cp postgresql-42.5.4.jar ${LINKIS_HOME}/lib/linkis-commons/public-module/
+```
+### 3.4 配置调整（可选）
> 以下操作，跟依赖的环境有关，根据实际情况，确定是否需要操作
-#### 3.3.1 Yarn的认证
+#### 3.4.1 Yarn的认证
执行spark任务时，需要使用到yarn的ResourceManager，通过配置项`YARN_RESTFUL_URL=http://xx.xx.xx.xx:8088 `控制。
执行安装部署时，会将`YARN_RESTFUL_URL=http://xx.xx.xx.xx:8088` 信息更新到数据库表中 `linkis_cg_rm_external_resource_provider`中时候，默认访问yarn资源是不需权限验证的，
如果yarn的ResourceManager开启了密码权限验证，请安装部署后，修改数据库表 `linkis_cg_rm_external_resource_provider` 中生成的yarn数据信息,
详细可以参考[查看yarn地址是否配置正确](#811-查看yarn地址是否配置正确)。
-#### 3.3.2 session
+#### 3.4.2 session
如果您是对Linkis的升级。同时部署DSS或者其他项目，但其它软件中引入的依赖linkis版本<1.1.1(主要看lib包中，所依赖的Linkis的linkis-module-x.x.x.jar包 <1.1.1），则需要修改位于`${LINKIS_HOME}/conf/linkis.properties`文件。
```shell
echo "wds.linkis.session.ticket.key=bdp-user-ticket-id" >> linkis.properties
```
-### 3.4 启动服务
+### 3.5 启动服务
```shell script
sh sbin/linkis-start-all.sh
```
-### 3.5 安装后配置的修改
+### 3.6 安装后配置的修改
安装完成后，如果需要修改配置（因端口冲突或则某些配置有问题需要调整配置），可以重新执行安装，或则修改对应服务的配置`${LINKIS_HOME}/conf/*properties`文件后，重启对应的服务，如：`sh sbin/linkis-daemon.sh start ps-publicservice`。
-### 3.6 检查服务是否正常启动
+### 3.7 检查服务是否正常启动
访问eureka服务页面（http://eurekaip:20303），
默认会启动6个 Linkis 微服务，其中下图linkis-cg-engineconn服务为运行任务才会启动。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@linkis.apache.org
For additional commands, e-mail: commits-help@linkis.apache.org