You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/11 14:38:01 UTC
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #11435: [Doc][Resources] Instruct users to use local storage if they have remote storage mounted to local
zhongjiajie commented on code in PR #11435:
URL: https://github.com/apache/dolphinscheduler/pull/11435#discussion_r943566481
##########
dolphinscheduler-python/pydolphinscheduler/README.md:
##########
@@ -84,4 +84,4 @@ If you are interested in how to release **PyDolphinScheduler**, you could go and
## What's more
-For more detail information, please go to see **PyDolphinScheduler** [document](https://dolphinscheduler.apache.org/python/index.html)
+For more detail information, please go to see **PyDolphinScheduler** [document](https://dolphinscheduler.apache.org/python/3.0.0/index.html)
Review Comment:
Good catch, I also add in https://github.com/apache/dolphinscheduler/pull/11170/files#diff-b26d7dd07b6c97abef7a204c696762ad9e04fc24713c3817d6ecb22fb84986f3
##########
docs/docs/en/guide/resource/configuration.md:
##########
@@ -1,21 +1,22 @@
-# HDFS Resource Configuration
+# Resource Center Configuration
-When it is necessary to use the Resource Center to create or upload relevant files, all files and resources will be stored on HDFS. Therefore the following configuration is required.
+- You could use `Resource Center` to upload text files, UDFs and other task-related files.
+- You could configure `Resource Center` to use distributed file system like `HDFS` (2.6+), `MinIO` or remote storage products like `AWS S3`, `Alibaba Cloud OSS`, etc.
+- You could configure `Resource Center` to use local file system. If you deploy `DolphinScheduler` in `Standalone` mode, you could configure it to use local file system for `Resouce Center` without the need of an external `HDFS` system or `S3`.
+- Furthermore, if you deploy `DolphinScheduler` in `Cluster` mode, you could use `S3FS-FUSE` to mount `S3` or `Jindo-FUSE` to mount `OSS` to your machines and use the local file system for `Resouce Center`. In this way, you could operate remote files as if on your local machines.
Review Comment:
can we add some links and remove ` for the example file system? like [MinIO](https://min.io)
##########
docs/docs/en/guide/resource/configuration.md:
##########
@@ -1,21 +1,22 @@
-# HDFS Resource Configuration
+# Resource Center Configuration
-When it is necessary to use the Resource Center to create or upload relevant files, all files and resources will be stored on HDFS. Therefore the following configuration is required.
+- You could use `Resource Center` to upload text files, UDFs and other task-related files.
+- You could configure `Resource Center` to use distributed file system like `HDFS` (2.6+), `MinIO` or remote storage products like `AWS S3`, `Alibaba Cloud OSS`, etc.
+- You could configure `Resource Center` to use local file system. If you deploy `DolphinScheduler` in `Standalone` mode, you could configure it to use local file system for `Resouce Center` without the need of an external `HDFS` system or `S3`.
+- Furthermore, if you deploy `DolphinScheduler` in `Cluster` mode, you could use `S3FS-FUSE` to mount `S3` or `Jindo-FUSE` to mount `OSS` to your machines and use the local file system for `Resouce Center`. In this way, you could operate remote files as if on your local machines.
-## Local File Resource Configuration
+## Use Local File System
-For a single machine, you can choose to use local file directory as the upload directory (no need to deploy Hadoop) by making the following configuration.
+### Configure `common.properties`
-### Configuring the `common.properties`
+Configure `api-server/conf/common.properties` and `worker-server/conf/common.properties` as follows:
-Configure the file in the following paths: `api-server/conf/common.properties` and `worker-server/conf/common.properties`.
+- Change `resource.storage.upload.base.path` to your local directory path. Please make sure the `tenant resource.hdfs.root.user` has read and write permissions for `resource.storage.upload.base.path`, e,g. `/tmp/storage/dolphinscheduler`. `DolphinScheduler` will create the directory you configure if it does not exist.
+- Modify `resource.storage.type=HDFS` and `resource.hdfs.fs.defaultFS=file:///`.
Review Comment:
Should we add a notice to tell users to modify `resource.storage.upload.base.path` if they want to change the base resource path?
```markdown
> NOTE: balabalabala
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org