You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by ke...@apache.org on 2022/04/18 08:09:03 UTC
[dolphinscheduler-website] branch master updated: Support fetch released docs from main repo (#775)
This is an automated email from the ASF dual-hosted git repository.
kerwin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new 5b5725fb9 Support fetch released docs from main repo (#775)
5b5725fb9 is described below
commit 5b5725fb9e4cee6056c1bcfbfc2d1ff7fee249ce
Author: Jiajie Zhong <zh...@hotmail.com>
AuthorDate: Mon Apr 18 16:08:59 2022 +0800
Support fetch released docs from main repo (#775)
* Support fetch released docs from main repo
All our history docs before we migrate doc to apache/dolphinscheduler
is in apache/dolphinscheduler-website branch `history-docs`.
After we migrate doc to apache/dolphinscheduler we have two ways to
handle its release docs, one of the ways is copy-paste docs from
apache/dolphinscheduler to apache/dolphinscheduler-website branch
`history-docs`, the other way is to find some black magic fetch docs
based on the release tag.
Of cause, we prefer to choose the second way. Fetch base on released
tags.
* rename bash script from release.conf.sh to conf.sh
* Change fetch branch instead of tag
* fix
---
.github/workflows/change-docs.yaml | 4 +-
HOW_PREPARE_WOKR.md | 42 +++++++++--
scripts/conf.sh | 45 ++++++++++++
scripts/prepare_docs.sh | 83 +++-------------------
scripts/rsync_content.sh | 140 +++++++++++++++++++++++++++++++++++++
5 files changed, 233 insertions(+), 81 deletions(-)
diff --git a/.github/workflows/change-docs.yaml b/.github/workflows/change-docs.yaml
index e4b4f9a74..78d6e01b9 100644
--- a/.github/workflows/change-docs.yaml
+++ b/.github/workflows/change-docs.yaml
@@ -22,7 +22,7 @@ on:
- master
env:
- DOCS_FLAG: ^docs
+ DOCS_FLAG: ^docs|^development
concurrency:
group: docs-${{ github.event.pull_request.number || github.ref }}
@@ -41,7 +41,7 @@ jobs:
- name: Get Branch origin/dev
run: |
git fetch --no-tags --prune --depth=1 origin master
- - name: Fail When Docs Dir Change
+ - name: Fail when change directory `docs` or `development`
run: |
if git diff --name-only origin/master HEAD | grep -q -E "${{ env.DOCS_FLAG }}"; then
exit 1
diff --git a/HOW_PREPARE_WOKR.md b/HOW_PREPARE_WOKR.md
index 6bb9626b1..7ad83ef9e 100644
--- a/HOW_PREPARE_WOKR.md
+++ b/HOW_PREPARE_WOKR.md
@@ -7,6 +7,8 @@ and the history documents before version 3.0.0-alpha is in branch [history-docs]
in [website repository](https://github.com/apache/dolphinscheduler-website). In this case, you have to collect them from
above repository into current working path before you compile them to HTML.
+## Execute Command and Prepare Resource
+
Of course, you could collect all content manually, but we already provider convenience script to do that, all you have to
do is run a single command in this project root directory:
@@ -14,9 +16,39 @@ do is run a single command in this project root directory:
./scripts/prepare_docs.sh
```
-It would do all prepare things for you.
+> Note: The default protocol to fetch repository from GitHub is via HTTPS. When you failed to run the command above and
+> see some log like "unable to access" in your terminal, or if you feel clone via HTTPS is a little slow. You can switch
+> clone protocol to SSH by change the environment variable by execute command `export PROTOCOL_MODE=ssh` in your terminal.
+> The next time you execute command `./scripts/prepare_docs.sh` will user protocol SSH instead of HTTPS, which is more stable
+> and fast in some cases, such as local development
+
+After the command finished, all prepare things you need to start DolphinScheduler website is being done.
+
+## How To Add or Change New Release Document From apache/dolphinscheduler
+
+After version 3.0.0-alpha, our release document is in repository [apache/dolphinscheduler](https://github.com/apache/dolphinscheduler)
+directory `docs`. Documentation and code will be released together, so when the new version is released, we need to get
+the corresponding documentation from the released tag.
+
+All you have to do it is add/change variable `DEV_RELEASE_DOCS_VERSIONS` in [conf.sh](scripts/conf.sh).
+For example, if you want to add new release docs named `10.0.0` and its docs in apache/dolphinscheduler branch `10.0.0-release`,
+you could add a new key-value to `DEV_RELEASE_DOCS_VERSIONS`
+
+```shell
+declare -A DEV_RELEASE_DOCS_VERSIONS=(
+ ...
+ ["10.0.0"]="10.0.0-release"
+ ...
+)
+```
+
+A few days later, you find out version `10.0.0`'s document has a bug. You fix it and push it to apache/dolphinscheduler
+branch `10.0.0-release` to fix the bug, you could keep your configuration in the old content.
-> Note: When you failed to run the command and see some log like "unable to access" in your terminal, you can set and
-> environment variable `export PROTOCOL_MODE=ssh` and then run command `./scripts/prepare_docs.sh` again. After setting the
-> variable will clone source code in SSH protocol instead of HTTPS protocol, and it will stable and fast in some cases like
-> local development.
+```shell
+declare -A DEV_RELEASE_DOCS_VERSIONS=(
+ ...
+ ["10.0.0"]="10.0.0-release"
+ ...
+)
+```
\ No newline at end of file
diff --git a/scripts/conf.sh b/scripts/conf.sh
new file mode 100644
index 000000000..4e3be27c7
--- /dev/null
+++ b/scripts/conf.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Codebase Repository
+PROJECT_NAME="dolphinscheduler"
+PROJECT_BRANCH_NAME="dev"
+PROJECT_WEBSITE_NAME="${PROJECT_NAME}-website"
+PROJECT_WEBSITE_BRANCH_NAME="history-docs"
+
+# Repository Website(current) Directory And Files
+SWAP_DIR="${SOURCE_PATH}/swap"
+PROJECT_SITE_DOC_DIR="${SOURCE_PATH}/docs"
+PROJECT_SITE_DEVELOP_DIR="${SOURCE_PATH}/development"
+PROJECT_DIR="${SWAP_DIR}/${PROJECT_NAME}"
+PROJECT_WEBSITE_DIR="${SWAP_DIR}/${PROJECT_WEBSITE_NAME}"
+
+# docs in apache/dolphinscheduler-webstie branch `history-docs`
+declare -a HISTORY_DOCS_VERSIONS=(
+"1.2.0" "1.2.1" "1.3.1" "1.3.2" "1.3.3" "1.3.4" "1.3.5" "1.3.6" "1.3.8" "1.3.9"
+"2.0.0" "2.0.1" "2.0.2" "2.0.3" "2.0.5"
+)
+
+# docs in apache/dolphinscheduler directory `docs` after 3.0.0-alpha(the time we migrate docs to this repo)
+declare -A DEV_RELEASE_DOCS_VERSIONS=(
+ # The key value is represents of version `key` document is from tag `val` from apache/dolphinscheduler
+ # example: ["key"]="val"
+ ["3.0.0"]="3.0.0-alpha-release"
+)
+
diff --git a/scripts/prepare_docs.sh b/scripts/prepare_docs.sh
index c4655f0bb..fac80380b 100755
--- a/scripts/prepare_docs.sh
+++ b/scripts/prepare_docs.sh
@@ -21,24 +21,7 @@ set -euo pipefail
SOURCE_PATH="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")" )" && pwd)"
-# Codebase Repository
-PROJECT_NAME="dolphinscheduler"
-PROJECT_BRANCH_NAME="dev"
-PROJECT_WEBSITE_NAME="${PROJECT_NAME}-website"
-PROJECT_WEBSITE_BRANCH_NAME="history-docs"
-
-# Repository Website(current) Directory And Files
-SWAP_DIR="${SOURCE_PATH}/swap"
-PROJECT_SITE_DOC_DIR="${SOURCE_PATH}/docs"
-PROJECT_SITE_DEVELOP_DIR="${SOURCE_PATH}/development"
-PROJECT_DIR="${SWAP_DIR}/${PROJECT_NAME}"
-PROJECT_WEBSITE_DIR="${SWAP_DIR}/${PROJECT_WEBSITE_NAME}"
-
-declare -a versions=(
-"1.2.0" "1.2.1" "1.3.1" "1.3.2" "1.3.3" "1.3.4" "1.3.5" "1.3.6" "1.3.8" "1.3.9"
-"2.0.0" "2.0.1" "2.0.2" "2.0.3" "2.0.5"
-)
-
+source "${SOURCE_PATH}/scripts/conf.sh"
# Choose the protocol for git communication to server, default is HTTP because it do not requests password or secret key,
# run command `export PROTOCOL_MODE=ssh` in terminal change protocol to SSH which in is faster and stable in many cases,
@@ -72,37 +55,6 @@ function rebuild_dirs() {
done
}
-##############################################################
-#
-# Sync command wrapper about copy from content from src to
-# dest. It will create a directory for dest to make sure the
-# rsync, and support rsync options.
-#
-# Arguments:
-#
-# src: rsync source parameter
-# dest: rsync destent parameter
-# rsync options: rsync command options
-#
-##############################################################
-function rsync_wrapper() {
- local src="${1}"
- local dest="${2}"
- local dir_flag=""
- # Create directory
- if [ -d "${src}" ]; then
- mkdir -p "${dest}"
- dir_flag="/"
- else
- mkdir -p "$(dirname "${dest}")"
- fi
- if [ "$#" -eq 2 ]; then
- rsync -av ${src}${dir_flag} "${dest}"
- else
- rsync -av ${3} ${src}${dir_flag} "${dest}"
- fi
-}
-
##############################################################
#
# Clone repository to target directory, it will only support
@@ -185,33 +137,16 @@ function prepare_docs() {
echo " ---> Clone dev documents from ${PROJECT_REPO} branch ${PROJECT_BRANCH_NAME}."
clone_repo "${PROJECT_REPO}" "${PROJECT_BRANCH_NAME}" "${PROJECT_DIR}"
+ source "${SOURCE_PATH}/scripts/rsync_content.sh"
echo "===>>> Sync content from cloned repository."
echo " ---> Sync history content."
- for version in "${versions[@]}"; do
- echo " --> Sync version ${version} content"
- rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/${version}/user_doc"
- rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/${version}/user_doc"
- echo " ---> Sync version ${version} configs."
- local config_file_stem="docs${version//./-}"
- rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/configs/${config_file_stem}.js" "${SOURCE_PATH}/site_config/${config_file_stem}.js"
- done
- echo " ---> Sync history img."
- rsync_wrapper "${PROJECT_WEBSITE_DIR}/img"/ "${SOURCE_PATH}/img"
-
- echo " ---> Sync dev user document content."
- rsync_wrapper "${PROJECT_DIR}/docs/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
- rsync_wrapper "${PROJECT_DIR}/docs/docs/en/*.md" "${PROJECT_SITE_DOC_DIR}/en-us/release"
- rsync_wrapper "${PROJECT_DIR}/docs/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
- rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/*.md" "${PROJECT_SITE_DOC_DIR}/zh-cn/release"
- echo " ---> Sync development document content."
- rsync_wrapper "${PROJECT_DIR}/docs/docs/en/development" "${PROJECT_SITE_DEVELOP_DIR}/en-us"
- rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/development" "${PROJECT_SITE_DEVELOP_DIR}/zh-cn"
- echo " ---> Sync dev img."
- rsync_wrapper "${PROJECT_DIR}/docs/img" "${SOURCE_PATH}/img"
- echo " ---> Sync dev configs."
- rsync_wrapper "${PROJECT_DIR}/docs/configs/site.js" "${SOURCE_PATH}/site_config/site.js"
- rsync_wrapper "${PROJECT_DIR}/docs/configs/index.md.jsx" "${SOURCE_PATH}/src/pages/docs/index.md.jsx"
- rsync_wrapper "${PROJECT_DIR}/docs/configs/docsdev.js" "${SOURCE_PATH}/site_config/docsdev.js"
+ rsync_history_docs
+
+ echo " ---> Sync released document after migrating docs into apache/dolphinscheduler."
+ rsync_released_docs
+
+ echo " ---> Sync latest document in dev branch."
+ rsync_latest_docs
echo "===>>> End prepare document."
}
diff --git a/scripts/rsync_content.sh b/scripts/rsync_content.sh
new file mode 100644
index 000000000..346849f05
--- /dev/null
+++ b/scripts/rsync_content.sh
@@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -euo pipefail
+
+SOURCE_PATH="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")" )" && pwd)"
+
+source "${SOURCE_PATH}/scripts/conf.sh"
+
+##############################################################
+#
+# Sync command wrapper about copy from content from src to
+# dest. It will create a directory for dest to make sure the
+# rsync, and support rsync options.
+#
+# Arguments:
+#
+# src: rsync source parameter
+# dest: rsync destent parameter
+# rsync options: rsync command options
+#
+##############################################################
+function rsync_wrapper() {
+ local src="${1}"
+ local dest="${2}"
+ local dir_flag=""
+ # Create directory
+ if [ -d "${src}" ]; then
+ mkdir -p "${dest}"
+ dir_flag="/"
+ else
+ mkdir -p "$(dirname "${dest}")"
+ fi
+ if [ "$#" -eq 2 ]; then
+ rsync -av ${src}${dir_flag} "${dest}"
+ else
+ rsync -av ${3} ${src}${dir_flag} "${dest}"
+ fi
+}
+
+##############################################################
+#
+# Rsync history document, from cloned repo
+# apache/dolphinscheduler-website branch `history-docs`, to
+# directory `docs` and `img`.
+#
+##############################################################
+function rsync_history_docs() {
+ for version in "${HISTORY_DOCS_VERSIONS[@]}"; do
+ echo " --> Sync version ${version} content"
+ rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/${version}/user_doc"
+ rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/${version}/user_doc"
+
+ echo " ---> Sync version ${version} configs."
+ local config_file_stem="docs${version//./-}"
+ rsync_wrapper "${PROJECT_WEBSITE_DIR}/docs/${version}/configs/${config_file_stem}.js" "${SOURCE_PATH}/site_config/${config_file_stem}.js"
+ done
+
+ echo " ---> Sync history img in batch mode."
+ rsync_wrapper "${PROJECT_WEBSITE_DIR}/img"/ "${SOURCE_PATH}/img"
+}
+
+##############################################################
+#
+# Rsync released document, from cloned repo
+# apache/dolphinscheduler tags after we migrate docs to the
+# main repo(after 3.0.0-alpha), to directory `docs`, `img` and
+# `site_config`.
+#
+##############################################################
+function rsync_released_docs() {
+ echo " ---> Directory change to ${PROJECT_DIR}."
+ cd "${PROJECT_DIR}"
+
+ for version in "${!DEV_RELEASE_DOCS_VERSIONS[@]}"; do
+ echo " ---> Git checkout to version ${version}."
+ git fetch origin "${DEV_RELEASE_DOCS_VERSIONS[$version]}" --no-tags
+ git checkout -b "${version}" FETCH_HEAD
+
+ echo " ---> Sync released version ${version} docs."
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/${version}/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/${version}/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
+
+ echo " ---> Sync released version ${version} img."
+ rsync_wrapper "${PROJECT_DIR}/docs/img" "${SOURCE_PATH}/img"
+ echo " ---> Sync released version ${version} configs."
+
+ local config_file_stem="docs${version//./-}"
+ # To avoid change file `docsdev.js` name each time before release, we change it during our sync process
+ rsync_wrapper "${PROJECT_DIR}/docs/configs/docsdev.js" "${SOURCE_PATH}/site_config/${config_file_stem}.js"
+ done
+}
+
+##############################################################
+#
+# Rsync released document, from cloned repo
+# apache/dolphinscheduler branch `dev`(latest), to directory
+# `docs`, `img`, also with rsync config files.
+#
+##############################################################
+function rsync_latest_docs() {
+ echo " ---> Directory change to ${PROJECT_DIR}."
+ cd "${PROJECT_DIR}"
+ echo " ---> Git checkout to version ${PROJECT_BRANCH_NAME}."
+ git checkout "${PROJECT_BRANCH_NAME}"
+
+ echo " ---> Sync dev docs."
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/en" "${PROJECT_SITE_DOC_DIR}/en-us/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/en/*.md" "${PROJECT_SITE_DOC_DIR}/en-us/release"
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/zh" "${PROJECT_SITE_DOC_DIR}/zh-cn/dev/user_doc" "--exclude=faq.md --exclude=history-versions.md --exclude=development"
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/*.md" "${PROJECT_SITE_DOC_DIR}/zh-cn/release"
+
+ echo " ---> Sync development document content."
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/en/development" "${PROJECT_SITE_DEVELOP_DIR}/en-us"
+ rsync_wrapper "${PROJECT_DIR}/docs/docs/zh/development" "${PROJECT_SITE_DEVELOP_DIR}/zh-cn"
+
+ echo " ---> Sync dev img."
+ rsync_wrapper "${PROJECT_DIR}/docs/img" "${SOURCE_PATH}/img"
+
+ echo " ---> Sync dev configs."
+ rsync_wrapper "${PROJECT_DIR}/docs/configs/site.js" "${SOURCE_PATH}/site_config/site.js"
+ rsync_wrapper "${PROJECT_DIR}/docs/configs/index.md.jsx" "${SOURCE_PATH}/src/pages/docs/index.md.jsx"
+ rsync_wrapper "${PROJECT_DIR}/docs/configs/docsdev.js" "${SOURCE_PATH}/site_config/docsdev.js"
+}