You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/04/23 13:37:18 UTC
[incubator-doris] branch master updated: [feature](image tool) support image load tool (#8982)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4a10b37ca2 [feature](image tool) support image load tool (#8982)
4a10b37ca2 is described below
commit 4a10b37ca28fd3b095c4125121567fce9c536c43
Author: Henry2SS <45...@users.noreply.github.com>
AuthorDate: Sat Apr 23 21:36:58 2022 +0800
[feature](image tool) support image load tool (#8982)
---
bin/start_fe.sh | 16 +++++++-
.../operation/metadata-operation.md | 13 ++++++-
.../operation/metadata-operation.md | 13 +++++--
.../src/main/java/org/apache/doris/PaloFe.java | 44 ++++++++++++++++++----
.../apache/doris/common/CommandLineOptions.java | 19 +++++++++-
.../doris/common/CommandLineOptionsTest.java | 11 +++++-
6 files changed, 100 insertions(+), 16 deletions(-)
diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index b9f1df0b8e..009448d2bc 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -27,12 +27,15 @@ OPTS=$(getopt \
-o '' \
-l 'daemon' \
-l 'helper:' \
+ -l 'image:' \
-- "$@")
eval set -- "$OPTS"
RUN_DAEMON=0
HELPER=
+IMAGE_PATH=
+IMAGE_TOOL=
while true; do
case "$1" in
--daemon)
@@ -43,6 +46,11 @@ while true; do
HELPER=$2
shift 2
;;
+ --image)
+ IMAGE_TOOL=1
+ IMAGE_PATH=$2
+ shift 2
+ ;;
--)
shift
break
@@ -163,7 +171,13 @@ if [ x"$HELPER" != x"" ]; then
HELPER="-helper $HELPER"
fi
-if [ ${RUN_DAEMON} -eq 1 ]; then
+if [ ${IMAGE_TOOL} -eq 1 ]; then
+ if [ ! -z ${IMAGE_PATH} ]; then
+ $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe -i ${IMAGE_PATH}
+ else
+ echo "Internal Error. USE IMAGE_TOOL like : ./start_fe.sh --image image_path"
+ fi
+elif [ ${RUN_DAEMON} -eq 1 ]; then
nohup $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" >> $LOG_DIR/fe.out 2>&1 < /dev/null &
else
$LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" < /dev/null
diff --git a/docs/en/administrator-guide/operation/metadata-operation.md b/docs/en/administrator-guide/operation/metadata-operation.md
index abc5aaca35..df8023cef9 100644
--- a/docs/en/administrator-guide/operation/metadata-operation.md
+++ b/docs/en/administrator-guide/operation/metadata-operation.md
@@ -267,9 +267,18 @@ In some extreme cases, the image file on the disk may be damaged, but the metada
```
curl -u $root_user:$password http://$master_hostname:8030/dump
```
-3. Replace the image file in the `meta_dir/image` directory on the OBSERVER FE node with the image_mem file, restart the OBSERVER FE node, and verify the integrity and correctness of the image_mem file. You can check whether the DB and Table metadata are normal on the FE Web page, whether there is an exception in `fe.log`, whether it is in a normal replayed jour.
+3. Execute the following command to verify the integrity and correctness of the generated image_mem file:
+```
+sh start_fe.sh --image path_to_image_mem
+```
+
+> note: `path_to_image_mem` is the path to the image_mem file.
+>
+> If the file is valid, the output will be `Load image success. Image file /absolute/path/to/image.xxxxxx valid`.
+>
+> If the file is invalid, the output will be `Load image failed. Image file /absolute/path/to/image.xxxxxx is invalid`.
-4. Replace the image file in the `meta_dir/image` directory on the FOLLOWER FE node with the image_mem file in turn, restart the FOLLOWER FE node, and confirm that the metadata and query services are normal.
+4. Replace the image file in the `meta_dir/image` directory on the OBSERVER/FOLLOWER FE node with the image_mem file in turn, restart the FOLLOWER FE node, and confirm that the metadata and query services are normal.
5. Replace the image file in the `meta_dir/image` directory on the Master FE node with the image_mem file, restart the Master FE node, and then confirm that the FE Master switch is normal and The Master FE node can generate a new image file through checkpoint.
diff --git a/docs/zh-CN/administrator-guide/operation/metadata-operation.md b/docs/zh-CN/administrator-guide/operation/metadata-operation.md
index 39c7891a86..b2534325d7 100644
--- a/docs/zh-CN/administrator-guide/operation/metadata-operation.md
+++ b/docs/zh-CN/administrator-guide/operation/metadata-operation.md
@@ -267,9 +267,16 @@ FE 目前有以下几个端口
```
curl -u $root_user:$password http://$master_hostname:8030/dump
```
-3. 用 image_mem 文件替换掉 OBSERVER FE 节点上`meta_dir/image`目录下的 image 文件，重启 OBSERVER FE 节点，
-验证 image_mem 文件的完整性和正确性（可以在 FE Web 页面查看 DB 和 Table 的元数据是否正常，查看fe.log 是否有异常，是否在正常 replayed journal）
-4. 依次用 image_mem 文件替换掉 FOLLOWER FE 节点上`meta_dir/image`目录下的 image 文件，重启 FOLLOWER FE 节点，
+3. 执行以下命令，验证生成的 image_mem 文件的完整性和正确性：
+```
+sh start_fe.sh --image path_to_image_mem
+```
+> 注意：`path_to_image_mem` 是 image_mem 文件的路径。
+>
+> 如果文件有效会输出 `Load image success. Image file /absolute/path/to/image.xxxxxx is valid`。
+>
+> 如果文件无效会输出 `Load image failed. Image file /absolute/path/to/image.xxxxxx is invalid`。
+4. 依次用 image_mem 文件替换掉 OBSERVER/FOLLOWER FE 节点上`meta_dir/image`目录下的 image 文件，重启 FOLLOWER FE 节点，
确认元数据和查询服务都正常
5. 用 image_mem 文件替换掉 Master FE 节点上`meta_dir/image`目录下的 image 文件，重启 Master FE 节点，
确认 FE Master 切换正常， Master FE 节点可以通过 checkpoint 正常生成新的 image 文件
diff --git a/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java b/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
index c23d3ec3db..0a1fb68c49 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
@@ -22,6 +22,7 @@ import org.apache.doris.common.CommandLineOptions;
import org.apache.doris.common.Config;
import org.apache.doris.common.LdapConfig;
import org.apache.doris.common.Log4jConfig;
+import org.apache.doris.common.MetaReader;
import org.apache.doris.common.ThreadPoolManager;
import org.apache.doris.common.Version;
import org.apache.doris.common.util.JdkUtils;
@@ -180,6 +181,8 @@ public class PaloFe {
* Print the version of Palo Frontend
* -h --helper
* Specify the helper node when joining a bdb je replication group
+ * -i --image
+ * Check if the specified image is valid
* -b --bdb
* Run bdbje debug tools
*
@@ -203,6 +206,7 @@ public class PaloFe {
Options options = new Options();
options.addOption("v", "version", false, "Print the version of Palo Frontend");
options.addOption("h", "helper", true, "Specify the helper node when joining a bdb je replication group");
+ options.addOption("i", "image", true, "Check if the specified image is valid");
options.addOption("b", "bdb", false, "Run bdbje debug tools");
options.addOption("l", "listdb", false, "List databases in bdbje");
options.addOption("d", "db", true, "Specify a database in bdbje");
@@ -222,7 +226,7 @@ public class PaloFe {
// version
if (cmd.hasOption('v') || cmd.hasOption("version")) {
- return new CommandLineOptions(true, "", null);
+ return new CommandLineOptions(true, "", null, "");
}
// helper
if (cmd.hasOption('h') || cmd.hasOption("helper")) {
@@ -231,14 +235,23 @@ public class PaloFe {
System.err.println("Missing helper node");
System.exit(-1);
}
- return new CommandLineOptions(false, helperNode, null);
+ return new CommandLineOptions(false, helperNode, null, "");
+ }
+ // image
+ if (cmd.hasOption('i') || cmd.hasOption("image")) {
+ // get image path
+ String imagePath = cmd.getOptionValue("image");
+ if (Strings.isNullOrEmpty(imagePath)) {
+ System.err.println("imagePath is not set");
+ System.exit(-1);
+ }
+ return new CommandLineOptions(false, "", null, imagePath);
}
-
if (cmd.hasOption('b') || cmd.hasOption("bdb")) {
if (cmd.hasOption('l') || cmd.hasOption("listdb")) {
// list bdb je databases
BDBToolOptions bdbOpts = new BDBToolOptions(true, "", false, "", "", 0);
- return new CommandLineOptions(false, "", bdbOpts);
+ return new CommandLineOptions(false, "", bdbOpts, "");
}
if (cmd.hasOption('d') || cmd.hasOption("db")) {
// specify a database
@@ -249,7 +262,7 @@ public class PaloFe {
}
if (cmd.hasOption('s') || cmd.hasOption("stat")) {
BDBToolOptions bdbOpts = new BDBToolOptions(false, dbName, true, "", "", 0);
- return new CommandLineOptions(false, "", bdbOpts);
+ return new CommandLineOptions(false, "", bdbOpts, "");
}
String fromKey = "";
String endKey = "";
@@ -278,7 +291,7 @@ public class PaloFe {
}
BDBToolOptions bdbOpts = new BDBToolOptions(false, dbName, false, fromKey, endKey, metaVersion);
- return new CommandLineOptions(false, "", bdbOpts);
+ return new CommandLineOptions(false, "", bdbOpts, "");
} else {
System.err.println("Invalid options when running bdb je tools");
@@ -287,7 +300,7 @@ public class PaloFe {
}
// helper node is null, means no helper node is specified
- return new CommandLineOptions(false, null, null);
+ return new CommandLineOptions(false, null, null, "");
}
private static void checkCommandLineOptions(CommandLineOptions cmdLineOpts) {
@@ -305,6 +318,23 @@ public class PaloFe {
} else {
System.exit(-1);
}
+ } else if (cmdLineOpts.runImageTool()) {
+ File imageFile = new File(cmdLineOpts.getImagePath());
+ if (!imageFile.exists()) {
+ System.out.println("image does not exist: " + imageFile.getAbsolutePath() + " . Please put an absolute path instead");
+ System.exit(-1);
+ } else {
+ System.out.println("Start to load image: ");
+ try {
+ MetaReader.read(imageFile, Catalog.getCurrentCatalog());
+ System.out.println("Load image success. Image file " + cmdLineOpts.getImagePath() + " is valid");
+ } catch (Exception e) {
+ System.out.println("Load image failed. Image file " + cmdLineOpts.getImagePath() + " is invalid");
+ e.printStackTrace();
+ } finally {
+ System.exit(0);
+ }
+ }
}
// go on
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/CommandLineOptions.java b/fe/fe-core/src/main/java/org/apache/doris/common/CommandLineOptions.java
index 79854e3adc..df0115cf4e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/CommandLineOptions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/CommandLineOptions.java
@@ -25,16 +25,24 @@ public class CommandLineOptions {
private String helperNode;
private boolean runBdbTools;
private BDBToolOptions bdbToolOpts = null;
+ private boolean runImageTool;
+ private String imagePath;
- public CommandLineOptions(boolean isVersion, String helperNode, BDBToolOptions bdbToolOptions) {
+ public CommandLineOptions(boolean isVersion, String helperNode, BDBToolOptions bdbToolOptions, String imagePath) {
this.isVersion = isVersion;
this.helperNode = helperNode;
this.bdbToolOpts = bdbToolOptions;
+ this.imagePath = imagePath;
if (this.bdbToolOpts != null) {
runBdbTools = true;
} else {
runBdbTools = false;
}
+ if (!imagePath.isEmpty()) {
+ runImageTool = true;
+ } else {
+ runImageTool = false;
+ }
}
public boolean isVersion() {
@@ -53,12 +61,21 @@ public class CommandLineOptions {
return bdbToolOpts;
}
+ public boolean runImageTool() {
+ return runImageTool;
+ }
+
+ public String getImagePath() {
+ return imagePath;
+ }
+
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append("print version: " + isVersion).append("\n");
sb.append("helper node: " + helperNode).append("\n");
sb.append("bdb tool options: \n(\n" + bdbToolOpts).append("\n)\n");
+ sb.append("image tool options: \n(\n" + imagePath).append("\n)\n");
return sb.toString();
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/CommandLineOptionsTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/CommandLineOptionsTest.java
index b8d5045988..8575da58ca 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/CommandLineOptionsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/CommandLineOptionsTest.java
@@ -26,13 +26,20 @@ public class CommandLineOptionsTest {
@Test
public void test() {
- CommandLineOptions options = new CommandLineOptions(true, "", null);
+ CommandLineOptions options = new CommandLineOptions(true, "", null, "");
Assert.assertTrue(options.isVersion());
Assert.assertFalse(options.runBdbTools());
+ Assert.assertFalse(options.runImageTool());
- options = new CommandLineOptions(false, "", new BDBToolOptions(true, "", false, "", "", 0));
+ options = new CommandLineOptions(false, "", new BDBToolOptions(true, "", false, "", "", 0), "");
Assert.assertFalse(options.isVersion());
Assert.assertTrue(options.runBdbTools());
+ Assert.assertFalse(options.runImageTool());
+
+ options = new CommandLineOptions(false, "", null, "image.0");
+ Assert.assertFalse(options.isVersion());
+ Assert.assertFalse(options.runBdbTools());
+ Assert.assertTrue(options.runImageTool());
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org