You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/12/08 04:09:15 UTC
[GitHub] [iotdb] Cpaulyz opened a new pull request, #8378: [To rel/0.13][IOTDB-5133] Export schema tool
Cpaulyz opened a new pull request, #8378:
URL: https://github.com/apache/iotdb/pull/8378
## Description
Add an export schema function to export MTree and Tag from the current system into a non-deleted mlog.bin for loading mlog in a non-empty iotdb correctly.
The schema export operation exports the information about store group, timeseries, and schema template in the current IoTDB in the form of `mlog.bin` and `tlog.txt` to the specified directory.
The exported `mlog.bin` and `tlog.txt` files can be incrementally loaded into an IoTDB instance.
### Export Schema SQL
```
EXPORT SCHEMA '<path/dir>'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a diff in pull request #8378: [To rel/0.13][IOTDB-5133] Export schema tool
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on code in PR #8378:
URL: https://github.com/apache/iotdb/pull/8378#discussion_r1043103360
##########
server/src/main/java/org/apache/iotdb/db/metadata/mtree/MTree.java:
##########
@@ -230,6 +234,139 @@ public void serializeTo(String snapshotPath) throws IOException {
}
}
+ public void exportSchema(MLogWriter mLogWriter) throws MetadataException, IOException {
+ PartialPath[] entityPath = new PartialPath[1];
+ List<IMeasurementSchema> alignedMeasurementSchemas = new ArrayList<>();
+ Traverser collector =
+ new Traverser(root, new PartialPath("root.**")) {
+ IEntityMNode entityNode;
+
+ @Override
+ protected boolean processInternalMatchedMNode(IMNode node, int idx, int level)
+ throws MetadataException {
+ if (node.getSchemaTemplate() != null) {
+ try {
+ mLogWriter.setSchemaTemplate(
+ new SetTemplatePlan(
+ node.getSchemaTemplate().getName(),
+ getCurrentPartialPath(node).getFullPath()));
+ if (node.isUseTemplate()) {
+ mLogWriter.setUsingSchemaTemplate(getCurrentPartialPath(node));
+ }
+ } catch (IOException e) {
+ logger.error(e.getMessage());
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #8378: [To rel/0.13][IOTDB-5133] Export schema tool
Posted by GitBox <gi...@apache.org>.
qiaojialin merged PR #8378:
URL: https://github.com/apache/iotdb/pull/8378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] MarcosZyk commented on a diff in pull request #8378: [To rel/0.13][IOTDB-5133] Export schema tool
Posted by GitBox <gi...@apache.org>.
MarcosZyk commented on code in PR #8378:
URL: https://github.com/apache/iotdb/pull/8378#discussion_r1043041221
##########
server/src/main/java/org/apache/iotdb/db/metadata/mtree/MTree.java:
##########
@@ -230,6 +234,139 @@ public void serializeTo(String snapshotPath) throws IOException {
}
}
+ public void exportSchema(MLogWriter mLogWriter) throws MetadataException, IOException {
+ PartialPath[] entityPath = new PartialPath[1];
+ List<IMeasurementSchema> alignedMeasurementSchemas = new ArrayList<>();
+ Traverser collector =
+ new Traverser(root, new PartialPath("root.**")) {
+ IEntityMNode entityNode;
+
+ @Override
+ protected boolean processInternalMatchedMNode(IMNode node, int idx, int level)
+ throws MetadataException {
+ if (node.getSchemaTemplate() != null) {
+ try {
+ mLogWriter.setSchemaTemplate(
+ new SetTemplatePlan(
+ node.getSchemaTemplate().getName(),
+ getCurrentPartialPath(node).getFullPath()));
+ if (node.isUseTemplate()) {
+ mLogWriter.setUsingSchemaTemplate(getCurrentPartialPath(node));
+ }
+ } catch (IOException e) {
+ logger.error(e.getMessage());
+ }
Review Comment:
Extract the common code.
##########
server/src/main/java/org/apache/iotdb/db/metadata/MManager.java:
##########
@@ -1336,6 +1339,81 @@ public Pair<List<MeasurementPath>, Integer> getMeasurementPathsWithAlias(
return mtree.getMeasurementPathsWithAlias(pathPattern, limit, offset, isPrefixMatch);
}
+ public void exportSchema(File dir) throws IOException, MetadataException {
+ if (!dir.exists()) {
+ dir.mkdirs();
+ } else {
+ if (!dir.isDirectory()) {
+ throw new IOException(String.format("%s is not a directory.", dir.getAbsolutePath()));
+ }
+ }
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(
+ new File(dir.getAbsolutePath(), MetadataConstant.METADATA_LOG).getAbsolutePath()));
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(new File(dir.getAbsolutePath(), MetadataConstant.TAG_LOG).getAbsolutePath()));
+
+ try (MLogWriter mLogWriter =
+ new MLogWriter(dir.getAbsolutePath(), MetadataConstant.METADATA_LOG)) {
+ // export storage group
+ for (PartialPath sg : mtree.getAllStorageGroupPaths()) {
+ mLogWriter.setStorageGroup(sg);
+ }
+ // export template
+ for (Map.Entry<String, Template> entry : templateManager.getTemplateMap().entrySet()) {
+ if (entry.getValue().isDirectAligned()) {
+ List<List<String>> measurements = Collections.singletonList(new ArrayList<>());
+ List<List<TSDataType>> dataTypes = Collections.singletonList(new ArrayList<>());
+ List<List<TSEncoding>> encodings = Collections.singletonList(new ArrayList<>());
+ List<List<CompressionType>> compressions = Collections.singletonList(new ArrayList<>());
+ entry
+ .getValue()
+ .getSchemaMap()
+ .values()
+ .forEach(
+ i -> {
+ measurements.get(0).add(i.getMeasurementId());
+ dataTypes.get(0).add(i.getType());
+ encodings.get(0).add(i.getEncodingType());
+ compressions.get(0).add(i.getCompressor());
+ });
+ mLogWriter.createSchemaTemplate(
+ new CreateTemplatePlan(
+ entry.getKey(), measurements, dataTypes, encodings, compressions));
+ } else {
+ List<List<String>> measurements = new ArrayList<>();
+ List<List<TSDataType>> dataTypes = new ArrayList<>();
+ List<List<TSEncoding>> encodings = new ArrayList<>();
+ List<List<CompressionType>> compressions = new ArrayList<>();
+ entry
+ .getValue()
+ .getSchemaMap()
+ .values()
+ .forEach(
+ i -> {
+ measurements.add(Collections.singletonList(i.getMeasurementId()));
+ dataTypes.add(Collections.singletonList(i.getType()));
+ encodings.add(Collections.singletonList(i.getEncodingType()));
+ compressions.add(Collections.singletonList(i.getCompressor()));
+ });
+ mLogWriter.createSchemaTemplate(
+ new CreateTemplatePlan(
+ entry.getKey(), measurements, dataTypes, encodings, compressions));
+ }
+ }
+ // export timeseries
+ mtree.exportSchema(mLogWriter);
+ // export tag
+ File tagTargetFile = new File(dir, MetadataConstant.TAG_LOG);
+ File tagSourceFile = new File(config.getSchemaDir(), MetadataConstant.TAG_LOG);
+ Path sourcePath = FileSystems.getDefault().getPath(tagSourceFile.getAbsolutePath());
+ Path targetPath = FileSystems.getDefault().getPath(tagTargetFile.getAbsolutePath());
+ Files.createLink(targetPath, sourcePath);
Review Comment:
If cross disk, this operation may fail.
##########
server/src/main/java/org/apache/iotdb/db/metadata/MManager.java:
##########
@@ -1336,6 +1339,81 @@ public Pair<List<MeasurementPath>, Integer> getMeasurementPathsWithAlias(
return mtree.getMeasurementPathsWithAlias(pathPattern, limit, offset, isPrefixMatch);
}
+ public void exportSchema(File dir) throws IOException, MetadataException {
+ if (!dir.exists()) {
+ dir.mkdirs();
+ } else {
+ if (!dir.isDirectory()) {
+ throw new IOException(String.format("%s is not a directory.", dir.getAbsolutePath()));
+ }
+ }
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(
+ new File(dir.getAbsolutePath(), MetadataConstant.METADATA_LOG).getAbsolutePath()));
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(new File(dir.getAbsolutePath(), MetadataConstant.TAG_LOG).getAbsolutePath()));
Review Comment:
It's better to remind the user there's existing files and let the user delete them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a diff in pull request #8378: [To rel/0.13][IOTDB-5133] Export schema tool
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on code in PR #8378:
URL: https://github.com/apache/iotdb/pull/8378#discussion_r1043103634
##########
server/src/main/java/org/apache/iotdb/db/metadata/MManager.java:
##########
@@ -1336,6 +1339,81 @@ public Pair<List<MeasurementPath>, Integer> getMeasurementPathsWithAlias(
return mtree.getMeasurementPathsWithAlias(pathPattern, limit, offset, isPrefixMatch);
}
+ public void exportSchema(File dir) throws IOException, MetadataException {
+ if (!dir.exists()) {
+ dir.mkdirs();
+ } else {
+ if (!dir.isDirectory()) {
+ throw new IOException(String.format("%s is not a directory.", dir.getAbsolutePath()));
+ }
+ }
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(
+ new File(dir.getAbsolutePath(), MetadataConstant.METADATA_LOG).getAbsolutePath()));
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(new File(dir.getAbsolutePath(), MetadataConstant.TAG_LOG).getAbsolutePath()));
+
+ try (MLogWriter mLogWriter =
+ new MLogWriter(dir.getAbsolutePath(), MetadataConstant.METADATA_LOG)) {
+ // export storage group
+ for (PartialPath sg : mtree.getAllStorageGroupPaths()) {
+ mLogWriter.setStorageGroup(sg);
+ }
+ // export template
+ for (Map.Entry<String, Template> entry : templateManager.getTemplateMap().entrySet()) {
+ if (entry.getValue().isDirectAligned()) {
+ List<List<String>> measurements = Collections.singletonList(new ArrayList<>());
+ List<List<TSDataType>> dataTypes = Collections.singletonList(new ArrayList<>());
+ List<List<TSEncoding>> encodings = Collections.singletonList(new ArrayList<>());
+ List<List<CompressionType>> compressions = Collections.singletonList(new ArrayList<>());
+ entry
+ .getValue()
+ .getSchemaMap()
+ .values()
+ .forEach(
+ i -> {
+ measurements.get(0).add(i.getMeasurementId());
+ dataTypes.get(0).add(i.getType());
+ encodings.get(0).add(i.getEncodingType());
+ compressions.get(0).add(i.getCompressor());
+ });
+ mLogWriter.createSchemaTemplate(
+ new CreateTemplatePlan(
+ entry.getKey(), measurements, dataTypes, encodings, compressions));
+ } else {
+ List<List<String>> measurements = new ArrayList<>();
+ List<List<TSDataType>> dataTypes = new ArrayList<>();
+ List<List<TSEncoding>> encodings = new ArrayList<>();
+ List<List<CompressionType>> compressions = new ArrayList<>();
+ entry
+ .getValue()
+ .getSchemaMap()
+ .values()
+ .forEach(
+ i -> {
+ measurements.add(Collections.singletonList(i.getMeasurementId()));
+ dataTypes.add(Collections.singletonList(i.getType()));
+ encodings.add(Collections.singletonList(i.getEncodingType()));
+ compressions.add(Collections.singletonList(i.getCompressor()));
+ });
+ mLogWriter.createSchemaTemplate(
+ new CreateTemplatePlan(
+ entry.getKey(), measurements, dataTypes, encodings, compressions));
+ }
+ }
+ // export timeseries
+ mtree.exportSchema(mLogWriter);
+ // export tag
+ File tagTargetFile = new File(dir, MetadataConstant.TAG_LOG);
+ File tagSourceFile = new File(config.getSchemaDir(), MetadataConstant.TAG_LOG);
+ Path sourcePath = FileSystems.getDefault().getPath(tagSourceFile.getAbsolutePath());
+ Path targetPath = FileSystems.getDefault().getPath(tagTargetFile.getAbsolutePath());
+ Files.createLink(targetPath, sourcePath);
Review Comment:
replaced by `Files.copyFile`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] Cpaulyz commented on a diff in pull request #8378: [To rel/0.13][IOTDB-5133] Export schema tool
Posted by GitBox <gi...@apache.org>.
Cpaulyz commented on code in PR #8378:
URL: https://github.com/apache/iotdb/pull/8378#discussion_r1043104002
##########
server/src/main/java/org/apache/iotdb/db/metadata/MManager.java:
##########
@@ -1336,6 +1339,81 @@ public Pair<List<MeasurementPath>, Integer> getMeasurementPathsWithAlias(
return mtree.getMeasurementPathsWithAlias(pathPattern, limit, offset, isPrefixMatch);
}
+ public void exportSchema(File dir) throws IOException, MetadataException {
+ if (!dir.exists()) {
+ dir.mkdirs();
+ } else {
+ if (!dir.isDirectory()) {
+ throw new IOException(String.format("%s is not a directory.", dir.getAbsolutePath()));
+ }
+ }
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(
+ new File(dir.getAbsolutePath(), MetadataConstant.METADATA_LOG).getAbsolutePath()));
+ Files.deleteIfExists(
+ FileSystems.getDefault()
+ .getPath(new File(dir.getAbsolutePath(), MetadataConstant.TAG_LOG).getAbsolutePath()));
Review Comment:
Done. Add test in `testExportSchemaFail`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org