You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/06/18 09:02:27 UTC
[GitHub] [incubator-iotdb] Ring-k opened a new pull request #1387: Create schema automatically
Ring-k opened a new pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387
This pull request implements schema automatic creation in distributed version.
If an InsertPlan is executed but an exception is encountered because the storage group is not set yet, or the timeseries is not created, the MetaGroupMember will first execute a SetStorageGroupPlan, or CreateTimeseriesPlan, and then execute the InsertPlan again.
These plans will be routed among the cluster, so that the corresponding nodes will be notified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647891127
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444700923
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,29 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ throw new MetadataException("Failed to set storage group " + storageGroupName);
Review comment:
Thanks for your suggestion. The status code will be included in the exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] neuyilan commented on a change in pull request #1387: Create schema automatically
Posted by GitBox <gi...@apache.org>.
neuyilan commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r442608933
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1520,12 +1534,31 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && IoTDBDescriptor.getInstance().getConfig()
Review comment:
How about CreateTimeSeriesPlan? and other plans which can cause first check storage group exist or not ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444697825
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java
##########
@@ -1073,7 +1072,12 @@ public void pullTimeSeriesSchema(PullSchemaRequest request,
/**
* Create an IPointReader of "path" with “timeFilter” and "valueFilter". A synchronization with
+<<<<<<< HEAD
+ * the leader will be performed first to preserve strong consistency. TODO-Cluster: also support
+ * weak consistency
+=======
* the leader will be performed according to consistency level
+>>>>>>> origin/cluster_new
Review comment:
Thanks for your reminding.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647634978
> Good job. But I have a question: if you enable AutoCreation, the receiver of the Insertion plan will create schemas locally and the sender will not sense a schema missing. How do you resolve it?
A new property `ENABLE_AUTO_CREATE_SCHEMA` is added in cluster configurations, which will replace the one in standalone version. In other word, the `ENABLE_AUTO_CREATE_SCHEMA` in iotdb-engine.properties is alwayes treated as `FALSE`. Therefore, the auto schema creation on storage engine level will never be triggered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k edited a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k edited a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647634978
> Good job. But I have a question: if you enable AutoCreation, the receiver of the Insertion plan will create schemas locally and the sender will not sense a schema missing. How do you resolve it?
A new property `ENABLE_AUTO_CREATE_SCHEMA` is added in cluster configurations, which will replace the one in standalone version. In other words, the `ENABLE_AUTO_CREATE_SCHEMA` in iotdb-engine.properties is alwayes treated as `FALSE`. Therefore, the auto schema creation on storage engine level will never be triggered.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-650711425
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444929700
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,39 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = processNonPartitionedMetaPlan(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode() &&
+ setStorageGroupResult.getCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ throw new MetadataException(
+ String.format("Status Code: %d, failed to set storage group ",
+ setStorageGroupResult.getCode(), storageGroupName)
+ );
+ }
+ // try to create timeseries
+ boolean isAutoCreateTimeseriesSuccess = autoCreateTimeseries((InsertPlan)plan);
+ if(!isAutoCreateTimeseriesSuccess){
+ throw new MetadataException(
+ String.format("Failed to create timeseries from InsertPlan automatically.")
+ );
+ }
+ return executeNonQuery(plan);
+ } catch (MetadataException e) {
+ logger.error("Failed to set storage group or create timeseries, because {}", e.getMessage());
Review comment:
Thanks for your suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] jt2594838 merged pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
jt2594838 merged pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444792728
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1684,6 +1743,77 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
return status;
}
+ /**
+ * Create timeseries automatically
+ *
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ String storageGroupName;
+ try {
+ storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ } catch (MetadataException e) {
+ logger.error("Failed to infer storage group from deviceId {}", deviceId);
+ return false;
+ }
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ PartitionGroup partitionGroup = partitionTable.route(storageGroupName, 0);
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ // TODO-Cluster: add executeNonQueryBatch()
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
+ }
+ }
+ return true;
+ }
+
+
+ /**
+ * To check which timeseries in the input list is unregistered
+ *
+ * @param seriesList
+ * @param partitionGroup
+ * @return
+ */
+ List<String> getUnregisteredSeriesList(List<String> seriesList, PartitionGroup partitionGroup) {
+ Set<String> unregistered = new HashSet<>();
+ for (Node node : partitionGroup) {
+ try {
+ DataClient client = getDataClient(node);
+ List<String> result = SyncClientAdaptor
+ .getUnregisteredMeasurements(client, partitionGroup.getHeader(), seriesList);
+ unregistered.addAll(result);
+ } catch (TException | IOException e) {
+ logger.error("{}: cannot getting unregistered {} and other {} paths from {}", name,
+ seriesList.get(0), seriesList.get(seriesList.size() - 1), node, e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ logger.error("{}: getting unregistered series list {} is interrupted from {}", name,
+ Arrays.toString(seriesList.toArray(new String[0])), node, e);
Review comment:
Please refactor this message like the former one to avoid printing too many seriesPaths.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,39 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = processNonPartitionedMetaPlan(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode() &&
+ setStorageGroupResult.getCode() != TSStatusCode.PATH_ALREADY_EXIST_ERROR.getStatusCode()) {
+ throw new MetadataException(
+ String.format("Status Code: %d, failed to set storage group ",
+ setStorageGroupResult.getCode(), storageGroupName)
+ );
+ }
+ // try to create timeseries
+ boolean isAutoCreateTimeseriesSuccess = autoCreateTimeseries((InsertPlan)plan);
+ if(!isAutoCreateTimeseriesSuccess){
+ throw new MetadataException(
+ String.format("Failed to create timeseries from InsertPlan automatically.")
+ );
+ }
+ return executeNonQuery(plan);
+ } catch (MetadataException e) {
+ logger.error("Failed to set storage group or create timeseries, because {}", e.getMessage());
Review comment:
It would be better to use `e` (preserve stack traces).
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1684,6 +1743,77 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
return status;
}
+ /**
+ * Create timeseries automatically
+ *
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ String storageGroupName;
+ try {
+ storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ } catch (MetadataException e) {
+ logger.error("Failed to infer storage group from deviceId {}", deviceId);
+ return false;
+ }
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ PartitionGroup partitionGroup = partitionTable.route(storageGroupName, 0);
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ // TODO-Cluster: add executeNonQueryBatch()
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
+ }
+ }
+ return true;
+ }
+
+
+ /**
+ * To check which timeseries in the input list is unregistered
+ *
+ * @param seriesList
+ * @param partitionGroup
+ * @return
+ */
+ List<String> getUnregisteredSeriesList(List<String> seriesList, PartitionGroup partitionGroup) {
+ Set<String> unregistered = new HashSet<>();
+ for (Node node : partitionGroup) {
+ try {
+ DataClient client = getDataClient(node);
+ List<String> result = SyncClientAdaptor
+ .getUnregisteredMeasurements(client, partitionGroup.getHeader(), seriesList);
+ unregistered.addAll(result);
Review comment:
I think you should break once a non-null result is returned, and I do not think using a Set is necessary.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -914,8 +914,26 @@ public void insert(InsertPlan insertPlan) throws QueryProcessException {
insertPlan.setSchemasAndTransferType(schemas);
StorageEngine.getInstance().insert(insertPlan);
if (insertPlan.getFailedMeasurements() != null) {
- throw new StorageEngineException(
- "failed to insert points " + insertPlan.getFailedMeasurements());
+ // check if all path not exist exceptions
+ List<String> failedPaths = new ArrayList<>(insertPlan.getFailedMeasurements().keySet());
+ List<Exception> exceptions = new ArrayList<>(insertPlan.getFailedMeasurements().values());
+ boolean isPathNotExistException = true;
+ for(Exception e : exceptions){
+ Exception curException = e;
+ while(curException.getCause() != null){
+ curException = (Exception) curException.getCause();
Review comment:
You may just define this as a Throwable so you may avoid the typecast.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -914,8 +914,26 @@ public void insert(InsertPlan insertPlan) throws QueryProcessException {
insertPlan.setSchemasAndTransferType(schemas);
StorageEngine.getInstance().insert(insertPlan);
if (insertPlan.getFailedMeasurements() != null) {
- throw new StorageEngineException(
- "failed to insert points " + insertPlan.getFailedMeasurements());
+ // check if all path not exist exceptions
+ List<String> failedPaths = new ArrayList<>(insertPlan.getFailedMeasurements().keySet());
+ List<Exception> exceptions = new ArrayList<>(insertPlan.getFailedMeasurements().values());
+ boolean isPathNotExistException = true;
+ for(Exception e : exceptions){
+ Exception curException = e;
+ while(curException.getCause() != null){
+ curException = (Exception) curException.getCause();
+ }
+ if(!(curException instanceof PathNotExistException)){
+ isPathNotExistException = false;
+ break;
+ }
+ }
+ if(isPathNotExistException){
+ throw new PathNotExistException(failedPaths.toString());
Review comment:
I suggest you just add a field of List<String> in PathNotExistException, since concatenating a long string list is time-consuming.
Besides, please perform a reformat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444928647
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -914,8 +914,26 @@ public void insert(InsertPlan insertPlan) throws QueryProcessException {
insertPlan.setSchemasAndTransferType(schemas);
StorageEngine.getInstance().insert(insertPlan);
if (insertPlan.getFailedMeasurements() != null) {
- throw new StorageEngineException(
- "failed to insert points " + insertPlan.getFailedMeasurements());
+ // check if all path not exist exceptions
+ List<String> failedPaths = new ArrayList<>(insertPlan.getFailedMeasurements().keySet());
+ List<Exception> exceptions = new ArrayList<>(insertPlan.getFailedMeasurements().values());
+ boolean isPathNotExistException = true;
+ for(Exception e : exceptions){
+ Exception curException = e;
+ while(curException.getCause() != null){
+ curException = (Exception) curException.getCause();
+ }
+ if(!(curException instanceof PathNotExistException)){
+ isPathNotExistException = false;
+ break;
+ }
+ }
+ if(isPathNotExistException){
+ throw new PathNotExistException(failedPaths.toString());
Review comment:
An attribute is added in the PathNotExistExeption. Thanks for your suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r443679064
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1520,12 +1534,31 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && IoTDBDescriptor.getInstance().getConfig()
+ .isAutoCreateSchemaEnabled()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ throw new MetadataException("Failed to set storage group " + storageGroupName);
+ }
+ return executeNonQuery(plan);
+ } catch (MetadataException e) {
+ logger.info("Failed to set storage group of device id {}", deviceId);
+ }
+ }
+ logger.error("{}: Cannot found storage groups for {}", name, plan);
return StatusUtils.NO_STORAGE_GROUP;
}
- logger.debug("{}: The data groups of {} are {}", name, plan, planGroupMap);
+ logger.error("{}: The data groups of {} are {}", name, plan, planGroupMap);
Review comment:
Alright, I will revert.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647631179
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647877532
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647852422
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r443679383
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
+ }
+ }
+ return true;
+ }
+
+
+
+ /**
+ * To check which timeseries in the input list is unregistered
+ * @param seriesList
+ * @param partitionGroup
+ * @return
+ */
+ List<String> getUnregisteredSeriesList(List<String> seriesList, PartitionGroup partitionGroup) {
+ Set<String> unregistered = new HashSet<>();
+ for (Node node : partitionGroup) {
+ try {
+ DataClient client = getDataClient(node);
+ Map<String, Boolean> result = SyncClientAdaptor
+ .getUnregisteredMeasurements(client, partitionGroup.getHeader(), seriesList);
+ for (Map.Entry<String, Boolean> entry : result.entrySet()) {
+ if (!entry.getValue()) {
+ unregistered.add(entry.getKey());
+ }
+ }
+ } catch (TException | IOException e) {
+ logger.error("{}: cannot getting unregistered series list {} from {}", name,
Review comment:
Thanks for your suggestion. I will change that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r446623903
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1586,7 +1586,7 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
}
return executeNonQuery(plan);
} catch (MetadataException e) {
- logger.error("Failed to set storage group or create timeseries, because {}", e);
+ logger.error(String.format("Failed to set storage group or create timeseries, because %s", e));
Review comment:
Why change this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444018418
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,38 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
Review comment:
maybe you can use `processNonPartitionedMetaPlan` directly to avoid unnecessary judgement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444699336
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java
##########
@@ -1610,6 +1623,34 @@ public void getAggrResult(GetAggrResultRequest request,
resultHandler.onComplete(resultBuffers);
}
+ /**
+ * Check if the given measurements are registered or not
+ * @param header
+ * @param timeseriesList
+ * @param resultHandler
+ * @throws TException
+ */
+ @Override
+ public void getUnregisteredTimeseries(Node header, List<String> timeseriesList,
+ AsyncMethodCallback<List<String>> resultHandler) throws TException {
+ if (!syncLeader()) {
Review comment:
Thank you for your suggestion. The method `syncLeaderWithConsistencyCheck` will be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r443933514
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1520,12 +1534,31 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && IoTDBDescriptor.getInstance().getConfig()
Review comment:
The problem you mentioned refers to `CreateTimeseriesPlan`, `InsertTimeseriesPlan`, `DeleteTimeseriesPlan` and `DeleteTimeseriesPlan`. However, if storage group is not set when executing `DeleteTimeseriesPlan` and `DeleteTimeseriesPlan`, logically there's no necessity for the storage group to be created. As in the method `executeNonQuery` in `MetaGroupMember`, these two plans will trigger `processNonPartitionDataPlan` and `processNonPartitionMetaPlan` respectively. `CreateTimeseriesPlan` and `InsertTimeseriesPlan` are "partitioned plans". As a result, if storage group is not set for these 2 types, auto creation will be performed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444028575
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1684,6 +1739,77 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
return status;
}
+ /**
+ * Create timeseries automatically
+ *
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ String storageGroupName;
+ try {
+ storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ } catch (MetadataException e) {
+ logger.error("Failed to infer storage group from deviceId {}", deviceId);
+ return false;
+ }
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ PartitionGroup partitionGroup = partitionTable.route(storageGroupName, 0);
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ // TODO-Cluster: add executeNonQueryBatch()
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
Review comment:
maybe you can use `processPartitionedPlan` directly to avoid unnecessary judgement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-648913698
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E.png' alt='E' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444018632
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,38 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
Review comment:
maybe you can use `processNonPartitionedMetaPlan` directly to avoid unnecessary judgement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647631179
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r443916570
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java
##########
@@ -1073,7 +1072,12 @@ public void pullTimeSeriesSchema(PullSchemaRequest request,
/**
* Create an IPointReader of "path" with “timeFilter” and "valueFilter". A synchronization with
+<<<<<<< HEAD
+ * the leader will be performed first to preserve strong consistency. TODO-Cluster: also support
+ * weak consistency
+=======
* the leader will be performed according to consistency level
+>>>>>>> origin/cluster_new
Review comment:
Please resolve the conflicts.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java
##########
@@ -1610,6 +1623,34 @@ public void getAggrResult(GetAggrResultRequest request,
resultHandler.onComplete(resultBuffers);
}
+ /**
+ * Check if the given measurements are registered or not
+ * @param header
+ * @param timeseriesList
+ * @param resultHandler
+ * @throws TException
+ */
+ @Override
+ public void getUnregisteredTimeseries(Node header, List<String> timeseriesList,
+ AsyncMethodCallback<List<String>> resultHandler) throws TException {
+ if (!syncLeader()) {
Review comment:
I think there is a method `syncLeaderWithConsistencyCheck` now, and that one should be used.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,29 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ throw new MetadataException("Failed to set storage group " + storageGroupName);
Review comment:
Report the status code and its message in the exception.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java
##########
@@ -1610,6 +1623,34 @@ public void getAggrResult(GetAggrResultRequest request,
resultHandler.onComplete(resultBuffers);
}
+ /**
+ * Check if the given measurements are registered or not
+ * @param header
+ * @param timeseriesList
+ * @param resultHandler
+ * @throws TException
+ */
+ @Override
+ public void getUnregisteredTimeseries(Node header, List<String> timeseriesList,
+ AsyncMethodCallback<List<String>> resultHandler) throws TException {
+ if (!syncLeader()) {
+ resultHandler.onError(new LeaderUnknownException(getAllNodes()));
+ return;
+ }
+ List<String> result = new ArrayList<>();
+ for (String seriesPath : timeseriesList) {
+ try {
+ List<String> path = MManager.getInstance().getAllTimeseriesName(seriesPath);
+ if (path.size() != 1) {
+ throw new MetadataException("Size of the path is not 1.");
Review comment:
Better add the seriesPath which triggers the exception into the message.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-650710033
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E.png' alt='E' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444014901
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1667,6 +1710,18 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
status = forwardPlan(plan, partitionGroup);
}
if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ if (plan instanceof InsertPlan
Review comment:
It seems that this function `forwardPlan` will only be used in `processNonPartitionedDataPlan`, which will be called only when the type of plan is `DeletePlan` or `DeleteTimeSeriesPlan`, so maybe these new code in this function will never be called?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-648913698
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E.png' alt='E' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-645916156
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='9.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [9.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444026878
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
Review comment:
maybe you can use `processPartitionedPlan` directly to avoid unnecessary judgement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444929928
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1684,6 +1739,77 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
return status;
}
+ /**
+ * Create timeseries automatically
+ *
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ String storageGroupName;
+ try {
+ storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ } catch (MetadataException e) {
+ logger.error("Failed to infer storage group from deviceId {}", deviceId);
+ return false;
+ }
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ PartitionGroup partitionGroup = partitionTable.route(storageGroupName, 0);
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ // TODO-Cluster: add executeNonQueryBatch()
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
Review comment:
You are right. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r446626622
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1586,7 +1586,7 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
}
return executeNonQuery(plan);
} catch (MetadataException e) {
- logger.error("Failed to set storage group or create timeseries, because {}", e);
+ logger.error(String.format("Failed to set storage group or create timeseries, because %s", e));
Review comment:
It was recognized as a bug according to rules of sonar, "Fewer arguments provided (0) than placeholders specified (1) ".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647603362
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444700377
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/DataGroupMember.java
##########
@@ -1610,6 +1623,34 @@ public void getAggrResult(GetAggrResultRequest request,
resultHandler.onComplete(resultBuffers);
}
+ /**
+ * Check if the given measurements are registered or not
+ * @param header
+ * @param timeseriesList
+ * @param resultHandler
+ * @throws TException
+ */
+ @Override
+ public void getUnregisteredTimeseries(Node header, List<String> timeseriesList,
+ AsyncMethodCallback<List<String>> resultHandler) throws TException {
+ if (!syncLeader()) {
+ resultHandler.onError(new LeaderUnknownException(getAllNodes()));
+ return;
+ }
+ List<String> result = new ArrayList<>();
+ for (String seriesPath : timeseriesList) {
+ try {
+ List<String> path = MManager.getInstance().getAllTimeseriesName(seriesPath);
+ if (path.size() != 1) {
+ throw new MetadataException("Size of the path is not 1.");
Review comment:
Thanks for your suggestion. The information of the series will be included.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444929324
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1684,6 +1743,77 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
return status;
}
+ /**
+ * Create timeseries automatically
+ *
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ String storageGroupName;
+ try {
+ storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ } catch (MetadataException e) {
+ logger.error("Failed to infer storage group from deviceId {}", deviceId);
+ return false;
+ }
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ PartitionGroup partitionGroup = partitionTable.route(storageGroupName, 0);
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ // TODO-Cluster: add executeNonQueryBatch()
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
+ }
+ }
+ return true;
+ }
+
+
+ /**
+ * To check which timeseries in the input list is unregistered
+ *
+ * @param seriesList
+ * @param partitionGroup
+ * @return
+ */
+ List<String> getUnregisteredSeriesList(List<String> seriesList, PartitionGroup partitionGroup) {
+ Set<String> unregistered = new HashSet<>();
+ for (Node node : partitionGroup) {
+ try {
+ DataClient client = getDataClient(node);
+ List<String> result = SyncClientAdaptor
+ .getUnregisteredMeasurements(client, partitionGroup.getHeader(), seriesList);
+ unregistered.addAll(result);
Review comment:
You are right. Finishing the loop is not necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647877532
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r443676776
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
Review comment:
Thanks for your reminding. I will add the comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444929476
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1684,6 +1743,77 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
return status;
}
+ /**
+ * Create timeseries automatically
+ *
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ String storageGroupName;
+ try {
+ storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ } catch (MetadataException e) {
+ logger.error("Failed to infer storage group from deviceId {}", deviceId);
+ return false;
+ }
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ PartitionGroup partitionGroup = partitionTable.route(storageGroupName, 0);
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ // TODO-Cluster: add executeNonQueryBatch()
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
+ }
+ }
+ return true;
+ }
+
+
+ /**
+ * To check which timeseries in the input list is unregistered
+ *
+ * @param seriesList
+ * @param partitionGroup
+ * @return
+ */
+ List<String> getUnregisteredSeriesList(List<String> seriesList, PartitionGroup partitionGroup) {
+ Set<String> unregistered = new HashSet<>();
+ for (Node node : partitionGroup) {
+ try {
+ DataClient client = getDataClient(node);
+ List<String> result = SyncClientAdaptor
+ .getUnregisteredMeasurements(client, partitionGroup.getHeader(), seriesList);
+ unregistered.addAll(result);
+ } catch (TException | IOException e) {
+ logger.error("{}: cannot getting unregistered {} and other {} paths from {}", name,
+ seriesList.get(0), seriesList.get(seriesList.size() - 1), node, e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ logger.error("{}: getting unregistered series list {} is interrupted from {}", name,
+ Arrays.toString(seriesList.toArray(new String[0])), node, e);
Review comment:
Thanks for your reminding.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-645916156
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='9.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [9.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647603362
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #1387: Create schema automatically
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r442598352
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1551,6 +1584,20 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
subStatus = forwardPlan(entry.getKey(), entry.getValue());
}
if (subStatus.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ if (entry.getKey() instanceof InsertPlan
+ && subStatus.getCode() == TSStatusCode.STORAGE_ENGINE_ERROR.getStatusCode()
+ && IoTDBDescriptor.getInstance().getConfig().isAutoCreateSchemaEnabled()) {
+ // try to create timeseries
+ boolean hasCreate = autoCreateTimeseries((InsertPlan) entry.getKey(), entry.getValue());
Review comment:
"entry.getValue()" is the data group you are going to send the plan to, and it may not be the group that should hold the metadata, so create timeseries plans should be forward to the group that should hold the metadata.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1520,12 +1534,31 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && IoTDBDescriptor.getInstance().getConfig()
+ .isAutoCreateSchemaEnabled()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
+ if (setStorageGroupResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ throw new MetadataException("Failed to set storage group " + storageGroupName);
+ }
+ return executeNonQuery(plan);
+ } catch (MetadataException e) {
+ logger.info("Failed to set storage group of device id {}", deviceId);
+ }
+ }
+ logger.error("{}: Cannot found storage groups for {}", name, plan);
return StatusUtils.NO_STORAGE_GROUP;
}
- logger.debug("{}: The data groups of {} are {}", name, plan, planGroupMap);
+ logger.error("{}: The data groups of {} are {}", name, plan, planGroupMap);
Review comment:
This should still be debug.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
Review comment:
The schema may be auto-created concurrently, so I think the status code should be carefully checked, and if it says that the timeseries are already created, we should also return true.
##########
File path: service-rpc/src/main/thrift/cluster.thrift
##########
@@ -350,6 +350,8 @@ service TSDataService extends RaftService {
list<binary> getAggrResult(1:GetAggrResultRequest request)
+ map<string, bool> isMeasurementsRegistered(1: Node header, 2: list<string> measurements)
Review comment:
A suggestion about the interface design: I think the return type can be a list<bool> with the length of `measurements` because the sender clearly know what series it queries and the receiver need not send them back.
Another thing, I think "measurement" is not proper here, better just call them timeseries.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1551,6 +1584,20 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
subStatus = forwardPlan(entry.getKey(), entry.getValue());
}
if (subStatus.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ if (entry.getKey() instanceof InsertPlan
+ && subStatus.getCode() == TSStatusCode.STORAGE_ENGINE_ERROR.getStatusCode()
Review comment:
I think we should add a specific status code like "TSStatusCode.NO_TIMESERIES", as this one is too abstract and we are not sure whether it is because of no metadata.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
Review comment:
Please add a TODO here: "TODO-Cluster: add executeNonQueryBatch()", so we can cut the number of communications later.
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
+ }
+ }
+ return true;
+ }
+
+
+
+ /**
+ * To check which timeseries in the input list is unregistered
+ * @param seriesList
+ * @param partitionGroup
+ * @return
+ */
+ List<String> getUnregisteredSeriesList(List<String> seriesList, PartitionGroup partitionGroup) {
+ Set<String> unregistered = new HashSet<>();
+ for (Node node : partitionGroup) {
+ try {
+ DataClient client = getDataClient(node);
+ Map<String, Boolean> result = SyncClientAdaptor
+ .getUnregisteredMeasurements(client, partitionGroup.getHeader(), seriesList);
+ for (Map.Entry<String, Boolean> entry : result.entrySet()) {
+ if (!entry.getValue()) {
+ unregistered.add(entry.getKey());
+ }
+ }
+ } catch (TException | IOException e) {
+ logger.error("{}: cannot getting unregistered series list {} from {}", name,
Review comment:
getting -> get
I would suggest print something like `"{} and other {} paths", seriesList.get(0), seriesList.size() - 1` instead of `"{}", serieseList`, in case that the list is too long.
And if you want to print a list in a log, you can just use the list as a parameter and there is no need to wrap it with "Arrays.toString(seriesList.toArray(new String[0]))".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-650710033
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E.png' alt='E' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [12 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647891127
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444719702
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1569,6 +1616,70 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
return status;
}
+ /**
+ * Create timeseries automatically
+ * @param insertPlan, some of the timeseries in it are not created yet
+ * @param partitionGroup
+ * @return true of all uncreated timeseries are created
+ */
+ boolean autoCreateTimeseries(InsertPlan insertPlan, PartitionGroup partitionGroup) {
+ List<String> seriesList = new ArrayList<>();
+ String deviceId = insertPlan.getDeviceId();
+ for (String measurementId : insertPlan.getMeasurements()) {
+ seriesList.add(
+ new StringContainer(new String[]{deviceId, measurementId}, TsFileConstant.PATH_SEPARATOR)
+ .toString());
+ }
+ List<String> unregisteredSeriesList = getUnregisteredSeriesList(seriesList, partitionGroup);
+ for (String seriesPath : unregisteredSeriesList) {
+ int index = seriesList.indexOf(seriesPath);
+ TSDataType dataType = TypeInferenceUtils
+ .getPredictedDataType(insertPlan.getValues()[index], true);
+ TSEncoding encoding = getDefaultEncoding(dataType);
+ CompressionType compressionType = TSFileDescriptor.getInstance().getConfig().getCompressor();
+ CreateTimeSeriesPlan createTimeSeriesPlan = new CreateTimeSeriesPlan(new Path(seriesPath),
+ dataType, encoding, compressionType, null, null, null, null);
+ TSStatus result = executeNonQuery(createTimeSeriesPlan);
+ if (result.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ logger.error("{} failed to execute create timeseries {}", thisNode, seriesPath);
+ return false;
Review comment:
Thanks for your suggestion. An extra condition will be added in the `if` clause to make sure of it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444928084
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -914,8 +914,26 @@ public void insert(InsertPlan insertPlan) throws QueryProcessException {
insertPlan.setSchemasAndTransferType(schemas);
StorageEngine.getInstance().insert(insertPlan);
if (insertPlan.getFailedMeasurements() != null) {
- throw new StorageEngineException(
- "failed to insert points " + insertPlan.getFailedMeasurements());
+ // check if all path not exist exceptions
+ List<String> failedPaths = new ArrayList<>(insertPlan.getFailedMeasurements().keySet());
+ List<Exception> exceptions = new ArrayList<>(insertPlan.getFailedMeasurements().values());
+ boolean isPathNotExistException = true;
+ for(Exception e : exceptions){
+ Exception curException = e;
+ while(curException.getCause() != null){
+ curException = (Exception) curException.getCause();
Review comment:
Thanks for yoru suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444703105
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1530,9 +1544,38 @@ private TSStatus processPartitionedPlan(PhysicalPlan plan) throws UnsupportedPla
} catch (MetadataException e) {
logger.error("Cannot route plan {}", plan, e);
}
- // the storage group is not found locally, forward it to the leader
+ // the storage group is not found locally
if (planGroupMap == null || planGroupMap.isEmpty()) {
- logger.debug("{}: Cannot found storage groups for {}", name, plan);
+ if (plan instanceof InsertPlan && ClusterDescriptor.getInstance().getConfig()
+ .isEnableAutoCreateSchema()) {
+ // try to set storage group
+ String deviceId = ((InsertPlan) plan).getDeviceId();
+ try {
+ String storageGroupName = MetaUtils
+ .getStorageGroupNameByLevel(deviceId, IoTDBDescriptor.getInstance()
+ .getConfig().getDefaultStorageGroupLevel());
+ SetStorageGroupPlan setStorageGroupPlan = new SetStorageGroupPlan(
+ new Path(storageGroupName));
+ TSStatus setStorageGroupResult = executeNonQuery(setStorageGroupPlan);
Review comment:
Good idea. Thanks for your suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
LebronAl commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444014901
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1667,6 +1710,18 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
status = forwardPlan(plan, partitionGroup);
}
if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ if (plan instanceof InsertPlan
Review comment:
It seems that this function `forwardPlan` will only be used in `processNonPartitionedDataPlan`, which will be called only when the type of plan is `DeletePlan` or `DeleteTimeSeriesPlan`, so maybe these new code in this function is useless?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444706948
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1551,6 +1584,20 @@ TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap) {
subStatus = forwardPlan(entry.getKey(), entry.getValue());
}
if (subStatus.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ if (entry.getKey() instanceof InsertPlan
+ && subStatus.getCode() == TSStatusCode.STORAGE_ENGINE_ERROR.getStatusCode()
Review comment:
Thanks for your suggestion. The error code `TIMESERIES_NOT_EXIST` will be introduced.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r444702688
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/server/member/MetaGroupMember.java
##########
@@ -1667,6 +1710,18 @@ TSStatus forwardPlan(List<PartitionGroup> partitionGroups, PhysicalPlan plan) {
status = forwardPlan(plan, partitionGroup);
}
if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ if (plan instanceof InsertPlan
Review comment:
You are right. Thanks for your reminding. These codes will be moved to `TSStatus forwardPlan(Map<PhysicalPlan, PartitionGroup> planGroupMap, PhysicalPlan plan)`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#issuecomment-647852422
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL) [11 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=1387&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=1387&metric=new_duplicated_lines_density&view=list)
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning.png' alt='warning' width='16' height='16' /> The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more [here](https://sonarcloud.io/documentation/upcoming/)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #1387: [IOTDB-709] Create schema automatically
Posted by GitBox <gi...@apache.org>.
Ring-k commented on a change in pull request #1387:
URL: https://github.com/apache/incubator-iotdb/pull/1387#discussion_r443680766
##########
File path: service-rpc/src/main/thrift/cluster.thrift
##########
@@ -350,6 +350,8 @@ service TSDataService extends RaftService {
list<binary> getAggrResult(1:GetAggrResultRequest request)
+ map<string, bool> isMeasurementsRegistered(1: Node header, 2: list<string> measurements)
Review comment:
I will change the interface, specifically, the return, to be a list f string, `List<String>,` which are unregistered series.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org