You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/05/24 08:23:54 UTC
[GitHub] [iotdb] SpriCoder opened a new pull request, #6002: [IOTDB-3258] implement count nodes
SpriCoder opened a new pull request, #6002:
URL: https://github.com/apache/iotdb/pull/6002
See details: https://issues.apache.org/jira/browse/IOTDB-3258.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r881287491
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/analyze/Analyzer.java:
##########
@@ -1282,6 +1283,31 @@ public Analysis visitCountLevelTimeSeries(
return analysis;
}
+ @Override
+ public Analysis visitCountNodes(CountNodesStatement countStatement, MPPQueryContext context) {
Review Comment:
Most part of codes in this method is duplicated with `visitShowChildPaths()`. Suggest to merge them with a private method
##########
confignode/src/main/java/org/apache/iotdb/confignode/persistence/executor/ConfigRequestExecutor.java:
##########
@@ -252,35 +253,53 @@ public void loadSnapshot(File latestSnapshotRootDir) {
});
}
- private DataSet getSchemaNodeManagementPartiton(ConfigRequest req)
+ private DataSet getSchemaNodeManagementPartition(ConfigRequest req)
throws UnknownPhysicalPlanTypeException {
- Pair<Set<String>, Set<PartialPath>> matchedChildInNextLevel;
+ int level = -1;
Review Comment:
Unnecessary initialization here for `level`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#issuecomment-1136583057
Test on 3 confignode and 3 datanode:
<img width="318" alt="image" src="https://user-images.githubusercontent.com/46039728/170156814-fd4b20fc-0550-48d3-9994-0879461d3944.png">
<img width="296" alt="image" src="https://user-images.githubusercontent.com/46039728/170156849-729ea459-1924-4177-8c93-1b2c3fe00196.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
qiaojialin merged PR #6002:
URL: https://github.com/apache/iotdb/pull/6002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r881137793
##########
confignode/src/test/java/org/apache/iotdb/confignode/service/thrift/ConfigNodeRPCServiceProcessorTest.java:
##########
@@ -1000,4 +1003,61 @@ private void cleanUserAndRole() throws TException {
Assert.assertEquals(TSStatusCode.SUCCESS_STATUS.getStatusCode(), status.getCode());
}
}
+
+ @Test
+ public void testDeleteStorageGroup() throws TException {
Review Comment:
Ok, get it! I will have a check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r881150358
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LogicalPlanner.java:
##########
@@ -423,6 +424,15 @@ public PlanNode visitCountLevelTimeSeries(
.getRoot();
}
+ @Override
+ public PlanNode visitCountNodes(CountNodesStatement countStatement, MPPQueryContext context) {
+ LogicalPlanBuilder planBuilder = new LogicalPlanBuilder(context);
+ return planBuilder
+ .planChildPathsSchemaSource(countStatement.getPartialPath(), countStatement.getLevel())
+ .planCountMerge()
+ .getRoot();
Review Comment:
Besides, I use get child path in count node, so if there is possible having same path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r882533402
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LocalExecutionPlanner.java:
##########
@@ -467,40 +467,51 @@ public Operator visitLevelTimeSeriesCount(
@Override
public Operator visitChildPathsSchemaScan(
Review Comment:
fixed.
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LogicalPlanBuilder.java:
##########
@@ -607,20 +607,31 @@ public LogicalPlanBuilder planLevelTimeSeriesCountSource(
return this;
}
- public LogicalPlanBuilder planChildPathsSchemaSource(PartialPath partialPath) {
- this.root = new ChildPathsSchemaScanNode(context.getQueryId().genPlanNodeId(), partialPath);
+ public LogicalPlanBuilder planChildPathsSchemaSource(PartialPath partialPath, Integer level) {
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] MarcosZyk commented on pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
MarcosZyk commented on PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#issuecomment-1136887353
If the conduction cost of the above suggestion is heave, the suggested simplified modification is that we should not extend the interface of child management with a parameter level, which is confusing, instead we should implement independent node management configRequest and NodeScanOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#issuecomment-1136683606
Already fixed! PTAL @yifuzhou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r882708653
##########
server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/schema/NodePathsConvertOperator.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.mpp.execution.operator.schema;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.db.mpp.common.header.HeaderConstant;
+import org.apache.iotdb.db.mpp.execution.operator.Operator;
+import org.apache.iotdb.db.mpp.execution.operator.OperatorContext;
+import org.apache.iotdb.db.mpp.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.tsfile.read.common.block.TsBlock;
+import org.apache.iotdb.tsfile.read.common.block.TsBlockBuilder;
+import org.apache.iotdb.tsfile.utils.Binary;
+
+import com.google.common.util.concurrent.ListenableFuture;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.util.Objects.requireNonNull;
+
+public class NodePathsConvertOperator implements ProcessOperator {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(NodePathsConvertOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator child;
+ private boolean isFinished;
+
+ public NodePathsConvertOperator(OperatorContext operatorContext, Operator child) {
+ this.operatorContext = requireNonNull(operatorContext, "operatorContext is null");
+ this.child = requireNonNull(child, "child operator is null");
+ isFinished = false;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+ return operatorContext;
+ }
+
+ @Override
+ public ListenableFuture<Void> isBlocked() {
+ return child.isBlocked();
+ }
+
+ @Override
+ public TsBlock next() {
+ isFinished = true;
+ TsBlock block = child.next();
+ TsBlockBuilder tsBlockBuilder =
+ new TsBlockBuilder(HeaderConstant.showChildNodesHeader.getRespDataTypes());
+
+ for (int i = 0; i < block.getPositionCount(); i++) {
+ String path = block.getColumn(0).getBinary(i).toString();
+ PartialPath partialPath;
+ try {
+ partialPath = new PartialPath(path);
+ } catch (IllegalPathException e) {
+ LOGGER.warn("Failed to convert node path to PartialPath {}", path);
+ continue;
+ }
Review Comment:
I have the same idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r881148308
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LogicalPlanner.java:
##########
@@ -423,6 +424,15 @@ public PlanNode visitCountLevelTimeSeries(
.getRoot();
}
+ @Override
+ public PlanNode visitCountNodes(CountNodesStatement countStatement, MPPQueryContext context) {
+ LogicalPlanBuilder planBuilder = new LogicalPlanBuilder(context);
+ return planBuilder
+ .planChildPathsSchemaSource(countStatement.getPartialPath(), countStatement.getLevel())
+ .planCountMerge()
+ .getRoot();
Review Comment:
From my perspective, there is not allowed to have both root.ln and root.ln.a1.a2.a3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] yifuzhou commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
yifuzhou commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r881128142
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LogicalPlanner.java:
##########
@@ -423,6 +424,15 @@ public PlanNode visitCountLevelTimeSeries(
.getRoot();
}
+ @Override
+ public PlanNode visitCountNodes(CountNodesStatement countStatement, MPPQueryContext context) {
+ LogicalPlanBuilder planBuilder = new LogicalPlanBuilder(context);
+ return planBuilder
+ .planChildPathsSchemaSource(countStatement.getPartialPath(), countStatement.getLevel())
+ .planCountMerge()
+ .getRoot();
Review Comment:
How about both sg above and sg below need to count together? like there has sg: root.ln, root.abc.a1.a2.a3 and ts:root.ln.a1.a2.a3. We want count nodes root.** level=3? I think planNodeManagementMemoryMerge() should be used.
##########
confignode/src/test/java/org/apache/iotdb/confignode/service/thrift/ConfigNodeRPCServiceProcessorTest.java:
##########
@@ -1000,4 +1003,61 @@ private void cleanUserAndRole() throws TException {
Assert.assertEquals(TSStatusCode.SUCCESS_STATUS.getStatusCode(), status.getCode());
}
}
+
+ @Test
+ public void testDeleteStorageGroup() throws TException {
Review Comment:
It seems that previous pr changes has been removed... This function is the same as deleteStorageGroupTest(). I just change the name. Test class uses ....Test format and test function uses test... format should be more normalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] MarcosZyk commented on pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
MarcosZyk commented on PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#issuecomment-1136842409
It seems the node management feature is a super feature of child node management. Shall we rename the concrete implementation class to something like "NodeQueryScanOperator" or "NodeQueryTask", rather than name start with child? The child query will be parsed to a node query for ```pattern.*```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] MarcosZyk commented on a diff in pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
MarcosZyk commented on code in PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#discussion_r882519153
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LogicalPlanBuilder.java:
##########
@@ -607,20 +607,31 @@ public LogicalPlanBuilder planLevelTimeSeriesCountSource(
return this;
}
- public LogicalPlanBuilder planChildPathsSchemaSource(PartialPath partialPath) {
- this.root = new ChildPathsSchemaScanNode(context.getQueryId().genPlanNodeId(), partialPath);
+ public LogicalPlanBuilder planChildPathsSchemaSource(PartialPath partialPath, Integer level) {
Review Comment:
Rename this to planNodePathsSchemaSource, or anything without child but related to node management
##########
server/src/main/java/org/apache/iotdb/db/mpp/plan/planner/LocalExecutionPlanner.java:
##########
@@ -467,40 +467,51 @@ public Operator visitLevelTimeSeriesCount(
@Override
public Operator visitChildPathsSchemaScan(
Review Comment:
Rename to ```visitNodePathsSchemaScan```
##########
server/src/main/java/org/apache/iotdb/db/mpp/execution/operator/schema/NodePathsConvertOperator.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.mpp.execution.operator.schema;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.db.mpp.common.header.HeaderConstant;
+import org.apache.iotdb.db.mpp.execution.operator.Operator;
+import org.apache.iotdb.db.mpp.execution.operator.OperatorContext;
+import org.apache.iotdb.db.mpp.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.tsfile.read.common.block.TsBlock;
+import org.apache.iotdb.tsfile.read.common.block.TsBlockBuilder;
+import org.apache.iotdb.tsfile.utils.Binary;
+
+import com.google.common.util.concurrent.ListenableFuture;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.util.Objects.requireNonNull;
+
+public class NodePathsConvertOperator implements ProcessOperator {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(NodePathsConvertOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator child;
+ private boolean isFinished;
+
+ public NodePathsConvertOperator(OperatorContext operatorContext, Operator child) {
+ this.operatorContext = requireNonNull(operatorContext, "operatorContext is null");
+ this.child = requireNonNull(child, "child operator is null");
+ isFinished = false;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+ return operatorContext;
+ }
+
+ @Override
+ public ListenableFuture<Void> isBlocked() {
+ return child.isBlocked();
+ }
+
+ @Override
+ public TsBlock next() {
+ isFinished = true;
+ TsBlock block = child.next();
+ TsBlockBuilder tsBlockBuilder =
+ new TsBlockBuilder(HeaderConstant.showChildNodesHeader.getRespDataTypes());
+
+ for (int i = 0; i < block.getPositionCount(); i++) {
+ String path = block.getColumn(0).getBinary(i).toString();
+ PartialPath partialPath;
+ try {
+ partialPath = new PartialPath(path);
+ } catch (IllegalPathException e) {
+ LOGGER.warn("Failed to convert node path to PartialPath {}", path);
+ continue;
+ }
Review Comment:
Hope for an improvement of generic support on TsBlock. Thus the try-catch will be saved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SpriCoder commented on pull request #6002: [IOTDB-3258] implement count nodes
Posted by GitBox <gi...@apache.org>.
SpriCoder commented on PR #6002:
URL: https://github.com/apache/iotdb/pull/6002#issuecomment-1138242785
test on 3c3d again:
<img width="261" alt="image" src="https://user-images.githubusercontent.com/46039728/170438274-8a461d92-9aa0-4cd3-9f89-b35406712a5b.png">
<img width="256" alt="image" src="https://user-images.githubusercontent.com/46039728/170438303-666dbd34-61a9-4548-8bb1-8bdfe96de433.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org