You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/05/11 16:02:07 UTC
[GitHub] [iotdb] mzp0514 opened a new pull request #3162: [IOTDB-1143] Add continuous query
mzp0514 opened a new pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162
## Description
### Content1 ...
### Content2 ...
### Content3 ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error
conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition,
design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design
(or naming) decision point and compare the alternatives with the designs that you've implemented
(or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere
(e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list),
link to that discussion from this PR description and explain what have changed in your final design
compared to your original proposal or the consensus version in the end of the discussion.
If something hasn't changed since the original discussion, you can omit a detailed discussion of
those aspects of the design here, perhaps apart from brief mentioning for the sake of readability
of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] concurrent read
- [ ] concurrent write
- [ ] concurrent read and write
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods.
- [ ] added or updated version, __license__, or notice information
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious
for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold
for code coverage.
- [ ] added integration tests.
- [ ] been tested in a test IoTDB cluster.
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items
apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items
from the checklist above are strictly necessary, but it would be very helpful if you at least
self-review the PR. -->
<hr>
##### Key changed/added classes (or packages if there are too many classes) in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656860970
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/CreateContinuousQueryPlan.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.qp.physical.sys;
+
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.physical.PhysicalPlan;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import java.nio.ByteBuffer;
+import java.util.*;
+
+public class CreateContinuousQueryPlan extends PhysicalPlan implements java.io.Serializable {
+
+ private String querySql;
+ private String continuousQueryName;
+ private PartialPath targetPath;
+ private long everyInterval;
+ private long forInterval;
+ private transient QueryOperator queryOperator;
Review comment:
because the field needn't serialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656864754
##########
File path: server/src/test/java/org/apache/iotdb/db/integration/IoTDBContinuousQueryIT.java
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.integration;
+
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.jdbc.Config;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.*;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Notice that, all test begins with "IoTDB" is integration test. All test which will start the
+ * IoTDB server should be defined as integration test.
+ */
+public class IoTDBContinuousQueryIT {
Review comment:
> Add tests for `show cq`.
has been tested in checkContinuousQueries method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r659418063
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.ShutdownException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static long CHECK_INTERVAL =
+ IoTDBDescriptor.getInstance().getConfig().getContinuousQueryMinimumEveryInterval() / 2;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final ConcurrentHashMap<String, CreateContinuousQueryPlan> continuousQueryPlans =
+ new ConcurrentHashMap<>();
+
+ private final ConcurrentHashMap<String, Long> nextExecutionTimestamps = new ConcurrentHashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService checkThread;
+
+ protected static final ContinuousQueryTaskPoolManager TASK_POOL_MANAGER =
+ ContinuousQueryTaskPoolManager.getInstance();
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long durationFromCreation = System.currentTimeMillis() - plan.getCreationTimestamp();
+ long nextExecutionTimestamp =
+ plan.getCreationTimestamp()
+ + plan.getEveryInterval()
+ * (durationFromCreation / plan.getEveryInterval()
+ + ((durationFromCreation % plan.getEveryInterval() == 0) ? 0 : 1));
+ nextExecutionTimestamps.put(plan.getContinuousQueryName(), nextExecutionTimestamp);
+ }
+
+ checkThread = Executors.newSingleThreadScheduledExecutor();
+ checkThread.scheduleAtFixedRate(
+ this::checkAndSubmitTasks,
+ 0,
+ CHECK_INTERVAL,
+ DatetimeUtils.timestampPrecisionStringToTimeUnit(
+ IoTDBDescriptor.getInstance().getConfig().getTimestampPrecision()));
+
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (checkThread != null) {
+ checkThread.shutdown();
+ try {
+ checkThread.awaitTermination(600, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ logger.warn("Check thread still doesn't exit after 60s");
+ checkThread.shutdownNow();
+ }
+ }
+ }
+
+ @Override
+ public void shutdown(long milliseconds) throws ShutdownException {
+ stop();
+ }
+
+ private void checkAndSubmitTasks() {
+ long currentTimestamp = System.currentTimeMillis();
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long nextExecutionTimestamp = nextExecutionTimestamps.get(plan.getContinuousQueryName());
+ if (currentTimestamp >= nextExecutionTimestamp) {
+ TASK_POOL_MANAGER.submit(new ContinuousQueryTask(plan, nextExecutionTimestamp));
+ nextExecutionTimestamps.replace(
+ plan.getContinuousQueryName(), nextExecutionTimestamp + plan.getEveryInterval());
+ }
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
+ acquireRegistrationLock();
+
+ if (continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ releaseRegistrationLock();
+ throw new ContinuousQueryAlreadyExistException(plan.getContinuousQueryName());
+ }
+
+ try {
+ if (writeLog) {
+ IoTDB.metaManager.createContinuousQuery(plan);
+ }
+ doRegister(plan);
+
+ } catch (Exception e) {
+ logger.error(e.getMessage());
+ } finally {
+ releaseRegistrationLock();
+ }
+ return true;
+ }
+
+ private void doRegister(CreateContinuousQueryPlan plan) {
+ continuousQueryPlans.put(plan.getContinuousQueryName(), plan);
+ nextExecutionTimestamps.put(plan.getContinuousQueryName(), plan.getCreationTimestamp());
+ }
+
+ public void deregisterAll() throws ContinuousQueryNotExistException {
+ for (String cqName : continuousQueryPlans.keySet()) {
+ deregister(new DropContinuousQueryPlan(cqName));
+ }
+ }
+
+ public boolean deregister(DropContinuousQueryPlan plan) throws ContinuousQueryNotExistException {
+
+ acquireRegistrationLock();
+
+ if (!continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ releaseRegistrationLock();
+ throw new ContinuousQueryNotExistException(plan.getContinuousQueryName());
+ }
+
+ try {
+ IoTDB.metaManager.dropContinuousQuery(plan);
+ doDeregister(plan);
+
+ } catch (Exception e) {
+ logger.error(e.getMessage());
+ } finally {
+ releaseRegistrationLock();
+ }
Review comment:
```suggestion
try {
if (!continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
throw new ContinuousQueryException(plan.getContinuousQueryName() + ": xxxx");
}
IoTDB.metaManager.dropContinuousQuery(plan);
doDeregister(plan);
} catch (ContinuousQueryException e) {
throw e;
} catch (Exception e) {
throw new ContinuousQueryException(e.getMessage());
} finally {
releaseRegistrationLock();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-869664710
Performance report: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181311232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656127991
##########
File path: antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/SqlBase.g4
##########
@@ -105,6 +105,13 @@ statement
| STOP TRIGGER triggerName=ID #stopTrigger
| SHOW TRIGGERS #showTriggers
| selectClause fromClause whereClause? specialClause? #selectStatement
+ | CREATE CONTINUOUS QUERY continuousQueryName=ID
Review comment:
```suggestion
| CREATE (CONTINUOUS QUERY | CQ) continuousQueryName=ID
```
What about the alias `CQ`?
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java
##########
@@ -1086,6 +1095,166 @@ private ResultColumn parseResultColumn(ResultColumnContext resultColumnContext)
resultColumnContext.AS() == null ? null : resultColumnContext.ID().getText());
}
+ @Override
+ public Operator visitDropContinuousQueryStatement(DropContinuousQueryStatementContext ctx) {
+ DropContinuousQueryOperator dropContinuousQueryOperator =
+ new DropContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_DROP);
+ dropContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+ return dropContinuousQueryOperator;
+ }
+
+ @Override
+ public Operator visitShowContinuousQueriesStatement(ShowContinuousQueriesStatementContext ctx) {
+ return new ShowContinuousQueriesOperator(SQLConstant.TOK_SHOW_CONTINUOUS_QUERIES);
+ }
+
+ @Override
+ public Operator visitCreateContinuousQueryStatement(CreateContinuousQueryStatementContext ctx) {
+ CreateContinuousQueryOperator createContinuousQueryOperator =
+ new CreateContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_CREATE);
+
+ createContinuousQueryOperator.setQuerySql(ctx.getText());
+
+ createContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+
+ if (ctx.resampleClause() != null) {
+ parseResampleClause(ctx.resampleClause(), createContinuousQueryOperator);
+ }
+
+ parseCqSelectIntoClause(ctx.cqSelectIntoClause(), createContinuousQueryOperator);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("select ");
+ sb.append(ctx.cqSelectIntoClause().selectClause().getText().substring(6));
+ sb.append(" from ");
+ sb.append(ctx.cqSelectIntoClause().fromClause().prefixPath(0).getText());
+
+ sb.append(" group by ([now() - ");
+ String groupByInterval = ctx.cqSelectIntoClause().cqGroupByTimeClause().DURATION().getText();
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ sb.append(groupByInterval);
+ } else {
+ List<TerminalNode> durations = ctx.resampleClause().DURATION();
+ sb.append(durations.get(durations.size() - 1).getText());
+ }
+ sb.append(", now()), ");
+ sb.append(groupByInterval);
+ sb.append(")");
+ if (queryOp.isGroupByLevel()) {
+ sb.append(", level = ");
+ sb.append(queryOp.getSpecialClauseComponent().getLevel());
+ }
+ createContinuousQueryOperator.setQuerySql(sb.toString());
+
+ if (createContinuousQueryOperator.getEveryInterval() == 0) {
+ createContinuousQueryOperator.setEveryInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ createContinuousQueryOperator.setForInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ return createContinuousQueryOperator;
+ }
+
+ public void parseResampleClause(
+ ResampleClauseContext ctx, CreateContinuousQueryOperator operator) {
+
+ if (ctx.DURATION().size() == 1) {
+ if (ctx.EVERY() != null) {
+ operator.setEveryInterval(
+ DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ } else if (ctx.FOR() != null) {
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ }
+ } else if (ctx.DURATION().size() == 2) {
+ operator.setEveryInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(1).getText()));
+ }
+ }
+
+ public void parseCqSelectIntoClause(
+ CqSelectIntoClauseContext ctx, CreateContinuousQueryOperator createContinuousQueryOperator) {
+
+ queryOp = new GroupByQueryOperator();
+
+ parseSelectClause(ctx.selectClause());
+ parseFromClause(ctx.fromClause());
+
+ if (queryOp.getSelectComponent().getResultColumns().size() > 1) {
+ throw new SQLParserException(
+ "Cq Select Into: CQ currently does not support multiple result columns.");
+ }
+
+ if (queryOp.getFromComponent().getPrefixPaths().size() > 1) {
+ throw new SQLParserException(
+ "Cq Select Into: CQ currently does not support multiple series .");
+ }
+
+ parseCqGroupByTimeClause(ctx.cqGroupByTimeClause());
+
+ int fromLen = queryOp.getFromComponent().getPrefixPaths().get(0).getNodeLength();
+ int queryLevel = queryOp.getSpecialClauseComponent().getLevel();
+ if (queryLevel >= fromLen) {
+ throw new SQLParserException(
+ "Cq Select Into: Level should not exceed the <from_prefix> length.");
+ }
+
+ PartialPath targetPath = null;
+ if (ctx.fullPath() != null) {
+ targetPath = parseFullPath(ctx.fullPath());
+ } else if (ctx.suffixPath() != null) {
+ List<String> targetNodes = new ArrayList<>();
+ int trueLevel = queryLevel;
+ if (trueLevel == -1) {
+ trueLevel = fromLen - 1;
+ }
+ for (int i = 0; i <= trueLevel; i++) {
+ targetNodes.add("${" + i + "}");
+ }
+ targetNodes.add(ctx.suffixPath().getText());
+ targetPath = new PartialPath(targetNodes.toArray(new String[0]));
+ }
+
+ createContinuousQueryOperator.setTargetPath(targetPath);
+ createContinuousQueryOperator.setQueryOperator(queryOp);
+ }
+
+ public void parseCqGroupByTimeClause(CqGroupByTimeClauseContext ctx) {
+
+ GroupByClauseComponent groupByClauseComponent = new GroupByClauseComponent();
+
+ groupByClauseComponent.setUnit(
+ parseTimeUnitOrSlidingStep(ctx.DURATION().getText(), true, groupByClauseComponent));
+
+ groupByClauseComponent.setSlidingStep(groupByClauseComponent.getUnit());
+
+ groupByClauseComponent.setLeftCRightO(true);
+
+ if (ctx.LEVEL() != null && ctx.INT() != null) {
+ groupByClauseComponent.setLevel(Integer.parseInt(ctx.INT().getText()));
+ }
+
+ queryOp.setSpecialClauseComponent(groupByClauseComponent);
+ }
+
+ @Override
+ public Operator visitResampleClause(ResampleClauseContext ctx) {
+ return visitChildren(ctx);
+ }
+
+ @Override
+ public Operator visitCqSelectIntoClause(CqSelectIntoClauseContext ctx) {
+ return visitChildren(ctx);
+ }
+
+ @Override
+ public Operator visitCqGroupByTimeClause(CqGroupByTimeClauseContext ctx) {
+ return visitChildren(ctx);
+ }
+
Review comment:
```suggestion
```
Remove them.
##########
File path: server/src/test/java/org/apache/iotdb/db/integration/IoTDBContinuousQueryIT.java
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.integration;
+
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.jdbc.Config;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.*;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Notice that, all test begins with "IoTDB" is integration test. All test which will start the
+ * IoTDB server should be defined as integration test.
+ */
+public class IoTDBContinuousQueryIT {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IoTDBContinuousQueryIT.class);
+
+ private Statement statement;
+ private Connection connection;
+ private DataGenerator generator;
+
+ String[] timeSeriesArray = {
+ "root.ln.wf01.wt01.ws01",
+ "root.ln.wf01.wt01.ws02",
+ "root.ln.wf01.wt02.ws01",
+ "root.ln.wf01.wt02.ws02",
+ "root.ln.wf02.wt01.ws01",
+ "root.ln.wf02.wt01.ws02",
+ "root.ln.wf02.wt02.ws01",
+ "root.ln.wf02.wt02.ws02"
+ };
+
+ private void createTimeSeries() throws SQLException {
+ for (String timeSeries : timeSeriesArray) {
+ statement.execute(
+ String.format(
+ "create timeseries %s.temperature with datatype=FLOAT,encoding=RLE", timeSeries));
+ }
+ }
+
+ @Before
+ public void setUp() throws Exception {
+ EnvironmentUtils.envSetUp();
+ Class.forName(Config.JDBC_DRIVER_NAME);
+ connection = DriverManager.getConnection("jdbc:iotdb://127.0.0.1:6667/", "root", "root");
+ statement = connection.createStatement();
+ generator = new DataGenerator();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+ generator.close();
+ statement.close();
+ connection.close();
+ EnvironmentUtils.cleanEnv();
+ }
+
+ @Test
+ public void testCreateAndDropContinuousQuery() throws Exception {
+
+ createTimeSeries();
+
+ statement.execute(
+ "CREATE CONTINUOUS QUERY cq1 "
+ + "BEGIN SELECT max_value(temperature) INTO temperature_max FROM root.ln.*.*.* "
+ + "GROUP BY time(10s) END");
+
+ statement.execute(
+ "CREATE CONTINUOUS QUERY cq2 "
+ + "BEGIN SELECT count(temperature) INTO temperature_cnt FROM root.ln.wf01.*.* "
+ + " GROUP BY time(10s), level=3 END");
+
+ statement.execute(
+ "CREATE CONTINUOUS QUERY cq3 "
+ + "RESAMPLE EVERY 20s FOR 20s "
+ + "BEGIN SELECT avg(temperature) INTO temperature_avg FROM root.ln.wf01.*.* "
+ + "GROUP BY time(10s), level=2 END");
+
+ statement.execute("DROP CONTINUOUS QUERY cq1");
+ statement.execute("DROP CONTINUOUS QUERY cq2");
+
+ checkContinuousQueries(new String[] {"cq3"});
+
+ EnvironmentUtils.stopDaemon();
+
+ setUp();
+
+ checkContinuousQueries(new String[] {"cq3"});
+
+ try {
+
+ statement.execute(
+ "CREATE CONTINUOUS QUERY cq3 "
+ + "RESAMPLE EVERY 20s FOR 20s "
+ + "BEGIN SELECT avg(temperature) INTO temperature_avg FROM root.ln.wf01.*.* "
+ + "GROUP BY time(10s), level=2 END");
+ } catch (Exception e) {
Review comment:
Try to create cq1 and cq2 again.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+ pool = IoTDBThreadPoolFactory.newScheduledThreadPool(10, "Continuous Query Service");
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ @Override
+ public void waitAndStop(long milliseconds) {
+ for (ScheduledFuture<?> future : continuousQueriesFutures.values()) {
+ future.cancel(false);
+ }
+ logger.info("Waiting for task pool to shut down");
+ try {
+ Thread.sleep(milliseconds);
+ } catch (InterruptedException e) {
+ logger.info("Thread interrupted");
+ Thread.currentThread().interrupt();
+ }
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
+ if (continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ throw new ContinuousQueryAlreadyExistException(plan.getContinuousQueryName());
+ }
+
+ if (writeLog) {
+ try {
+ IoTDB.metaManager.createContinuousQuery(plan);
+ } catch (IOException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ doRegister(plan);
+ return true;
+ }
+
+ private void doRegister(CreateContinuousQueryPlan plan) {
+ acquireRegistrationLock();
+
+ try {
+
+ ContinuousQuery cq = new ContinuousQuery(plan);
+ ScheduledFuture<?> future =
+ pool.scheduleAtFixedRate(
+ cq,
+ plan.getEveryInterval(),
Review comment:
Is `0` better?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+ pool = IoTDBThreadPoolFactory.newScheduledThreadPool(10, "Continuous Query Service");
Review comment:
Make the number of threads configurable.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java
##########
@@ -1086,6 +1095,166 @@ private ResultColumn parseResultColumn(ResultColumnContext resultColumnContext)
resultColumnContext.AS() == null ? null : resultColumnContext.ID().getText());
}
+ @Override
+ public Operator visitDropContinuousQueryStatement(DropContinuousQueryStatementContext ctx) {
+ DropContinuousQueryOperator dropContinuousQueryOperator =
+ new DropContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_DROP);
+ dropContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+ return dropContinuousQueryOperator;
+ }
+
+ @Override
+ public Operator visitShowContinuousQueriesStatement(ShowContinuousQueriesStatementContext ctx) {
+ return new ShowContinuousQueriesOperator(SQLConstant.TOK_SHOW_CONTINUOUS_QUERIES);
+ }
+
+ @Override
+ public Operator visitCreateContinuousQueryStatement(CreateContinuousQueryStatementContext ctx) {
+ CreateContinuousQueryOperator createContinuousQueryOperator =
+ new CreateContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_CREATE);
+
+ createContinuousQueryOperator.setQuerySql(ctx.getText());
+
+ createContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+
+ if (ctx.resampleClause() != null) {
+ parseResampleClause(ctx.resampleClause(), createContinuousQueryOperator);
+ }
+
+ parseCqSelectIntoClause(ctx.cqSelectIntoClause(), createContinuousQueryOperator);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("select ");
+ sb.append(ctx.cqSelectIntoClause().selectClause().getText().substring(6));
+ sb.append(" from ");
+ sb.append(ctx.cqSelectIntoClause().fromClause().prefixPath(0).getText());
+
+ sb.append(" group by ([now() - ");
+ String groupByInterval = ctx.cqSelectIntoClause().cqGroupByTimeClause().DURATION().getText();
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ sb.append(groupByInterval);
+ } else {
+ List<TerminalNode> durations = ctx.resampleClause().DURATION();
+ sb.append(durations.get(durations.size() - 1).getText());
+ }
+ sb.append(", now()), ");
+ sb.append(groupByInterval);
+ sb.append(")");
+ if (queryOp.isGroupByLevel()) {
+ sb.append(", level = ");
+ sb.append(queryOp.getSpecialClauseComponent().getLevel());
+ }
+ createContinuousQueryOperator.setQuerySql(sb.toString());
+
+ if (createContinuousQueryOperator.getEveryInterval() == 0) {
+ createContinuousQueryOperator.setEveryInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ createContinuousQueryOperator.setForInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ return createContinuousQueryOperator;
+ }
+
+ public void parseResampleClause(
+ ResampleClauseContext ctx, CreateContinuousQueryOperator operator) {
+
+ if (ctx.DURATION().size() == 1) {
+ if (ctx.EVERY() != null) {
+ operator.setEveryInterval(
+ DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ } else if (ctx.FOR() != null) {
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ }
+ } else if (ctx.DURATION().size() == 2) {
+ operator.setEveryInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(1).getText()));
+ }
+ }
+
+ public void parseCqSelectIntoClause(
+ CqSelectIntoClauseContext ctx, CreateContinuousQueryOperator createContinuousQueryOperator) {
+
+ queryOp = new GroupByQueryOperator();
+
+ parseSelectClause(ctx.selectClause());
+ parseFromClause(ctx.fromClause());
+
+ if (queryOp.getSelectComponent().getResultColumns().size() > 1) {
+ throw new SQLParserException(
+ "Cq Select Into: CQ currently does not support multiple result columns.");
Review comment:
Add tests?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
Review comment:
```suggestion
QueryDataSet result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
Review comment:
do while...
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
Review comment:
Can it handle an empty query?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+ pool = IoTDBThreadPoolFactory.newScheduledThreadPool(10, "Continuous Query Service");
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ @Override
+ public void waitAndStop(long milliseconds) {
+ for (ScheduledFuture<?> future : continuousQueriesFutures.values()) {
+ future.cancel(false);
+ }
+ logger.info("Waiting for task pool to shut down");
+ try {
+ Thread.sleep(milliseconds);
+ } catch (InterruptedException e) {
+ logger.info("Thread interrupted");
+ Thread.currentThread().interrupt();
+ }
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
Review comment:
```suggestion
acquireRegistrationLock();
```
Lock should be acquired here.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
Review comment:
```suggestion
private GroupByTimePlan generateQueryPlan() throws QueryProcessException {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/physical/sys/CreateContinuousQueryPlan.java
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.qp.physical.sys;
+
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.logical.Operator;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.physical.PhysicalPlan;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import java.nio.ByteBuffer;
+import java.util.*;
+
+public class CreateContinuousQueryPlan extends PhysicalPlan implements java.io.Serializable {
+
+ private String querySql;
+ private String continuousQueryName;
+ private PartialPath targetPath;
+ private long everyInterval;
+ private long forInterval;
+ private transient QueryOperator queryOperator;
Review comment:
Why transient?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
Review comment:
log level: error -> info
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java
##########
@@ -1086,6 +1095,166 @@ private ResultColumn parseResultColumn(ResultColumnContext resultColumnContext)
resultColumnContext.AS() == null ? null : resultColumnContext.ID().getText());
}
+ @Override
+ public Operator visitDropContinuousQueryStatement(DropContinuousQueryStatementContext ctx) {
+ DropContinuousQueryOperator dropContinuousQueryOperator =
+ new DropContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_DROP);
+ dropContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+ return dropContinuousQueryOperator;
+ }
+
+ @Override
+ public Operator visitShowContinuousQueriesStatement(ShowContinuousQueriesStatementContext ctx) {
+ return new ShowContinuousQueriesOperator(SQLConstant.TOK_SHOW_CONTINUOUS_QUERIES);
+ }
+
+ @Override
+ public Operator visitCreateContinuousQueryStatement(CreateContinuousQueryStatementContext ctx) {
+ CreateContinuousQueryOperator createContinuousQueryOperator =
+ new CreateContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_CREATE);
+
+ createContinuousQueryOperator.setQuerySql(ctx.getText());
+
+ createContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+
+ if (ctx.resampleClause() != null) {
+ parseResampleClause(ctx.resampleClause(), createContinuousQueryOperator);
+ }
+
+ parseCqSelectIntoClause(ctx.cqSelectIntoClause(), createContinuousQueryOperator);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("select ");
+ sb.append(ctx.cqSelectIntoClause().selectClause().getText().substring(6));
+ sb.append(" from ");
+ sb.append(ctx.cqSelectIntoClause().fromClause().prefixPath(0).getText());
+
+ sb.append(" group by ([now() - ");
+ String groupByInterval = ctx.cqSelectIntoClause().cqGroupByTimeClause().DURATION().getText();
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ sb.append(groupByInterval);
+ } else {
+ List<TerminalNode> durations = ctx.resampleClause().DURATION();
+ sb.append(durations.get(durations.size() - 1).getText());
+ }
+ sb.append(", now()), ");
+ sb.append(groupByInterval);
+ sb.append(")");
+ if (queryOp.isGroupByLevel()) {
+ sb.append(", level = ");
+ sb.append(queryOp.getSpecialClauseComponent().getLevel());
+ }
+ createContinuousQueryOperator.setQuerySql(sb.toString());
+
+ if (createContinuousQueryOperator.getEveryInterval() == 0) {
+ createContinuousQueryOperator.setEveryInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ createContinuousQueryOperator.setForInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ return createContinuousQueryOperator;
+ }
+
+ public void parseResampleClause(
+ ResampleClauseContext ctx, CreateContinuousQueryOperator operator) {
+
+ if (ctx.DURATION().size() == 1) {
+ if (ctx.EVERY() != null) {
+ operator.setEveryInterval(
+ DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ } else if (ctx.FOR() != null) {
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ }
+ } else if (ctx.DURATION().size() == 2) {
+ operator.setEveryInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(1).getText()));
+ }
+ }
+
+ public void parseCqSelectIntoClause(
+ CqSelectIntoClauseContext ctx, CreateContinuousQueryOperator createContinuousQueryOperator) {
+
+ queryOp = new GroupByQueryOperator();
+
+ parseSelectClause(ctx.selectClause());
+ parseFromClause(ctx.fromClause());
+
+ if (queryOp.getSelectComponent().getResultColumns().size() > 1) {
+ throw new SQLParserException(
+ "Cq Select Into: CQ currently does not support multiple result columns.");
+ }
+
+ if (queryOp.getFromComponent().getPrefixPaths().size() > 1) {
+ throw new SQLParserException(
+ "Cq Select Into: CQ currently does not support multiple series .");
+ }
+
+ parseCqGroupByTimeClause(ctx.cqGroupByTimeClause());
+
+ int fromLen = queryOp.getFromComponent().getPrefixPaths().get(0).getNodeLength();
+ int queryLevel = queryOp.getSpecialClauseComponent().getLevel();
+ if (queryLevel >= fromLen) {
+ throw new SQLParserException(
+ "Cq Select Into: Level should not exceed the <from_prefix> length.");
+ }
+
+ PartialPath targetPath = null;
+ if (ctx.fullPath() != null) {
+ targetPath = parseFullPath(ctx.fullPath());
+ } else if (ctx.suffixPath() != null) {
+ List<String> targetNodes = new ArrayList<>();
+ int trueLevel = queryLevel;
+ if (trueLevel == -1) {
+ trueLevel = fromLen - 1;
+ }
+ for (int i = 0; i <= trueLevel; i++) {
+ targetNodes.add("${" + i + "}");
+ }
+ targetNodes.add(ctx.suffixPath().getText());
+ targetPath = new PartialPath(targetNodes.toArray(new String[0]));
+ }
+
+ createContinuousQueryOperator.setTargetPath(targetPath);
+ createContinuousQueryOperator.setQueryOperator(queryOp);
+ }
+
+ public void parseCqGroupByTimeClause(CqGroupByTimeClauseContext ctx) {
+
+ GroupByClauseComponent groupByClauseComponent = new GroupByClauseComponent();
+
+ groupByClauseComponent.setUnit(
+ parseTimeUnitOrSlidingStep(ctx.DURATION().getText(), true, groupByClauseComponent));
+
+ groupByClauseComponent.setSlidingStep(groupByClauseComponent.getUnit());
+
+ groupByClauseComponent.setLeftCRightO(true);
+
+ if (ctx.LEVEL() != null && ctx.INT() != null) {
+ groupByClauseComponent.setLevel(Integer.parseInt(ctx.INT().getText()));
Review comment:
Can it support `mo`?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+ pool = IoTDBThreadPoolFactory.newScheduledThreadPool(10, "Continuous Query Service");
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ @Override
+ public void waitAndStop(long milliseconds) {
+ for (ScheduledFuture<?> future : continuousQueriesFutures.values()) {
+ future.cancel(false);
+ }
+ logger.info("Waiting for task pool to shut down");
+ try {
+ Thread.sleep(milliseconds);
+ } catch (InterruptedException e) {
+ logger.info("Thread interrupted");
+ Thread.currentThread().interrupt();
+ }
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
+ if (continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ throw new ContinuousQueryAlreadyExistException(plan.getContinuousQueryName());
+ }
+
+ if (writeLog) {
+ try {
+ IoTDB.metaManager.createContinuousQuery(plan);
+ } catch (IOException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ doRegister(plan);
+ return true;
+ }
+
+ private void doRegister(CreateContinuousQueryPlan plan) {
+ acquireRegistrationLock();
+
+ try {
+
+ ContinuousQuery cq = new ContinuousQuery(plan);
+ ScheduledFuture<?> future =
+ pool.scheduleAtFixedRate(
+ cq,
+ plan.getEveryInterval(),
+ plan.getEveryInterval(),
+ DatetimeUtils.toTimeUnit(
+ IoTDBDescriptor.getInstance().getConfig().getTimestampPrecision()));
+ continuousQueriesFutures.put(plan.getContinuousQueryName(), future);
+ continuousQueryPlans.put(plan.getContinuousQueryName(), plan);
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ } finally {
+ releaseRegistrationLock();
+ }
+ }
+
+ public void deregisterAll() throws ContinuousQueryNotExistException {
+ for (String cqName : continuousQueryPlans.keySet()) {
+ deregister(new DropContinuousQueryPlan(cqName));
+ }
+ }
+
+ public boolean deregister(DropContinuousQueryPlan plan) throws ContinuousQueryNotExistException {
+ if (!continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
Review comment:
Lock should be acquired here.
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##########
@@ -299,20 +306,42 @@ private int initFromLog(File logFile) throws IOException {
private int applyMlog(MLogReader mLogReader) {
int idx = 0;
+ HashMap<String, CreateContinuousQueryPlan> recoveredCQs = new HashMap<>();
while (mLogReader.hasNext()) {
PhysicalPlan plan = null;
try {
plan = mLogReader.next();
if (plan == null) {
continue;
}
- operation(plan);
+ if (plan.getOperatorType() == Operator.OperatorType.CREATE_CONTINUOUS_QUERY) {
Review comment:
I think it's ok to simply remove the recovery logic.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
Review comment:
Add comments to explain why we need a copy here.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
Review comment:
log level: error -> info
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+ pool = IoTDBThreadPoolFactory.newScheduledThreadPool(10, "Continuous Query Service");
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ @Override
+ public void waitAndStop(long milliseconds) {
+ for (ScheduledFuture<?> future : continuousQueriesFutures.values()) {
+ future.cancel(false);
+ }
+ logger.info("Waiting for task pool to shut down");
+ try {
+ Thread.sleep(milliseconds);
+ } catch (InterruptedException e) {
+ logger.info("Thread interrupted");
+ Thread.currentThread().interrupt();
+ }
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
+ if (continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ throw new ContinuousQueryAlreadyExistException(plan.getContinuousQueryName());
+ }
+
+ if (writeLog) {
+ try {
+ IoTDB.metaManager.createContinuousQuery(plan);
+ } catch (IOException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ doRegister(plan);
+ return true;
+ }
+
+ private void doRegister(CreateContinuousQueryPlan plan) {
+ acquireRegistrationLock();
+
+ try {
+
+ ContinuousQuery cq = new ContinuousQuery(plan);
+ ScheduledFuture<?> future =
+ pool.scheduleAtFixedRate(
+ cq,
+ plan.getEveryInterval(),
+ plan.getEveryInterval(),
+ DatetimeUtils.toTimeUnit(
+ IoTDBDescriptor.getInstance().getConfig().getTimestampPrecision()));
+ continuousQueriesFutures.put(plan.getContinuousQueryName(), future);
+ continuousQueryPlans.put(plan.getContinuousQueryName(), plan);
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ } finally {
+ releaseRegistrationLock();
+ }
+ }
+
+ public void deregisterAll() throws ContinuousQueryNotExistException {
+ for (String cqName : continuousQueryPlans.keySet()) {
+ deregister(new DropContinuousQueryPlan(cqName));
+ }
+ }
+
+ public boolean deregister(DropContinuousQueryPlan plan) throws ContinuousQueryNotExistException {
+ if (!continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ throw new ContinuousQueryNotExistException(plan.getContinuousQueryName());
+ }
+
+ try {
+
+ IoTDB.metaManager.dropContinuousQuery(plan);
+ doDeregister(plan);
+ } catch (Exception e) {
+
+ logger.error(e.getMessage());
+ }
+
+ return true;
+ }
+
+ private void doDeregister(DropContinuousQueryPlan plan) {
+ String cqName = plan.getContinuousQueryName();
+ continuousQueriesFutures.get(cqName).cancel(false);
+ continuousQueriesFutures.remove(cqName);
+ continuousQueryPlans.remove(cqName);
+ }
+
+ public List<ShowContinuousQueriesResult> getContinuousQueryPlans() {
Review comment:
```suggestion
public List<ShowContinuousQueriesResult> getShowContinuousQueriesResultList() {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
Review comment:
Add comments to explain why we need these field here.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
Review comment:
1024? Make it a private static field.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
+ InsertTabletPlan[] insertTabletPlans = new InsertTabletPlan[columnSize];
+ String[] measurements = new String[] {targetPaths.get(0).getMeasurement()};
+ List<Integer> dataTypes = Collections.singletonList(dataType.ordinal());
+
+ for (int i = 0; i < columnSize; i++) {
+ insertTabletPlans[i] =
+ new InsertTabletPlan(
+ new PartialPath(targetPaths.get(i).getDevice()), measurements, dataTypes);
+ }
+
+ return insertTabletPlans;
+ }
+
+ private Object[][] getColumns(int columnSize, int fetchSize, TSDataType dataType) {
Review comment:
```suggestion
private Object[][] constructColumns(int columnSize, int fetchSize, TSDataType dataType) {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
Review comment:
fetchSize is not a good name here. It is only used in the query process.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
+ InsertTabletPlan[] insertTabletPlans = new InsertTabletPlan[columnSize];
+ String[] measurements = new String[] {targetPaths.get(0).getMeasurement()};
+ List<Integer> dataTypes = Collections.singletonList(dataType.ordinal());
+
+ for (int i = 0; i < columnSize; i++) {
+ insertTabletPlans[i] =
+ new InsertTabletPlan(
+ new PartialPath(targetPaths.get(i).getDevice()), measurements, dataTypes);
+ }
+
+ return insertTabletPlans;
+ }
+
+ private Object[][] getColumns(int columnSize, int fetchSize, TSDataType dataType) {
+ Object[][] columns = new Object[columnSize][1];
+ for (int i = 0; i < columnSize; i++) {
+ switch (dataType) {
+ case DOUBLE:
+ columns[i][0] = new double[fetchSize];
+ break;
+ case INT64:
+ columns[i][0] = new long[fetchSize];
+ break;
+ case INT32:
+ columns[i][0] = new int[fetchSize];
+ break;
+ case FLOAT:
+ columns[i][0] = new float[fetchSize];
+ break;
+ default:
+ break;
+ }
+ }
+ return columns;
+ }
+
+ private void fillColumns(
+ Object[][] columns,
+ TSDataType dataType,
+ RowRecord record,
+ int[] rowNums,
+ long[][] timestamps) {
+ List<Field> fields = record.getFields();
+ long ts = record.getTimestamp();
+
+ for (int i = 0; i < columns.length; i++) {
+ Field field = fields.get(i);
+ if (field != null) {
+ timestamps[i][rowNums[i]] = ts;
+ switch (dataType) {
+ case DOUBLE:
+ ((double[]) columns[i][0])[rowNums[i]] = field.getDoubleV();
+ break;
+ case INT64:
+ ((long[]) columns[i][0])[rowNums[i]] = field.getLongV();
+ break;
+ case INT32:
+ ((int[]) columns[i][0])[rowNums[i]] = field.getIntV();
+ break;
+ case FLOAT:
+ ((float[]) columns[i][0])[rowNums[i]] = field.getFloatV();
+ break;
+ default:
+ }
+
+ rowNums[i]++;
+ }
+ }
+ }
+
+ private List<PartialPath> getTargetPaths(List<Path> rawPaths) throws IllegalPathException {
+ List<PartialPath> targetPaths = new ArrayList<>(rawPaths.size());
+ for (Path rawPath : rawPaths) {
+ targetPaths.add(new PartialPath(fillTemplate((PartialPath) rawPath)));
+ }
+ return targetPaths;
+ }
+
+ private String fillTemplate(PartialPath rawPath) {
+ String[] nodes = rawPath.getNodes();
+ int indexOfLeftBracket = nodes[0].indexOf("(");
+ if (indexOfLeftBracket != -1) {
+ nodes[0] = nodes[0].substring(indexOfLeftBracket + 1);
+ }
+ int indexOfRightBracket = nodes[nodes.length - 1].indexOf(")");
+ if (indexOfRightBracket != -1) {
+ nodes[nodes.length - 1] = nodes[nodes.length - 1].substring(0, indexOfRightBracket);
+ }
+ StringBuffer sb = new StringBuffer();
+ Matcher m = Pattern.compile("\\$\\{\\w+}").matcher(this.plan.getTargetPath().getFullPath());
Review comment:
The return value of Pattern.compile("\\$\\{\\w+}") can be reused to improve performance. Make the return value of Pattern.compile("\\$\\{\\w+}") a static field.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
Review comment:
10?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
+ InsertTabletPlan[] insertTabletPlans = new InsertTabletPlan[columnSize];
+ String[] measurements = new String[] {targetPaths.get(0).getMeasurement()};
+ List<Integer> dataTypes = Collections.singletonList(dataType.ordinal());
+
+ for (int i = 0; i < columnSize; i++) {
+ insertTabletPlans[i] =
+ new InsertTabletPlan(
+ new PartialPath(targetPaths.get(i).getDevice()), measurements, dataTypes);
+ }
+
+ return insertTabletPlans;
+ }
+
+ private Object[][] getColumns(int columnSize, int fetchSize, TSDataType dataType) {
+ Object[][] columns = new Object[columnSize][1];
+ for (int i = 0; i < columnSize; i++) {
+ switch (dataType) {
+ case DOUBLE:
+ columns[i][0] = new double[fetchSize];
+ break;
+ case INT64:
+ columns[i][0] = new long[fetchSize];
+ break;
+ case INT32:
+ columns[i][0] = new int[fetchSize];
+ break;
+ case FLOAT:
+ columns[i][0] = new float[fetchSize];
+ break;
+ default:
+ break;
+ }
+ }
+ return columns;
+ }
+
+ private void fillColumns(
+ Object[][] columns,
+ TSDataType dataType,
+ RowRecord record,
+ int[] rowNums,
+ long[][] timestamps) {
+ List<Field> fields = record.getFields();
+ long ts = record.getTimestamp();
+
+ for (int i = 0; i < columns.length; i++) {
+ Field field = fields.get(i);
+ if (field != null) {
+ timestamps[i][rowNums[i]] = ts;
+ switch (dataType) {
+ case DOUBLE:
+ ((double[]) columns[i][0])[rowNums[i]] = field.getDoubleV();
+ break;
+ case INT64:
+ ((long[]) columns[i][0])[rowNums[i]] = field.getLongV();
+ break;
+ case INT32:
+ ((int[]) columns[i][0])[rowNums[i]] = field.getIntV();
+ break;
+ case FLOAT:
+ ((float[]) columns[i][0])[rowNums[i]] = field.getFloatV();
+ break;
+ default:
+ }
+
+ rowNums[i]++;
+ }
+ }
+ }
+
+ private List<PartialPath> getTargetPaths(List<Path> rawPaths) throws IllegalPathException {
+ List<PartialPath> targetPaths = new ArrayList<>(rawPaths.size());
+ for (Path rawPath : rawPaths) {
+ targetPaths.add(new PartialPath(fillTemplate((PartialPath) rawPath)));
+ }
+ return targetPaths;
+ }
+
+ private String fillTemplate(PartialPath rawPath) {
Review comment:
```suggestion
private String fillTargetPathTemplate(PartialPath rawPath) {
```
##########
File path: server/src/test/java/org/apache/iotdb/db/integration/IoTDBContinuousQueryIT.java
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.integration;
+
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.jdbc.Config;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.*;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Notice that, all test begins with "IoTDB" is integration test. All test which will start the
+ * IoTDB server should be defined as integration test.
+ */
+public class IoTDBContinuousQueryIT {
Review comment:
Add tests for wrong cases.
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
+ InsertTabletPlan[] insertTabletPlans = new InsertTabletPlan[columnSize];
+ String[] measurements = new String[] {targetPaths.get(0).getMeasurement()};
+ List<Integer> dataTypes = Collections.singletonList(dataType.ordinal());
+
+ for (int i = 0; i < columnSize; i++) {
+ insertTabletPlans[i] =
+ new InsertTabletPlan(
+ new PartialPath(targetPaths.get(i).getDevice()), measurements, dataTypes);
+ }
+
+ return insertTabletPlans;
+ }
+
+ private Object[][] getColumns(int columnSize, int fetchSize, TSDataType dataType) {
+ Object[][] columns = new Object[columnSize][1];
+ for (int i = 0; i < columnSize; i++) {
+ switch (dataType) {
+ case DOUBLE:
+ columns[i][0] = new double[fetchSize];
+ break;
+ case INT64:
+ columns[i][0] = new long[fetchSize];
+ break;
+ case INT32:
+ columns[i][0] = new int[fetchSize];
+ break;
+ case FLOAT:
+ columns[i][0] = new float[fetchSize];
+ break;
+ default:
+ break;
+ }
+ }
+ return columns;
+ }
+
+ private void fillColumns(
+ Object[][] columns,
+ TSDataType dataType,
+ RowRecord record,
+ int[] rowNums,
+ long[][] timestamps) {
+ List<Field> fields = record.getFields();
+ long ts = record.getTimestamp();
+
+ for (int i = 0; i < columns.length; i++) {
+ Field field = fields.get(i);
+ if (field != null) {
+ timestamps[i][rowNums[i]] = ts;
+ switch (dataType) {
+ case DOUBLE:
+ ((double[]) columns[i][0])[rowNums[i]] = field.getDoubleV();
+ break;
+ case INT64:
+ ((long[]) columns[i][0])[rowNums[i]] = field.getLongV();
+ break;
+ case INT32:
+ ((int[]) columns[i][0])[rowNums[i]] = field.getIntV();
+ break;
+ case FLOAT:
+ ((float[]) columns[i][0])[rowNums[i]] = field.getFloatV();
+ break;
+ default:
+ }
+
+ rowNums[i]++;
+ }
+ }
+ }
+
+ private List<PartialPath> getTargetPaths(List<Path> rawPaths) throws IllegalPathException {
Review comment:
```suggestion
private List<PartialPath> generateTargetPaths(List<Path> rawPaths) throws IllegalPathException {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
Review comment:
Can the return value be reused?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
Review comment:
```suggestion
private InsertTabletPlan[] generateInsertTabletPlans(
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
Review comment:
It's unnecessary for JAVA
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
Review comment:
It can be a bigger number for better performance :D
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
+ InsertTabletPlan[] insertTabletPlans = new InsertTabletPlan[columnSize];
+ String[] measurements = new String[] {targetPaths.get(0).getMeasurement()};
+ List<Integer> dataTypes = Collections.singletonList(dataType.ordinal());
+
+ for (int i = 0; i < columnSize; i++) {
+ insertTabletPlans[i] =
+ new InsertTabletPlan(
+ new PartialPath(targetPaths.get(i).getDevice()), measurements, dataTypes);
+ }
+
+ return insertTabletPlans;
+ }
+
+ private Object[][] getColumns(int columnSize, int fetchSize, TSDataType dataType) {
+ Object[][] columns = new Object[columnSize][1];
+ for (int i = 0; i < columnSize; i++) {
+ switch (dataType) {
+ case DOUBLE:
+ columns[i][0] = new double[fetchSize];
+ break;
+ case INT64:
+ columns[i][0] = new long[fetchSize];
+ break;
+ case INT32:
+ columns[i][0] = new int[fetchSize];
+ break;
+ case FLOAT:
+ columns[i][0] = new float[fetchSize];
+ break;
+ default:
+ break;
+ }
+ }
+ return columns;
+ }
+
+ private void fillColumns(
+ Object[][] columns,
+ TSDataType dataType,
+ RowRecord record,
+ int[] rowNums,
+ long[][] timestamps) {
+ List<Field> fields = record.getFields();
+ long ts = record.getTimestamp();
+
+ for (int i = 0; i < columns.length; i++) {
+ Field field = fields.get(i);
+ if (field != null) {
+ timestamps[i][rowNums[i]] = ts;
+ switch (dataType) {
+ case DOUBLE:
+ ((double[]) columns[i][0])[rowNums[i]] = field.getDoubleV();
+ break;
+ case INT64:
+ ((long[]) columns[i][0])[rowNums[i]] = field.getLongV();
+ break;
+ case INT32:
+ ((int[]) columns[i][0])[rowNums[i]] = field.getIntV();
+ break;
+ case FLOAT:
+ ((float[]) columns[i][0])[rowNums[i]] = field.getFloatV();
+ break;
+ default:
+ }
+
+ rowNums[i]++;
+ }
+ }
+ }
+
+ private List<PartialPath> getTargetPaths(List<Path> rawPaths) throws IllegalPathException {
+ List<PartialPath> targetPaths = new ArrayList<>(rawPaths.size());
+ for (Path rawPath : rawPaths) {
+ targetPaths.add(new PartialPath(fillTemplate((PartialPath) rawPath)));
+ }
+ return targetPaths;
+ }
+
+ private String fillTemplate(PartialPath rawPath) {
Review comment:
We'd better add some uts for this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-867787710
Dont forget to add some descriptions for the "cq" alias in the user doc. :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r657569350
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/utils/DatetimeUtils.java
##########
@@ -621,6 +622,16 @@ public static long convertDurationStrToLong(
}
}
+ public static TimeUnit toTimeUnit(String timestampPrecision) {
Review comment:
timestamp 的 精度 只有 ms us 和 ns？见 上一个方法 convertDurationStrToLong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r657586947
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java
##########
@@ -272,6 +283,9 @@
private static final String DELETE_RANGE_ERROR_MSG =
"For delete statement, where clause can only contain atomic expressions like : "
+ "time > XXX, time <= XXX, or two atomic expressions connected by 'AND'";
+
+ private static final Pattern pattern = Pattern.compile("\\$\\{\\w+}");
Review comment:
pattern?
##########
File path: server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByTimeDataSet.java
##########
@@ -80,8 +82,13 @@ public GroupByTimeDataSet(
this.dataTypes = new ArrayList<>();
this.paths = new ArrayList<>();
- for (int i = 0; i < finalPaths.size(); i++) {
- this.dataTypes.add(TSDataType.INT64);
+ for (Map.Entry<String, AggregateResult> entry : finalPaths.entrySet()) {
+ try {
+ this.paths.add(new PartialPath(entry.getKey()));
+ } catch (IllegalPathException e) {
+ logger.error("Query result IllegalPathException occurred.");
Review comment:
```suggestion
logger.error("Query result IllegalPathException occurred: {}.", entry.getKey());
```
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,206 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <full_path> | <node_name>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
+* `<function>` 指定聚合函数，目前支持 `count`, `sum`, `avg`, `last_value`, `first_value`, `min_time`, `max_time`, `min_value`, `max_value` 等。
+* `<path_prefix>` 与 `<path_suffix>` 拼接成完整的查询原时间序列。
+* `<full_path>` 或 `<node_name>` 指定将查询出的数据写入的结果序列路径。
+* `<group_by_interval>` 指定时间分组长度。
+* `<level>`指按照序列第 `<level>` 层分组，将第 `<level>` 层以下的所有序列聚合。
+
+
+注：
+
+* `<for_interval>`,`<every_interval>` 可选择指定。如果用户没有指定其中的某一项，则未指定项的值按照`<group_by_interval>` 处理。
+ * `<every_interval>`，`<for_interval>`，`<group_by_interval>` 的值均应大于 0。
+ * `<group_by_interval>` 的值应小于`<for_interval>`的值，否则系统会按照等于`<for_interval>`的值处理。
+ * 用户应当结合实际需求指定合适的 `<for_interval>` 与 `<every_interval>`。
+ * 若 `<for_interval>` 大于 `<every_interval>`，每次的查询窗口会有部分数据重叠，从查询性能角度这种配置不被建议。
+ * 若 `<for_interval>` 小于 `<every_interval>`，每次的查询窗口之间可能会有未覆盖到的数据。
+* 对于结果序列路径
+ * 用户可以选择指定`<full_path>`，即以 `root` 开头的完整的时间序列路径，用户可以在路径中使用 `${x}` 变量来表示原始时间序列中 `level = x` 的节点名称，`x`应当大于等于 1 且小于等于 `<level>` 值
+ （若未指定 `level`，则应小于等于 `<path_prefix>` 长度）。
+ * 用户也可以仅指定`<node_name>`，即生成时间序列路径的最后一个结点名。
+ * 若用户指定 `<level> = l`，则系统生成的结果时间序列路径为 `root.${1}. ... .${l}.<node_name>`
+ * 若用户未指定 `<level>`，令原始时间序列最大层数为 `L`，
+ 则系统生成的结果时间序列路径为 `root.${1}. ... .${L - 1}.<node_name>`。
+
Review comment:
level ..
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+ pool = IoTDBThreadPoolFactory.newScheduledThreadPool(10, "Continuous Query Service");
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ @Override
+ public void waitAndStop(long milliseconds) {
+ for (ScheduledFuture<?> future : continuousQueriesFutures.values()) {
+ future.cancel(false);
+ }
+ logger.info("Waiting for task pool to shut down");
+ try {
+ Thread.sleep(milliseconds);
+ } catch (InterruptedException e) {
+ logger.info("Thread interrupted");
+ Thread.currentThread().interrupt();
+ }
+ if (pool != null) {
+ pool.shutdownNow();
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
Review comment:
try .. finally
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,206 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <full_path> | <node_name>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
Review comment:
units?
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,206 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <full_path> | <node_name>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
+* `<function>` 指定聚合函数，目前支持 `count`, `sum`, `avg`, `last_value`, `first_value`, `min_time`, `max_time`, `min_value`, `max_value` 等。
+* `<path_prefix>` 与 `<path_suffix>` 拼接成完整的查询原时间序列。
+* `<full_path>` 或 `<node_name>` 指定将查询出的数据写入的结果序列路径。
+* `<group_by_interval>` 指定时间分组长度。
+* `<level>`指按照序列第 `<level>` 层分组，将第 `<level>` 层以下的所有序列聚合。
+
+
+注：
+
+* `<for_interval>`,`<every_interval>` 可选择指定。如果用户没有指定其中的某一项，则未指定项的值按照`<group_by_interval>` 处理。
+ * `<every_interval>`，`<for_interval>`，`<group_by_interval>` 的值均应大于 0。
+ * `<group_by_interval>` 的值应小于`<for_interval>`的值，否则系统会按照等于`<for_interval>`的值处理。
+ * 用户应当结合实际需求指定合适的 `<for_interval>` 与 `<every_interval>`。
+ * 若 `<for_interval>` 大于 `<every_interval>`，每次的查询窗口会有部分数据重叠，从查询性能角度这种配置不被建议。
+ * 若 `<for_interval>` 小于 `<every_interval>`，每次的查询窗口之间可能会有未覆盖到的数据。
+* 对于结果序列路径
+ * 用户可以选择指定`<full_path>`，即以 `root` 开头的完整的时间序列路径，用户可以在路径中使用 `${x}` 变量来表示原始时间序列中 `level = x` 的节点名称，`x`应当大于等于 1 且小于等于 `<level>` 值
+ （若未指定 `level`，则应小于等于 `<path_prefix>` 长度）。
+ * 用户也可以仅指定`<node_name>`，即生成时间序列路径的最后一个结点名。
+ * 若用户指定 `<level> = l`，则系统生成的结果时间序列路径为 `root.${1}. ... .${l}.<node_name>`
+ * 若用户未指定 `<level>`，令原始时间序列最大层数为 `L`，
+ 则系统生成的结果时间序列路径为 `root.${1}. ... .${L - 1}.<node_name>`。
+
+### 例子
+
+#### 原始时间序列
+````
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+
+````
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+| Time|root.ln.wf02.wt02.temperature|root.ln.wf02.wt01.temperature|root.ln.wf01.wt02.temperature|root.ln.wf01.wt01.temperature|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+|2021-05-11T22:18:14.598+08:00| 121.0| 72.0| 183.0| 115.0|
+|2021-05-11T22:18:19.941+08:00| 0.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:24.949+08:00| 122.0| 45.0| 11.0| 14.0|
+|2021-05-11T22:18:29.967+08:00| 47.0| 14.0| 59.0| 181.0|
+|2021-05-11T22:18:34.979+08:00| 182.0| 113.0| 29.0| 180.0|
+|2021-05-11T22:18:39.990+08:00| 42.0| 11.0| 52.0| 19.0|
+|2021-05-11T22:18:44.995+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:49.999+08:00| 137.0| 172.0| 135.0| 193.0|
+|2021-05-11T22:18:55.003+08:00| 16.0| 124.0| 183.0| 18.0|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+````
Review comment:
#### into ...
用户也可以指定写入序列为 `root.${1}_cq.${2}.temperature_avg`、`root.${1}.${2}_cq.temperature_avg`，效果与上例相同。
用户也可以按需要指定写入序列为 `root.${2}.${1}.temperature_avg` 等。
需要注意的是，`${x}` 中的 `x` 应当大于等于 1 且小于等于 `<level>` 值
（若未指定 `<level>`，则应小于等于 `<path_prefix>` 层级）。在上例中，`x` 应当小于等于 `2`。
##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
##########
@@ -530,6 +530,11 @@
*/
private int compactionThreadNum = 10;
+ /*
+ * How many thread will be set up to perform continuous queries
+ */
+ private int continuousQueryThreadNum = 10;
Review comment:
iotdb-engine.properties
cq.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-865752485
[](https://coveralls.io/builds/40773170)
Coverage increased (+0.001%) to 67.888% when pulling **776f145c27da9492ba045d5bd6ee21c3a09050e0 on mzp0514:cq** into **d8b4ce8e6fbc4e25a55c7244a9e86592b8487002 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656085091
##########
File path: server/src/main/java/org/apache/iotdb/db/auth/AuthorityChecker.java
##########
@@ -157,6 +156,10 @@ private static int translateToPermissionId(Operator.OperatorType type) {
return PrivilegeType.START_TRIGGER.ordinal();
case STOP_TRIGGER:
return PrivilegeType.STOP_TRIGGER.ordinal();
+ case CREATE_CONTINUOUS_QUERY:
Review comment:
Administration.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu merged pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu merged pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-869664710
Performance report: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181311232
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r659407471
##########
File path: server/src/assembly/resources/conf/iotdb-engine.properties
##########
@@ -797,6 +797,24 @@ timestamp_precision=ms
# Datatype: int
# max_pending_window_evaluation_tasks=64
+####################
+### Continuous Query Configuration
+####################
+
+# How many thread will be set up to perform continuous queries. When <= 0, use max(1, CPU core number / 2).
+# Datatype: int
+# continuous_query_execution_thread=2
+
+# Maximum number of continuous query tasks that can be pending for execution. When <= 0, the value is
+# 64 by default.
+# Datatype: int
+# max_pending_continuous_query_tasks=64
Review comment:
Should it be larger?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryTask.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.WrappedRunnable;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQueryTask extends WrappedRunnable {
+
+ private static final int FETCH_SIZE = 2048;
+ private static final int BATCH_SIZE = 10;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryTask.class);
+
+ // To execute the query plan
+ private static PlanExecutor planExecutor;
+
+ static {
+ try {
+ planExecutor = new PlanExecutor();
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ // To save the continuous query info
+ private final CreateContinuousQueryPlan plan;
+ // To transform query operator to query plan
+ private static final Planner planner = new Planner();
+ // Next timestamp to execute a query
+ private long executionTimestamp;
+
+ private static final Pattern pattern = Pattern.compile("\\$\\{\\w+}");
+
+ public ContinuousQueryTask(CreateContinuousQueryPlan plan, long timestamp) {
+ this.plan = plan;
+ this.executionTimestamp = timestamp;
+ }
+
+ @Override
+ public void runMayThrow() {
+
+ try {
+ GroupByTimePlan queryPlan = generateQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.info(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null || result.getPaths().size() == 0) {
+ logger.info(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
Review comment:
No need to catch exceptions here. The super method can catch all exceptions.
##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
##########
@@ -487,6 +488,40 @@ private void loadProps() {
Integer.parseInt(
properties.getProperty(
"compaction_thread_num", Integer.toString(conf.getCompactionThreadNum()))));
+
+ conf.setContinuousQueryThreadNum(
+ Integer.parseInt(
+ properties.getProperty(
+ "continuous_query_thread_num",
+ Integer.toString(conf.getContinuousQueryThreadNum()))));
+
+ if (conf.getContinuousQueryThreadNum() <= 0) {
+ conf.setContinuousQueryThreadNum(Runtime.getRuntime().availableProcessors() / 2);
+ }
+
+ conf.setContinuousQueryThreadNum(
+ Integer.parseInt(
+ properties.getProperty(
+ "continuous_query_thread_num",
+ Integer.toString(conf.getContinuousQueryThreadNum()))));
+ if (conf.getContinuousQueryThreadNum() <= 0) {
+ conf.setContinuousQueryThreadNum(Runtime.getRuntime().availableProcessors() / 2);
+ }
+
Review comment:
```suggestion
```
##########
File path: server/src/assembly/resources/conf/iotdb-engine.properties
##########
@@ -797,6 +797,24 @@ timestamp_precision=ms
# Datatype: int
# max_pending_window_evaluation_tasks=64
+####################
+### Continuous Query Configuration
+####################
+
+# How many thread will be set up to perform continuous queries. When <= 0, use max(1, CPU core number / 2).
+# Datatype: int
+# continuous_query_execution_thread=2
+
+# Maximum number of continuous query tasks that can be pending for execution. When <= 0, the value is
+# 64 by default.
+# Datatype: int
+# max_pending_continuous_query_tasks=64
+
+# Minimum every interval to perform continuous query.
+# The every interval of continuous query instances should not be lower than this limit.
+# Datatype: duration
+# continuous_query_min_every_interval=1s
Review comment:
What if users create a cq whose `every interval` is less than 1s?
##########
File path: server/src/main/java/org/apache/iotdb/db/concurrent/ThreadName.java
##########
@@ -45,7 +45,8 @@
TIME_COST_STATISTIC("TIME_COST_STATISTIC"),
QUERY_SERVICE("Query"),
WINDOW_EVALUATION_SERVICE("WindowEvaluationTaskPoolManager"),
- CLUSTER_INFO_SERVICE("ClusterInfoClient");
+ CLUSTER_INFO_SERVICE("ClusterInfoClient"),
+ CONTINUOUS_QUERY_SERVICE("ContinuousQueryTaskPoolManager");
Review comment:
```suggestion
CONTINUOUS_QUERY_SERVICE("ContinuousQueryTaskPoolManager");
CLUSTER_INFO_SERVICE("ClusterInfoClient"),
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryTask.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.WrappedRunnable;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQueryTask extends WrappedRunnable {
+
+ private static final int FETCH_SIZE = 2048;
+ private static final int BATCH_SIZE = 10;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryTask.class);
+
+ // To execute the query plan
+ private static PlanExecutor planExecutor;
+
+ static {
+ try {
+ planExecutor = new PlanExecutor();
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ // To save the continuous query info
+ private final CreateContinuousQueryPlan plan;
+ // To transform query operator to query plan
+ private static final Planner planner = new Planner();
+ // Next timestamp to execute a query
+ private long executionTimestamp;
Review comment:
Rename to executionEndTimestamp?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryTask.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.WrappedRunnable;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQueryTask extends WrappedRunnable {
+
+ private static final int FETCH_SIZE = 2048;
+ private static final int BATCH_SIZE = 10;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryTask.class);
+
+ // To execute the query plan
+ private static PlanExecutor planExecutor;
+
+ static {
+ try {
+ planExecutor = new PlanExecutor();
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ // To save the continuous query info
+ private final CreateContinuousQueryPlan plan;
+ // To transform query operator to query plan
+ private static final Planner planner = new Planner();
+ // Next timestamp to execute a query
+ private long executionTimestamp;
+
+ private static final Pattern pattern = Pattern.compile("\\$\\{\\w+}");
+
+ public ContinuousQueryTask(CreateContinuousQueryPlan plan, long timestamp) {
Review comment:
```suggestion
public ContinuousQueryTask(CreateContinuousQueryPlan plan, long executionEndTimestamp) {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryTask.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.WrappedRunnable;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQueryTask extends WrappedRunnable {
+
+ private static final int FETCH_SIZE = 2048;
+ private static final int BATCH_SIZE = 10;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryTask.class);
+
+ // To execute the query plan
+ private static PlanExecutor planExecutor;
+
+ static {
+ try {
+ planExecutor = new PlanExecutor();
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ // To save the continuous query info
+ private final CreateContinuousQueryPlan plan;
+ // To transform query operator to query plan
+ private static final Planner planner = new Planner();
+ // Next timestamp to execute a query
+ private long executionTimestamp;
+
+ private static final Pattern pattern = Pattern.compile("\\$\\{\\w+}");
+
+ public ContinuousQueryTask(CreateContinuousQueryPlan plan, long timestamp) {
+ this.plan = plan;
+ this.executionTimestamp = timestamp;
+ }
+
+ @Override
+ public void runMayThrow() {
+
+ try {
+ GroupByTimePlan queryPlan = generateQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.info(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null || result.getPaths().size() == 0) {
+ logger.info(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ public void onRejection() {
+ logger.info("Continuous Query Task {} rejected", plan.getContinuousQueryName());
Review comment:
```suggestion
logger.warn("Continuous Query Task {} rejected", plan.getContinuousQueryName());
```
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryTask.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.WrappedRunnable;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQueryTask extends WrappedRunnable {
+
+ private static final int FETCH_SIZE = 2048;
+ private static final int BATCH_SIZE = 10;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryTask.class);
+
+ // To execute the query plan
+ private static PlanExecutor planExecutor;
+
+ static {
+ try {
+ planExecutor = new PlanExecutor();
+ } catch (QueryProcessException e) {
+ logger.error(e.getMessage());
+ }
+ }
+
+ // To save the continuous query info
+ private final CreateContinuousQueryPlan plan;
+ // To transform query operator to query plan
+ private static final Planner planner = new Planner();
+ // Next timestamp to execute a query
+ private long executionTimestamp;
+
+ private static final Pattern pattern = Pattern.compile("\\$\\{\\w+}");
+
+ public ContinuousQueryTask(CreateContinuousQueryPlan plan, long timestamp) {
+ this.plan = plan;
+ this.executionTimestamp = timestamp;
+ }
+
+ @Override
+ public void runMayThrow() {
+
+ try {
+ GroupByTimePlan queryPlan = generateQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.info(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null || result.getPaths().size() == 0) {
+ logger.info(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ public void onRejection() {
+ logger.info("Continuous Query Task {} rejected", plan.getContinuousQueryName());
+ }
+
+ private GroupByTimePlan generateQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ // To handle the time series meta changes in different queries, i.e. creation & deletion,
+ // we need to apply concatenation optimization to SelectComponent before every query.
+ // Since the concatenation optimization will change resultColumns information of
+ // SelectComponent,
+ // we need to save one copy of the original SelectComponent.
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.cqQueryOperatorToGroupByTimePlan(queryOperator, FETCH_SIZE);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ queryPlan.setStartTime(executionTimestamp - plan.getForInterval());
+ queryPlan.setEndTime(executionTimestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, FETCH_SIZE, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
Review comment:
```suggestion
try {
return planExecutor.processQuery(queryPlan, new QueryContext(queryId));
} finally {
QueryResourceManager.getInstance().endQuery(queryId);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r659407087
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,206 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <full_path> | <node_name>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
+* `<function>` 指定聚合函数，目前支持 `count`, `sum`, `avg`, `last_value`, `first_value`, `min_time`, `max_time`, `min_value`, `max_value` 等。
+* `<path_prefix>` 与 `<path_suffix>` 拼接成完整的查询原时间序列。
+* `<full_path>` 或 `<node_name>` 指定将查询出的数据写入的结果序列路径。
+* `<group_by_interval>` 指定时间分组长度。
+* `<level>`指按照序列第 `<level>` 层分组，将第 `<level>` 层以下的所有序列聚合。
+
+
+注：
+
+* `<for_interval>`,`<every_interval>` 可选择指定。如果用户没有指定其中的某一项，则未指定项的值按照`<group_by_interval>` 处理。
+ * `<every_interval>`，`<for_interval>`，`<group_by_interval>` 的值均应大于 0。
+ * `<group_by_interval>` 的值应小于`<for_interval>`的值，否则系统会按照等于`<for_interval>`的值处理。
+ * 用户应当结合实际需求指定合适的 `<for_interval>` 与 `<every_interval>`。
+ * 若 `<for_interval>` 大于 `<every_interval>`，每次的查询窗口会有部分数据重叠，从查询性能角度这种配置不被建议。
+ * 若 `<for_interval>` 小于 `<every_interval>`，每次的查询窗口之间可能会有未覆盖到的数据。
+* 对于结果序列路径
+ * 用户可以选择指定`<full_path>`，即以 `root` 开头的完整的时间序列路径，用户可以在路径中使用 `${x}` 变量来表示原始时间序列中 `level = x` 的节点名称，`x`应当大于等于 1 且小于等于 `<level>` 值
+ （若未指定 `level`，则应小于等于 `<path_prefix>` 长度）。
+ * 用户也可以仅指定`<node_name>`，即生成时间序列路径的最后一个结点名。
+ * 若用户指定 `<level> = l`，则系统生成的结果时间序列路径为 `root.${1}. ... .${l}.<node_name>`
+ * 若用户未指定 `<level>`，令原始时间序列最大层数为 `L`，
+ 则系统生成的结果时间序列路径为 `root.${1}. ... .${L - 1}.<node_name>`。
+
+### 例子
+
+#### 原始时间序列
+````
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+
+````
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+| Time|root.ln.wf02.wt02.temperature|root.ln.wf02.wt01.temperature|root.ln.wf01.wt02.temperature|root.ln.wf01.wt01.temperature|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+|2021-05-11T22:18:14.598+08:00| 121.0| 72.0| 183.0| 115.0|
+|2021-05-11T22:18:19.941+08:00| 0.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:24.949+08:00| 122.0| 45.0| 11.0| 14.0|
+|2021-05-11T22:18:29.967+08:00| 47.0| 14.0| 59.0| 181.0|
+|2021-05-11T22:18:34.979+08:00| 182.0| 113.0| 29.0| 180.0|
+|2021-05-11T22:18:39.990+08:00| 42.0| 11.0| 52.0| 19.0|
+|2021-05-11T22:18:44.995+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:49.999+08:00| 137.0| 172.0| 135.0| 193.0|
+|2021-05-11T22:18:55.003+08:00| 16.0| 124.0| 183.0| 18.0|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+````
+#### 创建 `cq1`
+````
+CREATE CONTINUOUS QUERY cq1 BEGIN SELECT max_value(temperature) INTO temperature_max FROM root.ln.*.* GROUP BY time(10s) END
+````
+
+每隔 10s 查询 `root.ln.*.*.temperature` 在前 10s 内的最大值（结果以10s为一组），
+将结果写入到 `root.${1}.${2}.${3}.temperature_max` 中，
+结果将产生4条新序列：
+````
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+| Time|root.ln.wf02.wt02.temperature_max|root.ln.wf02.wt01.temperature_max|root.ln.wf01.wt02.temperature_max|root.ln.wf01.wt01.temperature_max|
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+|2021-05-11T22:18:16.964+08:00| 122.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:26.964+08:00| 182.0| 113.0| 59.0| 181.0|
+|2021-05-11T22:18:36.964+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:46.964+08:00| 137.0| 172.0| 183.0| 193.0|
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+````
+#### 创建 `cq2`
+````
+CREATE CONTINUOUS QUERY cq2 RESAMPLE EVERY 20s FOR 20s BEGIN SELECT avg(temperature) INTO temperature_avg FROM root.ln.*.* GROUP BY time(10s), level=2 END
+````
+
+每隔 20s 查询 `root.ln.*.*.temperature` 在前 20s 内的平均值（结果以10s为一组，按照第2层节点分组），
+将结果写入到 `root.${1}.${2}.temperature_avg` 中。
+结果将产生如下两条新序列，
+其中 `root.ln.wf02.temperature_avg` 由 `root.ln.wf02.wt02.temperature` 和 `root.ln.wf02.wt01.temperature` 聚合计算生成，
+`root.ln.wf01.temperature_avg` 由 `root.ln.wf01.wt02.temperature` 和 `root.ln.wf01.wt01.temperature` 聚合计算生成。
+
+````
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.temperature_avg| null| root.ln| DOUBLE| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.temperature_avg| null| root.ln| DOUBLE| GORILLA| SNAPPY|null| null|
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+----------------------------+----------------------------+
+| Time|root.ln.wf02.temperature_avg|root.ln.wf01.temperature_avg|
++-----------------------------+----------------------------+----------------------------+
+|2021-05-11T22:18:16.969+08:00| 58.75| 49.0|
+|2021-05-11T22:18:26.969+08:00| 89.0| 112.25|
+|2021-05-11T22:18:36.969+08:00| 42.25| 61.5|
+|2021-05-11T22:18:46.969+08:00| 112.25| 132.25|
++-----------------------------+----------------------------+----------------------------+
+````
+#### 创建 `cq3`
+````
+CREATE CONTINUOUS QUERY cq3 RESAMPLE EVERY 20s FOR 20s BEGIN SELECT avg(temperature) INTO root.ln_cq.${2}.temperature_avg FROM root.ln.*.* GROUP BY time(10s), level=2 END
+````
+查询模式与 cq2 相同，在这个例子中，用户自行指定结果写入到 `root.ln_cq.${2}.temperature_avg` 中。
+结果将产生如下两条新序列，
+其中 `root.ln_cq.wf02.temperature_avg` 由 `root.ln.wf02.wt02.temperature` 和 `root.ln.wf02.wt01.temperature` 聚合计算生成，
+`root.ln_cq.wf01.temperature_avg` 由 `root.ln.wf01.wt02.temperature` 和 `root.ln.wf01.wt01.temperature` 聚合计算生成。
+````
++-------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++-------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln_cq.wf02.temperature_avg| null| root.ln_cq| DOUBLE| GORILLA| SNAPPY|null| null|
+|root.ln_cq.wf01.temperature_avg| null| root.ln_cq| DOUBLE| GORILLA| SNAPPY|null| null|
++-------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+-------------------------------+-------------------------------+
+| Time|root.ln_cq.wf02.temperature_avg|root.ln_cq.wf01.temperature_avg|
++-----------------------------+-------------------------------+-------------------------------+
+|2021-05-11T22:18:16.971+08:00| 58.75| 49.0|
+|2021-05-11T22:18:26.971+08:00| 89.0| 112.25|
+|2021-05-11T22:18:36.971+08:00| 42.25| 61.5|
+|2021-05-11T22:18:46.971+08:00| 112.25| 132.25|
++-----------------------------+-------------------------------+-------------------------------+
+````
+
+用户也可以指定写入序列为 `root.${1}_cq.${2}.temperature_avg`、`root.${1}.${2}_cq.temperature_avg`，效果与上例相同。
+用户也可以按需要指定写入序列为 `root.${2}.${1}.temperature_avg` 等。
+需要注意的是，`${x}` 中的 `x` 应当大于等于 1 且小于等于 `<level>` 值
+（若未指定 `<level>`，则应小于等于 `<path_prefix>` 层级）。在上例中，`x` 应当小于等于 `2`。
+
+### 展示 CQ 信息
+````
+SHOW CONTINUOUS QUERIES
+````
+````
++-------+--------------+------------+----------------------------------------------------------------------------------------+-----------------------------------+
+|cq name|every interval|for interval| query sql| target path|
++-------+--------------+------------+----------------------------------------------------------------------------------------+-----------------------------------+
+| cq1| 10000| 10000| select max_value(temperature) from root.ln.*.* group by ([now() - 10s, now()), 10s)|root.${1}.${2}.${3}.temperature_max|
+| cq3| 20000| 20000|select avg(temperature) from root.ln.*.* group by ([now() - 20s, now()), 10s), level = 2| root.ln_cq.${2}.temperature_avg|
+| cq2| 20000| 20000|select avg(temperature) from root.ln.*.* group by ([now() - 20s, now()), 10s), level = 2| root.${1}.${2}.temperature_avg|
++-------+--------------+------------+----------------------------------------------------------------------------------------+-----------------------------------+
+````
+### 删除 CQ
+````
+DROP CONTINUOUS QUERY <cq_id>
+````
+例如：
+````
+DROP CONTINUOUS QUERY cq3
+````
Review comment:
配置参数说明文档
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r657593681
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final HashMap<String, ScheduledFuture<?>> continuousQueriesFutures = new HashMap<>();
+
+ private final HashMap<String, CreateContinuousQueryPlan> continuousQueryPlans = new HashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService pool;
Review comment:
WindowEvaluationTaskPoolManager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r659423945
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.ShutdownException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static long CHECK_INTERVAL =
+ IoTDBDescriptor.getInstance().getConfig().getContinuousQueryMinimumEveryInterval() / 2;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final ConcurrentHashMap<String, CreateContinuousQueryPlan> continuousQueryPlans =
+ new ConcurrentHashMap<>();
+
+ private final ConcurrentHashMap<String, Long> nextExecutionTimestamps = new ConcurrentHashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService checkThread;
+
+ protected static final ContinuousQueryTaskPoolManager TASK_POOL_MANAGER =
+ ContinuousQueryTaskPoolManager.getInstance();
+
Review comment:
```suggestion
private ContinuousQueryService() {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r659447884
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.ShutdownException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static long CHECK_INTERVAL =
+ IoTDBDescriptor.getInstance().getConfig().getContinuousQueryMinimumEveryInterval() / 2;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final ConcurrentHashMap<String, CreateContinuousQueryPlan> continuousQueryPlans =
+ new ConcurrentHashMap<>();
+
+ private final ConcurrentHashMap<String, Long> nextExecutionTimestamps = new ConcurrentHashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService checkThread;
+
+ protected static final ContinuousQueryTaskPoolManager TASK_POOL_MANAGER =
+ ContinuousQueryTaskPoolManager.getInstance();
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long durationFromCreation = System.currentTimeMillis() - plan.getCreationTimestamp();
+ long nextExecutionTimestamp =
+ plan.getCreationTimestamp()
+ + plan.getEveryInterval()
+ * (durationFromCreation / plan.getEveryInterval()
+ + ((durationFromCreation % plan.getEveryInterval() == 0) ? 0 : 1));
+ nextExecutionTimestamps.put(plan.getContinuousQueryName(), nextExecutionTimestamp);
+ }
+
+ checkThread = Executors.newSingleThreadScheduledExecutor();
+ checkThread.scheduleAtFixedRate(
+ this::checkAndSubmitTasks,
+ 0,
+ CHECK_INTERVAL,
+ DatetimeUtils.timestampPrecisionStringToTimeUnit(
+ IoTDBDescriptor.getInstance().getConfig().getTimestampPrecision()));
+
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (checkThread != null) {
+ checkThread.shutdown();
+ try {
+ checkThread.awaitTermination(600, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ logger.warn("Check thread still doesn't exit after 60s");
+ checkThread.shutdownNow();
+ }
+ }
+ }
+
+ @Override
+ public void shutdown(long milliseconds) throws ShutdownException {
+ stop();
+ }
+
+ private void checkAndSubmitTasks() {
+ long currentTimestamp = System.currentTimeMillis();
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long nextExecutionTimestamp = nextExecutionTimestamps.get(plan.getContinuousQueryName());
+ if (currentTimestamp >= nextExecutionTimestamp) {
Review comment:
```suggestion
while (currentTimestamp >= nextExecutionTimestamp) {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java
##########
@@ -1086,6 +1100,174 @@ private ResultColumn parseResultColumn(ResultColumnContext resultColumnContext)
resultColumnContext.AS() == null ? null : resultColumnContext.ID().getText());
}
+ @Override
+ public Operator visitDropContinuousQueryStatement(DropContinuousQueryStatementContext ctx) {
+ DropContinuousQueryOperator dropContinuousQueryOperator =
+ new DropContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_DROP);
+ dropContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+ return dropContinuousQueryOperator;
+ }
+
+ @Override
+ public Operator visitShowContinuousQueriesStatement(ShowContinuousQueriesStatementContext ctx) {
+ return new ShowContinuousQueriesOperator(SQLConstant.TOK_SHOW_CONTINUOUS_QUERIES);
+ }
+
+ @Override
+ public Operator visitCreateContinuousQueryStatement(CreateContinuousQueryStatementContext ctx) {
+ CreateContinuousQueryOperator createContinuousQueryOperator =
+ new CreateContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_CREATE);
+
+ createContinuousQueryOperator.setQuerySql(ctx.getText());
+
+ createContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+
+ if (ctx.resampleClause() != null) {
+ parseResampleClause(ctx.resampleClause(), createContinuousQueryOperator);
+ }
+
+ parseCqSelectIntoClause(ctx.cqSelectIntoClause(), createContinuousQueryOperator);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("select ");
+ sb.append(ctx.cqSelectIntoClause().selectClause().getText().substring(6));
+ sb.append(" from ");
+ sb.append(ctx.cqSelectIntoClause().fromClause().prefixPath(0).getText());
+
+ sb.append(" group by ([now() - ");
+ String groupByInterval = ctx.cqSelectIntoClause().cqGroupByTimeClause().DURATION().getText();
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ sb.append(groupByInterval);
+ } else {
+ List<TerminalNode> durations = ctx.resampleClause().DURATION();
+ sb.append(durations.get(durations.size() - 1).getText());
+ }
+ sb.append(", now()), ");
+ sb.append(groupByInterval);
+ sb.append(")");
+ if (queryOp.isGroupByLevel()) {
+ sb.append(", level = ");
+ sb.append(queryOp.getSpecialClauseComponent().getLevel());
+ }
+ createContinuousQueryOperator.setQuerySql(sb.toString());
+
+ if (createContinuousQueryOperator.getEveryInterval() == 0) {
+ createContinuousQueryOperator.setEveryInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ if (createContinuousQueryOperator.getEveryInterval()
Review comment:
?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.ShutdownException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static long CHECK_INTERVAL =
+ IoTDBDescriptor.getInstance().getConfig().getContinuousQueryMinimumEveryInterval() / 2;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final ConcurrentHashMap<String, CreateContinuousQueryPlan> continuousQueryPlans =
+ new ConcurrentHashMap<>();
+
+ private final ConcurrentHashMap<String, Long> nextExecutionTimestamps = new ConcurrentHashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService checkThread;
+
+ protected static final ContinuousQueryTaskPoolManager TASK_POOL_MANAGER =
+ ContinuousQueryTaskPoolManager.getInstance();
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long durationFromCreation = System.currentTimeMillis() - plan.getCreationTimestamp();
+ long nextExecutionTimestamp =
+ plan.getCreationTimestamp()
+ + plan.getEveryInterval()
+ * (durationFromCreation / plan.getEveryInterval()
+ + ((durationFromCreation % plan.getEveryInterval() == 0) ? 0 : 1));
+ nextExecutionTimestamps.put(plan.getContinuousQueryName(), nextExecutionTimestamp);
+ }
+
+ checkThread = Executors.newSingleThreadScheduledExecutor();
+ checkThread.scheduleAtFixedRate(
+ this::checkAndSubmitTasks,
+ 0,
+ CHECK_INTERVAL,
+ DatetimeUtils.timestampPrecisionStringToTimeUnit(
+ IoTDBDescriptor.getInstance().getConfig().getTimestampPrecision()));
+
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (checkThread != null) {
+ checkThread.shutdown();
+ try {
+ checkThread.awaitTermination(600, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ logger.warn("Check thread still doesn't exit after 60s");
+ checkThread.shutdownNow();
+ }
+ }
+ }
+
+ @Override
+ public void shutdown(long milliseconds) throws ShutdownException {
+ stop();
+ }
+
+ private void checkAndSubmitTasks() {
+ long currentTimestamp = System.currentTimeMillis();
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long nextExecutionTimestamp = nextExecutionTimestamps.get(plan.getContinuousQueryName());
+ if (currentTimestamp >= nextExecutionTimestamp) {
+ TASK_POOL_MANAGER.submit(new ContinuousQueryTask(plan, nextExecutionTimestamp));
+ nextExecutionTimestamps.replace(
+ plan.getContinuousQueryName(), nextExecutionTimestamp + plan.getEveryInterval());
+ }
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
+ acquireRegistrationLock();
+
+ if (continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ releaseRegistrationLock();
+ throw new ContinuousQueryAlreadyExistException(plan.getContinuousQueryName());
+ }
+
+ try {
+ if (writeLog) {
+ IoTDB.metaManager.createContinuousQuery(plan);
+ }
+ doRegister(plan);
+
+ } catch (Exception e) {
+ logger.error(e.getMessage());
Review comment:
throw it !
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryTask.java
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.concurrent.WrappedRunnable;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQueryTask extends WrappedRunnable {
+
+ private static final int FETCH_SIZE = 2048;
+ private static final int BATCH_SIZE = 10;
Review comment:
...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656861736
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/sql/IoTDBSqlVisitor.java
##########
@@ -1086,6 +1095,166 @@ private ResultColumn parseResultColumn(ResultColumnContext resultColumnContext)
resultColumnContext.AS() == null ? null : resultColumnContext.ID().getText());
}
+ @Override
+ public Operator visitDropContinuousQueryStatement(DropContinuousQueryStatementContext ctx) {
+ DropContinuousQueryOperator dropContinuousQueryOperator =
+ new DropContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_DROP);
+ dropContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+ return dropContinuousQueryOperator;
+ }
+
+ @Override
+ public Operator visitShowContinuousQueriesStatement(ShowContinuousQueriesStatementContext ctx) {
+ return new ShowContinuousQueriesOperator(SQLConstant.TOK_SHOW_CONTINUOUS_QUERIES);
+ }
+
+ @Override
+ public Operator visitCreateContinuousQueryStatement(CreateContinuousQueryStatementContext ctx) {
+ CreateContinuousQueryOperator createContinuousQueryOperator =
+ new CreateContinuousQueryOperator(SQLConstant.TOK_CONTINUOUS_QUERY_CREATE);
+
+ createContinuousQueryOperator.setQuerySql(ctx.getText());
+
+ createContinuousQueryOperator.setContinuousQueryName(ctx.continuousQueryName.getText());
+
+ if (ctx.resampleClause() != null) {
+ parseResampleClause(ctx.resampleClause(), createContinuousQueryOperator);
+ }
+
+ parseCqSelectIntoClause(ctx.cqSelectIntoClause(), createContinuousQueryOperator);
+
+ StringBuilder sb = new StringBuilder();
+ sb.append("select ");
+ sb.append(ctx.cqSelectIntoClause().selectClause().getText().substring(6));
+ sb.append(" from ");
+ sb.append(ctx.cqSelectIntoClause().fromClause().prefixPath(0).getText());
+
+ sb.append(" group by ([now() - ");
+ String groupByInterval = ctx.cqSelectIntoClause().cqGroupByTimeClause().DURATION().getText();
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ sb.append(groupByInterval);
+ } else {
+ List<TerminalNode> durations = ctx.resampleClause().DURATION();
+ sb.append(durations.get(durations.size() - 1).getText());
+ }
+ sb.append(", now()), ");
+ sb.append(groupByInterval);
+ sb.append(")");
+ if (queryOp.isGroupByLevel()) {
+ sb.append(", level = ");
+ sb.append(queryOp.getSpecialClauseComponent().getLevel());
+ }
+ createContinuousQueryOperator.setQuerySql(sb.toString());
+
+ if (createContinuousQueryOperator.getEveryInterval() == 0) {
+ createContinuousQueryOperator.setEveryInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ if (createContinuousQueryOperator.getForInterval() == 0) {
+ createContinuousQueryOperator.setForInterval(
+ ((GroupByClauseComponent) queryOp.getSpecialClauseComponent()).getUnit());
+ }
+
+ return createContinuousQueryOperator;
+ }
+
+ public void parseResampleClause(
+ ResampleClauseContext ctx, CreateContinuousQueryOperator operator) {
+
+ if (ctx.DURATION().size() == 1) {
+ if (ctx.EVERY() != null) {
+ operator.setEveryInterval(
+ DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ } else if (ctx.FOR() != null) {
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ }
+ } else if (ctx.DURATION().size() == 2) {
+ operator.setEveryInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(0).getText()));
+ operator.setForInterval(DatetimeUtils.convertDurationStrToLong(ctx.DURATION(1).getText()));
+ }
+ }
+
+ public void parseCqSelectIntoClause(
+ CqSelectIntoClauseContext ctx, CreateContinuousQueryOperator createContinuousQueryOperator) {
+
+ queryOp = new GroupByQueryOperator();
+
+ parseSelectClause(ctx.selectClause());
+ parseFromClause(ctx.fromClause());
+
+ if (queryOp.getSelectComponent().getResultColumns().size() > 1) {
+ throw new SQLParserException(
+ "Cq Select Into: CQ currently does not support multiple result columns.");
Review comment:
has been tested in PhysicalPlanTest.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r659418063
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQueryService.java
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.ContinuousQueryAlreadyExistException;
+import org.apache.iotdb.db.exception.ContinuousQueryNotExistException;
+import org.apache.iotdb.db.exception.ShutdownException;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.qp.physical.sys.DropContinuousQueryPlan;
+import org.apache.iotdb.db.qp.utils.DatetimeUtils;
+import org.apache.iotdb.db.query.dataset.ShowContinuousQueriesResult;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.IoTDB;
+import org.apache.iotdb.db.service.ServiceType;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+
+public class ContinuousQueryService implements IService {
+
+ private static long CHECK_INTERVAL =
+ IoTDBDescriptor.getInstance().getConfig().getContinuousQueryMinimumEveryInterval() / 2;
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQueryService.class);
+
+ private final ConcurrentHashMap<String, CreateContinuousQueryPlan> continuousQueryPlans =
+ new ConcurrentHashMap<>();
+
+ private final ConcurrentHashMap<String, Long> nextExecutionTimestamps = new ConcurrentHashMap<>();
+
+ private final ReentrantLock registrationLock = new ReentrantLock();
+
+ private static final ContinuousQueryService INSTANCE = new ContinuousQueryService();
+
+ private ScheduledExecutorService checkThread;
+
+ protected static final ContinuousQueryTaskPoolManager TASK_POOL_MANAGER =
+ ContinuousQueryTaskPoolManager.getInstance();
+
+ public static ContinuousQueryService getInstance() {
+ return INSTANCE;
+ }
+
+ public void acquireRegistrationLock() {
+ registrationLock.lock();
+ }
+
+ public void releaseRegistrationLock() {
+ registrationLock.unlock();
+ }
+
+ @Override
+ public ServiceType getID() {
+ return ServiceType.CONTINUOUS_QUERY_SERVICE;
+ }
+
+ @Override
+ public void start() {
+
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long durationFromCreation = System.currentTimeMillis() - plan.getCreationTimestamp();
+ long nextExecutionTimestamp =
+ plan.getCreationTimestamp()
+ + plan.getEveryInterval()
+ * (durationFromCreation / plan.getEveryInterval()
+ + ((durationFromCreation % plan.getEveryInterval() == 0) ? 0 : 1));
+ nextExecutionTimestamps.put(plan.getContinuousQueryName(), nextExecutionTimestamp);
+ }
+
+ checkThread = Executors.newSingleThreadScheduledExecutor();
+ checkThread.scheduleAtFixedRate(
+ this::checkAndSubmitTasks,
+ 0,
+ CHECK_INTERVAL,
+ DatetimeUtils.timestampPrecisionStringToTimeUnit(
+ IoTDBDescriptor.getInstance().getConfig().getTimestampPrecision()));
+
+ logger.info("Continuous query service started.");
+ }
+
+ @Override
+ public void stop() {
+ if (checkThread != null) {
+ checkThread.shutdown();
+ try {
+ checkThread.awaitTermination(600, TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ logger.warn("Check thread still doesn't exit after 60s");
+ checkThread.shutdownNow();
+ }
+ }
+ }
+
+ @Override
+ public void shutdown(long milliseconds) throws ShutdownException {
+ stop();
+ }
+
+ private void checkAndSubmitTasks() {
+ long currentTimestamp = System.currentTimeMillis();
+ for (CreateContinuousQueryPlan plan : continuousQueryPlans.values()) {
+ long nextExecutionTimestamp = nextExecutionTimestamps.get(plan.getContinuousQueryName());
+ if (currentTimestamp >= nextExecutionTimestamp) {
+ TASK_POOL_MANAGER.submit(new ContinuousQueryTask(plan, nextExecutionTimestamp));
+ nextExecutionTimestamps.replace(
+ plan.getContinuousQueryName(), nextExecutionTimestamp + plan.getEveryInterval());
+ }
+ }
+ }
+
+ public boolean register(CreateContinuousQueryPlan plan, boolean writeLog)
+ throws ContinuousQueryAlreadyExistException {
+
+ acquireRegistrationLock();
+
+ if (continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ releaseRegistrationLock();
+ throw new ContinuousQueryAlreadyExistException(plan.getContinuousQueryName());
+ }
+
+ try {
+ if (writeLog) {
+ IoTDB.metaManager.createContinuousQuery(plan);
+ }
+ doRegister(plan);
+
+ } catch (Exception e) {
+ logger.error(e.getMessage());
+ } finally {
+ releaseRegistrationLock();
+ }
+ return true;
+ }
+
+ private void doRegister(CreateContinuousQueryPlan plan) {
+ continuousQueryPlans.put(plan.getContinuousQueryName(), plan);
+ nextExecutionTimestamps.put(plan.getContinuousQueryName(), plan.getCreationTimestamp());
+ }
+
+ public void deregisterAll() throws ContinuousQueryNotExistException {
+ for (String cqName : continuousQueryPlans.keySet()) {
+ deregister(new DropContinuousQueryPlan(cqName));
+ }
+ }
+
+ public boolean deregister(DropContinuousQueryPlan plan) throws ContinuousQueryNotExistException {
+
+ acquireRegistrationLock();
+
+ if (!continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
+ releaseRegistrationLock();
+ throw new ContinuousQueryNotExistException(plan.getContinuousQueryName());
+ }
+
+ try {
+ IoTDB.metaManager.dropContinuousQuery(plan);
+ doDeregister(plan);
+
+ } catch (Exception e) {
+ logger.error(e.getMessage());
+ } finally {
+ releaseRegistrationLock();
+ }
Review comment:
```suggestion
try {
if (!continuousQueryPlans.containsKey(plan.getContinuousQueryName())) {
throw new ContinuousQueryException(plan.getContinuousQueryName() + ": xxxx");
}
IoTDB.metaManager.dropContinuousQuery(plan);
doDeregister(plan);
} catch (ContinuousQueryException e) {
throw e;
} catch (Exception e) {
logger.error(e.getMessage());
} finally {
releaseRegistrationLock();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu merged pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu merged pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656323329
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
+ Math.ceil(
+ (float) plan.getForInterval()
+ / ((GroupByClauseComponent)
+ plan.getQueryOperator().getSpecialClauseComponent())
+ .getUnit()));
+
+ Object[][] columns = getColumns(columnSize, fetchSize, dataType);
+ long[][] timestamps = new long[columnSize][fetchSize];
+ int[] rowNums = new int[columnSize];
+
+ boolean hasNext = true;
+
+ while (hasNext) {
+ int rowNum = 0;
+ Arrays.fill(rowNums, 0);
+
+ while (++rowNum <= fetchSize) {
+ if (!result.hasNextWithoutConstraint()) {
+ hasNext = false;
+ break;
+ }
+ RowRecord record = result.nextWithoutConstraint();
+ fillColumns(columns, dataType, record, rowNums, timestamps);
+ }
+
+ for (int i = 0; i < columnSize; i++) {
+ if (rowNums[i] > 0) {
+ insertTabletPlans[i].setTimes(timestamps[i]);
+ insertTabletPlans[i].setColumns(columns[i]);
+ insertTabletPlans[i].setRowCount(rowNums[i]);
+ planExecutor.insertTablet(insertTabletPlans[i]);
+ }
+ }
+ }
+ }
+
+ private InsertTabletPlan[] getInsertTabletPlans(
+ int columnSize, QueryDataSet result, TSDataType dataType) throws IllegalPathException {
+ List<PartialPath> targetPaths = getTargetPaths(result.getPaths());
Review comment:
no, since paths may be different in every query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-865752485
[](https://coveralls.io/builds/40771419)
Coverage increased (+0.1%) to 68.001% when pulling **53271f4188ce89ae0bb06d26b6fe4188956366b0 on mzp0514:cq** into **d8b4ce8e6fbc4e25a55c7244a9e86592b8487002 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656094337
##########
File path: antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/SqlBase.g4
##########
@@ -329,6 +336,23 @@ sequenceClause
: LR_BRACKET constant (COMMA constant)* RR_BRACKET
;
+resampleClause
+ : RESAMPLE (EVERY DURATION)? (FOR DURATION)?;
Review comment:
Does your tests cover all combinations of (EVERY DURATION) and (FOR DURATION) ?
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,202 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <new_path> | <new_path_suffix>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
+* `<function>` 指定聚合函数，目前支持 `count`, `sum`, `avg`, `last_value`, `first_value`, `min_time`, `max_time`, `min_value`, `max_value` 等。
+* `<path_prefix>` 与 `<path_suffix>` 拼接成完整的查询原时间序列。
+* `<new_path>` 或 `<new_path_suffix>` 指定将查询出的数据写入的结果序列路径。
+* `<group_by_interval>` 指定时间分组长度。
+* `<level>`指按照序列第 `<level>` 层分组，将第 `<level>` 层以下的所有序列聚合。
+
+
+注：
+
+* `<for_interval>`,`<every_interval>` 可选择指定。如果用户没有指定其中的某一项，则未指定项的值按照`<group_by_interval>` 处理。
Review comment:
Are these rules covered by tests?
Right & wrong test cases should be included.
##########
File path: server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByTimeDataSet.java
##########
@@ -80,8 +82,13 @@ public GroupByTimeDataSet(
this.dataTypes = new ArrayList<>();
this.paths = new ArrayList<>();
- for (int i = 0; i < finalPaths.size(); i++) {
- this.dataTypes.add(TSDataType.INT64);
+ for (Map.Entry<String, AggregateResult> entry : finalPaths.entrySet()) {
+ try {
+ this.paths.add(new PartialPath(entry.getKey()));
+ } catch (IllegalPathException e) {
+ e.printStackTrace();
Review comment:
???
##########
File path: server/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java
##########
@@ -102,4 +102,27 @@ public static TSDataType getPredictedDataType(Object value, boolean inferType) {
return TSDataType.TEXT;
}
+
+ public static TSDataType getAggrDataType(String aggrFuncName, TSDataType dataType) {
Review comment:
Why we need this new method here?
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,202 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <new_path> | <new_path_suffix>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
+* `<function>` 指定聚合函数，目前支持 `count`, `sum`, `avg`, `last_value`, `first_value`, `min_time`, `max_time`, `min_value`, `max_value` 等。
+* `<path_prefix>` 与 `<path_suffix>` 拼接成完整的查询原时间序列。
+* `<new_path>` 或 `<new_path_suffix>` 指定将查询出的数据写入的结果序列路径。
+* `<group_by_interval>` 指定时间分组长度。
+* `<level>`指按照序列第 `<level>` 层分组，将第 `<level>` 层以下的所有序列聚合。
+
+
+注：
+
+* `<for_interval>`,`<every_interval>` 可选择指定。如果用户没有指定其中的某一项，则未指定项的值按照`<group_by_interval>` 处理。
+ * `<every_interval>`，`<for_interval>`，`<group_by_interval>` 的值均应大于 0。
+ * `<group_by_interval>` 的值应小于`<for_interval>`的值，否则系统会按照等于`<for_interval>`的值处理。
+ * 用户应当结合实际需求指定合适的 `<for_interval>` 与 `<every_interval>`。
+ * 若 `<for_interval>` 大于 `<every_interval>`，每次的查询窗口会有部分数据重叠，从查询性能角度这种配置不被建议。
+ * 若 `<for_interval>` 小于 `<every_interval>`，每次的查询窗口之间可能会有未覆盖到的数据。
+* 对于结果序列路径
+ * 用户可以选择指定`<new_path>`，即完整的时间序列路径，用户可以在路径中使用 `${x}` 变量来表示原始时间序列中 `level = x` 的节点名称。
+
+ * 用户也可以仅指定`<new_path_suffix>`
+ * 若用户指定 `<level> = l`，则系统生成的结果时间序列路径为 `${0}.${1}. ... .${l}.<new_path_suffix>`
+ * 若用户未指定 `<level>`，令原始时间序列最大层数为 `L`，
+ 则系统生成的结果时间序列路径为 `${0}.${1}. ... .${L - 1}.<new_path_suffix>`。
+
+### 例子
+
+#### 原始时间序列
+````
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+
+````
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+| Time|root.ln.wf02.wt02.temperature|root.ln.wf02.wt01.temperature|root.ln.wf01.wt02.temperature|root.ln.wf01.wt01.temperature|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+|2021-05-11T22:18:14.598+08:00| 121.0| 72.0| 183.0| 115.0|
+|2021-05-11T22:18:19.941+08:00| 0.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:24.949+08:00| 122.0| 45.0| 11.0| 14.0|
+|2021-05-11T22:18:29.967+08:00| 47.0| 14.0| 59.0| 181.0|
+|2021-05-11T22:18:34.979+08:00| 182.0| 113.0| 29.0| 180.0|
+|2021-05-11T22:18:39.990+08:00| 42.0| 11.0| 52.0| 19.0|
+|2021-05-11T22:18:44.995+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:49.999+08:00| 137.0| 172.0| 135.0| 193.0|
+|2021-05-11T22:18:55.003+08:00| 16.0| 124.0| 183.0| 18.0|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+````
+#### 创建 `cq1`
+````
+CREATE CONTINUOUS QUERY cq1 BEGIN SELECT max_value(temperature) INTO temperature_max FROM root.ln.*.* GROUP BY time(10s) END
+````
+
+每隔 10s 查询 `root.ln.*.*.temperature` 在前 10s 内的最大值（结果以10s为一组），
+将结果写入到 `${0}.${1}.${2}.${3}.temperature_max` 中，
+结果将产生4条新序列：
+````
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+| Time|root.ln.wf02.wt02.temperature_max|root.ln.wf02.wt01.temperature_max|root.ln.wf01.wt02.temperature_max|root.ln.wf01.wt01.temperature_max|
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+|2021-05-11T22:18:16.964+08:00| 122.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:26.949+08:00| 182.0| 113.0| 59.0| 181.0|
+|2021-05-11T22:18:36.950+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:46.948+08:00| 137.0| 172.0| 183.0| 193.0|
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+````
+#### 创建 `cq2`
+````
+CREATE CONTINUOUS QUERY cq2 RESAMPLE EVERY 20s FOR 20s BEGIN SELECT avg(temperature) INTO temperature_avg FROM root.ln.*.* GROUP BY time(10s), level=2 END
+````
+
+每隔 20s 查询 `root.ln.*.*.temperature` 在前 20s 内的平均值（结果以10s为一组，按照第2层节点分组），
+将结果写入到 `${0}.${1}.${2}.temperature_avg` 中。
+结果将产生如下两条新序列，
+其中 `root.ln.wf02.temperature_avg` 由 `root.ln.wf02.wt02.temperature` 和 `root.ln.wf02.wt01.temperature` 聚合计算生成，
+`root.ln.wf01.temperature_avg` 由 `root.ln.wf01.wt02.temperature` 和 `root.ln.wf01.wt01.temperature` 聚合计算生成。
+
+````
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.temperature_avg| null| root.ln| DOUBLE| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.temperature_avg| null| root.ln| DOUBLE| GORILLA| SNAPPY|null| null|
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+----------------------------+----------------------------+
+| Time|root.ln.wf02.temperature_avg|root.ln.wf01.temperature_avg|
++-----------------------------+----------------------------+----------------------------+
+|2021-05-11T22:18:16.969+08:00| 58.75| 49.0|
+|2021-05-11T22:18:26.969+08:00| 89.0| 112.25|
+|2021-05-11T22:18:36.967+08:00| 42.25| 61.5|
+|2021-05-11T22:18:46.967+08:00| 112.25| 132.25|
++-----------------------------+----------------------------+----------------------------+
+````
+#### 创建 `cq3`
+````
+CREATE CONTINUOUS QUERY cq3 RESAMPLE EVERY 20s FOR 20s BEGIN SELECT avg(temperature) INTO root.ln_cq.${2}.temperature_avg FROM root.ln.*.* GROUP BY time(10s), level=2 END
+````
+查询模式与 cq2 相同，
+将结果写入到 `root.ln_cq.${2}.temperature_avg` 中。
Review comment:
Question:
Is the target path pattern `root.${3}.ln_cq.${2}.temperature_avg` allowed?
##########
File path: antlr/src/main/antlr4/org/apache/iotdb/db/qp/sql/SqlBase.g4
##########
@@ -105,6 +105,13 @@ statement
| STOP TRIGGER triggerName=ID #stopTrigger
| SHOW TRIGGERS #showTriggers
| selectClause fromClause whereClause? specialClause? #selectStatement
+ | CREATE CONTINUOUS QUERY continuousQueryName=ID
+ resampleClause?
+ BEGIN
+ cqSelectIntoClause
+ END #createContinuousQueryStatement
Review comment:
Move BEGIN and END into cqSelectIntoClause?
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
Review comment:
Rename it to ContinuousQueryTask?
##########
File path: server/src/test/java/org/apache/iotdb/db/integration/IoTDBContinuousQueryIT.java
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.integration;
+
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.jdbc.Config;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.*;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Notice that, all test begins with "IoTDB" is integration test. All test which will start the
+ * IoTDB server should be defined as integration test.
+ */
+public class IoTDBContinuousQueryIT {
Review comment:
Add tests for `show cq`.
##########
File path: server/src/main/java/org/apache/iotdb/db/auth/entity/PrivilegeType.java
##########
@@ -46,6 +46,8 @@
DROP_TRIGGER,
START_TRIGGER,
STOP_TRIGGER,
+ CREATE_CONTINUOUS_QUERY,
Review comment:
Please add UTs for these.
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/utils/DatetimeUtils.java
##########
@@ -621,6 +622,16 @@ public static long convertDurationStrToLong(
}
}
+ public static TimeUnit toTimeUnit(String timestampPrecision) {
Review comment:
What about `y` / `m` / `d` ?
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/Planner.java
##########
@@ -64,6 +65,15 @@ public PhysicalPlan parseSQLToPhysicalPlan(String sqlStr, ZoneId zoneId, int fet
return new PhysicalGenerator().transformToPhysicalPlan(operator, fetchSize);
}
+ public GroupByTimePlan operatorToPhysicalPlan(QueryOperator operator, int fetchSize)
Review comment:
```suggestion
public GroupByTimePlan cqQueryOperatorToGroupByTimePlan(QueryOperator operator, int fetchSize)
```
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/logical/Operator.java
##########
@@ -165,5 +165,8 @@ public PhysicalPlan generatePhysicalPlan(PhysicalGenerator generator)
DELETE_PARTITION,
LOAD_CONFIGURATION,
CREATE_SCHEMA_SNAPSHOT,
+ CREATE_CONTINUOUS_QUERY,
+ DROP_CONTINUOUS_QUERY,
+ SHOW_CONTINUOUS_QUERIES
Review comment:
```suggestion
CREATE_CONTINUOUS_QUERY,
DROP_CONTINUOUS_QUERY,
SHOW_CONTINUOUS_QUERIES
```
##########
File path: docs/zh/UserGuide/Advanced-Features/Continuous-Query.md
##########
@@ -0,0 +1,202 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 连续查询（Continuous Query, CQ）
+
+我们可以通过 SQL 语句注册、或卸载一个 CQ 实例，以及查询到所有已经注册的 CQ 配置信息。
+
+## 注册 CQ
+
+### 语法
+
+```sql
+CREATE CONTINUOUS QUERY <cq_id>
+[RESAMPLE EVERY <every_interval> FOR <for_interval>]
+BEGIN
+SELECT <function>(<path_suffix>) INTO <new_path> | <new_path_suffix>
+FROM <path_prefix>
+GROUP BY time(<group_by_interval>) [, level = <level>]
+END
+```
+
+其中：
+
+* `<cq_id>` 指定 CQ 全局唯一的 id。
+* `<every_interval>` 指定查询执行时间间隔，可选择指定。
+* `<for_interval>` 指定每次查询的窗口大小，即查询时间范围为`[now() - <for_interval>, now())`，其中 `now()` 指查询时的时间戳。可选择指定。
+* `<function>` 指定聚合函数，目前支持 `count`, `sum`, `avg`, `last_value`, `first_value`, `min_time`, `max_time`, `min_value`, `max_value` 等。
+* `<path_prefix>` 与 `<path_suffix>` 拼接成完整的查询原时间序列。
+* `<new_path>` 或 `<new_path_suffix>` 指定将查询出的数据写入的结果序列路径。
+* `<group_by_interval>` 指定时间分组长度。
+* `<level>`指按照序列第 `<level>` 层分组，将第 `<level>` 层以下的所有序列聚合。
+
+
+注：
+
+* `<for_interval>`,`<every_interval>` 可选择指定。如果用户没有指定其中的某一项，则未指定项的值按照`<group_by_interval>` 处理。
+ * `<every_interval>`，`<for_interval>`，`<group_by_interval>` 的值均应大于 0。
+ * `<group_by_interval>` 的值应小于`<for_interval>`的值，否则系统会按照等于`<for_interval>`的值处理。
+ * 用户应当结合实际需求指定合适的 `<for_interval>` 与 `<every_interval>`。
+ * 若 `<for_interval>` 大于 `<every_interval>`，每次的查询窗口会有部分数据重叠，从查询性能角度这种配置不被建议。
+ * 若 `<for_interval>` 小于 `<every_interval>`，每次的查询窗口之间可能会有未覆盖到的数据。
+* 对于结果序列路径
+ * 用户可以选择指定`<new_path>`，即完整的时间序列路径，用户可以在路径中使用 `${x}` 变量来表示原始时间序列中 `level = x` 的节点名称。
+
+ * 用户也可以仅指定`<new_path_suffix>`
+ * 若用户指定 `<level> = l`，则系统生成的结果时间序列路径为 `${0}.${1}. ... .${l}.<new_path_suffix>`
+ * 若用户未指定 `<level>`，令原始时间序列最大层数为 `L`，
+ 则系统生成的结果时间序列路径为 `${0}.${1}. ... .${L - 1}.<new_path_suffix>`。
+
+### 例子
+
+#### 原始时间序列
+````
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++-----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+
+````
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+| Time|root.ln.wf02.wt02.temperature|root.ln.wf02.wt01.temperature|root.ln.wf01.wt02.temperature|root.ln.wf01.wt01.temperature|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+|2021-05-11T22:18:14.598+08:00| 121.0| 72.0| 183.0| 115.0|
+|2021-05-11T22:18:19.941+08:00| 0.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:24.949+08:00| 122.0| 45.0| 11.0| 14.0|
+|2021-05-11T22:18:29.967+08:00| 47.0| 14.0| 59.0| 181.0|
+|2021-05-11T22:18:34.979+08:00| 182.0| 113.0| 29.0| 180.0|
+|2021-05-11T22:18:39.990+08:00| 42.0| 11.0| 52.0| 19.0|
+|2021-05-11T22:18:44.995+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:49.999+08:00| 137.0| 172.0| 135.0| 193.0|
+|2021-05-11T22:18:55.003+08:00| 16.0| 124.0| 183.0| 18.0|
++-----------------------------+-----------------------------+-----------------------------+-----------------------------+-----------------------------+
+````
+#### 创建 `cq1`
+````
+CREATE CONTINUOUS QUERY cq1 BEGIN SELECT max_value(temperature) INTO temperature_max FROM root.ln.*.* GROUP BY time(10s) END
+````
+
+每隔 10s 查询 `root.ln.*.*.temperature` 在前 10s 内的最大值（结果以10s为一组），
+将结果写入到 `${0}.${1}.${2}.${3}.temperature_max` 中，
+结果将产生4条新序列：
+````
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.wt02.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf02.wt01.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt02.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.wt01.temperature_max| null| root.ln| FLOAT| GORILLA| SNAPPY|null| null|
++---------------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+| Time|root.ln.wf02.wt02.temperature_max|root.ln.wf02.wt01.temperature_max|root.ln.wf01.wt02.temperature_max|root.ln.wf01.wt01.temperature_max|
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+|2021-05-11T22:18:16.964+08:00| 122.0| 68.0| 68.0| 103.0|
+|2021-05-11T22:18:26.949+08:00| 182.0| 113.0| 59.0| 181.0|
+|2021-05-11T22:18:36.950+08:00| 78.0| 38.0| 123.0| 52.0|
+|2021-05-11T22:18:46.948+08:00| 137.0| 172.0| 183.0| 193.0|
++-----------------------------+---------------------------------+---------------------------------+---------------------------------+---------------------------------+
+````
+#### 创建 `cq2`
+````
+CREATE CONTINUOUS QUERY cq2 RESAMPLE EVERY 20s FOR 20s BEGIN SELECT avg(temperature) INTO temperature_avg FROM root.ln.*.* GROUP BY time(10s), level=2 END
+````
+
+每隔 20s 查询 `root.ln.*.*.temperature` 在前 20s 内的平均值（结果以10s为一组，按照第2层节点分组），
+将结果写入到 `${0}.${1}.${2}.temperature_avg` 中。
+结果将产生如下两条新序列，
+其中 `root.ln.wf02.temperature_avg` 由 `root.ln.wf02.wt02.temperature` 和 `root.ln.wf02.wt01.temperature` 聚合计算生成，
+`root.ln.wf01.temperature_avg` 由 `root.ln.wf01.wt02.temperature` 和 `root.ln.wf01.wt01.temperature` 聚合计算生成。
+
+````
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+| timeseries|alias|storage group|dataType|encoding|compression|tags|attributes|
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+|root.ln.wf02.temperature_avg| null| root.ln| DOUBLE| GORILLA| SNAPPY|null| null|
+|root.ln.wf01.temperature_avg| null| root.ln| DOUBLE| GORILLA| SNAPPY|null| null|
++----------------------------+-----+-------------+--------+--------+-----------+----+----------+
+````
+````
++-----------------------------+----------------------------+----------------------------+
+| Time|root.ln.wf02.temperature_avg|root.ln.wf01.temperature_avg|
++-----------------------------+----------------------------+----------------------------+
+|2021-05-11T22:18:16.969+08:00| 58.75| 49.0|
+|2021-05-11T22:18:26.969+08:00| 89.0| 112.25|
+|2021-05-11T22:18:36.967+08:00| 42.25| 61.5|
+|2021-05-11T22:18:46.967+08:00| 112.25| 132.25|
++-----------------------------+----------------------------+----------------------------+
+````
+#### 创建 `cq3`
+````
+CREATE CONTINUOUS QUERY cq3 RESAMPLE EVERY 20s FOR 20s BEGIN SELECT avg(temperature) INTO root.ln_cq.${2}.temperature_avg FROM root.ln.*.* GROUP BY time(10s), level=2 END
+````
+查询模式与 cq2 相同，
+将结果写入到 `root.ln_cq.${2}.temperature_avg` 中。
Review comment:
We can add more allowed target path patterns as examples (no need to show the query results).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656322435
##########
File path: server/src/main/java/org/apache/iotdb/db/cq/ContinuousQuery.java
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.cq;
+
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.exception.metadata.IllegalPathException;
+import org.apache.iotdb.db.exception.metadata.MetadataException;
+import org.apache.iotdb.db.exception.query.QueryProcessException;
+import org.apache.iotdb.db.metadata.PartialPath;
+import org.apache.iotdb.db.qp.Planner;
+import org.apache.iotdb.db.qp.executor.PlanExecutor;
+import org.apache.iotdb.db.qp.logical.crud.GroupByClauseComponent;
+import org.apache.iotdb.db.qp.logical.crud.QueryOperator;
+import org.apache.iotdb.db.qp.logical.crud.SelectComponent;
+import org.apache.iotdb.db.qp.physical.crud.GroupByTimePlan;
+import org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan;
+import org.apache.iotdb.db.qp.physical.sys.CreateContinuousQueryPlan;
+import org.apache.iotdb.db.query.context.QueryContext;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.utils.TypeInferenceUtils;
+import org.apache.iotdb.tsfile.exception.filter.QueryFilterOptimizationException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.read.common.Field;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.read.common.RowRecord;
+import org.apache.iotdb.tsfile.read.query.dataset.QueryDataSet;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class ContinuousQuery implements Runnable {
+
+ private static final Logger logger = LoggerFactory.getLogger(ContinuousQuery.class);
+
+ private final PlanExecutor planExecutor;
+ private final CreateContinuousQueryPlan plan;
+ private final Planner planner;
+
+ public ContinuousQuery(CreateContinuousQueryPlan plan) throws QueryProcessException {
+ this.plan = plan;
+ this.planExecutor = new PlanExecutor();
+ this.planner = new Planner();
+ }
+
+ @Override
+ public void run() {
+
+ try {
+
+ GroupByTimePlan queryPlan = getQueryPlan();
+
+ if (queryPlan.getDeduplicatedPaths().isEmpty()) {
+ logger.error(plan.getContinuousQueryName() + ": deduplicated paths empty");
+ return;
+ }
+
+ QueryDataSet result = doQuery(queryPlan);
+
+ if (result == null) {
+ logger.error(plan.getContinuousQueryName() + ": query result empty");
+ return;
+ }
+
+ doInsert(result, queryPlan);
+
+ } catch (Exception e) {
+ logger.error(plan.getContinuousQueryName() + ": run error", e);
+ }
+ }
+
+ private GroupByTimePlan getQueryPlan() throws QueryProcessException {
+
+ QueryOperator queryOperator = plan.getQueryOperator();
+
+ SelectComponent selectComponentCopy = new SelectComponent(queryOperator.getSelectComponent());
+
+ GroupByTimePlan queryPlan = planner.operatorToPhysicalPlan(queryOperator, 1024);
+
+ queryOperator.setSelectComponent(selectComponentCopy);
+
+ long timestamp = System.currentTimeMillis();
+ queryPlan.setStartTime(timestamp - plan.getForInterval());
+ queryPlan.setEndTime(timestamp);
+
+ return queryPlan;
+ }
+
+ private QueryDataSet doQuery(GroupByTimePlan queryPlan)
+ throws StorageEngineException, QueryFilterOptimizationException, MetadataException,
+ IOException, InterruptedException, QueryProcessException {
+ long queryId =
+ QueryResourceManager.getInstance()
+ .assignQueryId(true, 1024, queryPlan.getDeduplicatedPaths().size());
+
+ QueryDataSet result;
+ result = planExecutor.processQuery(queryPlan, new QueryContext(queryId));
+ QueryResourceManager.getInstance().endQuery(queryId);
+ return result;
+ }
+
+ private void doInsert(QueryDataSet result, GroupByTimePlan queryPlan)
+ throws QueryProcessException, IOException, IllegalPathException {
+
+ int columnSize = result.getDataTypes().size();
+ TSDataType dataType =
+ TypeInferenceUtils.getAggrDataType(
+ queryPlan.getAggregations().get(0), queryPlan.getDataTypes().get(0));
+
+ InsertTabletPlan[] insertTabletPlans = getInsertTabletPlans(columnSize, result, dataType);
+
+ int fetchSize =
+ (int)
+ Math.min(
+ 10,
Review comment:
if empty query, rowNum will be 0 < batchSize
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-865752485
[](https://coveralls.io/builds/40771764)
Coverage increased (+0.02%) to 67.907% when pulling **776f145c27da9492ba045d5bd6ee21c3a09050e0 on mzp0514:cq** into **d8b4ce8e6fbc4e25a55c7244a9e86592b8487002 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] mzp0514 commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
mzp0514 commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656863923
##########
File path: server/src/main/java/org/apache/iotdb/db/utils/TypeInferenceUtils.java
##########
@@ -102,4 +102,27 @@ public static TSDataType getPredictedDataType(Object value, boolean inferType) {
return TSDataType.TEXT;
}
+
+ public static TSDataType getAggrDataType(String aggrFuncName, TSDataType dataType) {
Review comment:
because we need to know the data type of the aggregation result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] SteveYurongSu commented on a change in pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
SteveYurongSu commented on a change in pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#discussion_r656070582
##########
File path: server/src/main/java/org/apache/iotdb/db/auth/AuthorityChecker.java
##########
@@ -157,6 +156,10 @@ private static int translateToPermissionId(Operator.OperatorType type) {
return PrivilegeType.START_TRIGGER.ordinal();
case STOP_TRIGGER:
return PrivilegeType.STOP_TRIGGER.ordinal();
+ case CREATE_CONTINUOUS_QUERY:
Review comment:
Please update the auth doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3162: [IOTDB-1143] Continuous query
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3162:
URL: https://github.com/apache/iotdb/pull/3162#issuecomment-865752485
[](https://coveralls.io/builds/40771564)
Coverage increased (+0.04%) to 67.93% when pulling **776f145c27da9492ba045d5bd6ee21c3a09050e0 on mzp0514:cq** into **d8b4ce8e6fbc4e25a55c7244a9e86592b8487002 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org