You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/01 13:17:08 UTC
[GitHub] [incubator-doris] chaoyli opened a new pull request #3744: [Feature] Add QueryDetail to store query statistics.
chaoyli opened a new pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744
1. Store the query statistics in memory.
2. Supporting RESTFUL interface to get the statistics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r438853945
##########
File path: be/src/service/http_service.cpp
##########
@@ -91,7 +91,7 @@ Status HttpService::start() {
// register metrics
{
auto action = new MetricsAction(DorisMetrics::instance()->metrics());
- _ev_http_server->register_handler(HttpMethod::GET, "/metrics", action);
+ _ev_http_server->register_handler(HttpMethod::GET, "/api/metrics", action);
Review comment:
Better not changing it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#issuecomment-636907661
@chaoyli Hi, you should add the doc for new http api.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433261194
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
+ return;
+ }
+ queryNumber++;
+ queryDetails.put(queryDetail.getQueryId(), queryDetail);
Review comment:
Do we need to clean up outdated querydetails When getquerydetails() is not called for a long time ？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r439830887
##########
File path: fe/src/main/java/org/apache/doris/common/Log4jConfig.java
##########
@@ -179,10 +179,6 @@ private static void reconfig() throws IOException {
strSub = new StrSubstitutor(new Interpolator(properties));
newXmlConfTemplate = strSub.replace(newXmlConfTemplate);
- System.out.println("=====");
Review comment:
Why removing this?
Because it add many a long xml upon start but it has little sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433259804
##########
File path: fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -126,6 +129,11 @@ private void auditAfterExec(String origStmt, StatementBase parsedStmt, PQuerySta
MetricRepo.HISTO_QUERY_LATENCY.update(elapseMs);
}
ctx.getAuditEventBuilder().setIsQuery(true);
+ ctx.getQueryDetail().setEventTime(endTime);
Review comment:
Can you explain why set eventTime as endTime?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433261194
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
+ return;
+ }
+ queryNumber++;
+ queryDetails.put(queryDetail.getQueryId(), queryDetail);
Review comment:
Do we need to clean up outdated querydetails when getquerydetails() is not called for a long time ？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433591288
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetail.java
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+public class QueryDetail {
+ public enum QueryMemState {
+ RUNNING,
+ FINISHED,
+ FAILED,
+ CANCELLED
+ };
+
+ // When query received, FE will construct a QueryDtail
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433261194
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
+ return;
+ }
+ queryNumber++;
+ queryDetails.put(queryDetail.getQueryId(), queryDetail);
Review comment:
Do we need to clean up outdated querydetails when getQueryDetails() is not called for a long time ？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433281258
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
Review comment:
You could use putIfAbsent
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433280630
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
Review comment:
Why not use queue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r437151250
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetail.java
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+public class QueryDetail {
+ public enum QueryMemState {
+ RUNNING,
+ FINISHED,
+ FAILED,
+ CANCELLED
+ };
+
+ // When query received, FE will construct a QueryDtail
+ // object. This object will set queryId, startTime, sql
+ // fields. As well state is be set as RUNNING.
+ // After query finished, endTime and latency will
+ // be set and state will be updated to be FINISHED/FAILED/CANCELLED
+ // according to the query execution results.
+ // So, one query will be inserted into as a item and
+ // be udpated upon finished. To indicate the two event,
+ // a extra field named eventTime is added.
+ private long eventTime;
Review comment:
Every query will have two event.
When starts a query, it will set QueryMemState as RUNNING.
When finished, it will set QueryMemState as FINISHED/FAILED/CANCELLED according to query execution situation.
QueryMonitor will collect QueryMemState starts a event time to handle this situation.
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
Review comment:
Because I first add QueryDetail to this data structure.
Then, I will update it when query finished.
To find the item to updated fast, I use a map to address the problem.
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
Review comment:
I will change it to remove the prior items.
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
Review comment:
Because I need to increment queryNumber according to the judgement.
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
+ return;
+ }
+ queryNumber++;
+ queryDetails.put(queryDetail.getQueryId(), queryDetail);
Review comment:
I will clear it in the following request to getQueryDetails().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
xy720 commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433261194
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
+ return;
+ }
+ queryNumber++;
+ queryDetails.put(queryDetail.getQueryId(), queryDetail);
Review comment:
Do we need to clean up outdated querydetails？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli merged pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433294323
##########
File path: fe/src/main/java/org/apache/doris/http/rest/ProfileAction.java
##########
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.rest;
+
+import org.apache.doris.common.util.ProfileManager;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+
+public class ProfileAction extends RestBaseAction {
Review comment:
Add some comment about this class.
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
Review comment:
You call this class a `Queue`, so I think it some kind of FIFO struct.
But here if you find `queryNumber > queryCapacity`, you just discard the new incoming queries?
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetail.java
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+public class QueryDetail {
+ public enum QueryMemState {
+ RUNNING,
+ FINISHED,
+ FAILED,
+ CANCELLED
+ };
+
+ // When query received, FE will construct a QueryDtail
Review comment:
```suggestion
// When query received, FE will construct a QueryDetail
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r438739390
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> runningQueries = Maps.newHashMap();
+ private static LinkedList<QueryDetail> totalQueries = new LinkedList<QueryDetail>();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (runningQueries.get(queryDetail.getQueryId()) == null) {
+ queryNumber++;
+ runningQueries.put(queryDetail.getQueryId(), queryDetail);
+ totalQueries.add(queryDetail);
+ } else {
+ QueryDetail qDetail = runningQueries.remove(queryDetail.getQueryId());
+ qDetail.setLatency(queryDetail.getLatency());
+ qDetail.setState(queryDetail.getState());
+ }
+ if (queryNumber > queryCapacity) {
+ QueryDetail qDetail = totalQueries.remove();
+ runningQueries.remove(qDetail.getQueryId());
Review comment:
OK, It can use totalQueries instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r438736209
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetail.java
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+public class QueryDetail {
+ public enum QueryMemState {
Review comment:
Because it has a class use QueryState in another meaning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433589126
##########
File path: fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -126,6 +129,11 @@ private void auditAfterExec(String origStmt, StatementBase parsedStmt, PQuerySta
MetricRepo.HISTO_QUERY_LATENCY.update(elapseMs);
}
ctx.getAuditEventBuilder().setIsQuery(true);
+ ctx.getQueryDetail().setEventTime(endTime);
Review comment:
Because one query will be set when started, and then update when finished.
If query monitor collect between the two point. The query monitor can use eventTime to distinguish the two event. You can get the message from annotation in QueryDetail class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433591664
##########
File path: fe/src/main/java/org/apache/doris/qe/SessionVariable.java
##########
@@ -109,7 +109,7 @@
// if true, need report to coordinator when plan fragment execute successfully.
@VariableMgr.VarAttr(name = IS_REPORT_SUCCESS)
- private boolean isReportSucc = false;
+ private boolean isReportSucc = true;
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r438226057
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetail.java
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+public class QueryDetail {
+ public enum QueryMemState {
Review comment:
Why call it "Mem" State? How about just QueryState？
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> runningQueries = Maps.newHashMap();
+ private static LinkedList<QueryDetail> totalQueries = new LinkedList<QueryDetail>();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (runningQueries.get(queryDetail.getQueryId()) == null) {
+ queryNumber++;
+ runningQueries.put(queryDetail.getQueryId(), queryDetail);
+ totalQueries.add(queryDetail);
+ } else {
+ QueryDetail qDetail = runningQueries.remove(queryDetail.getQueryId());
+ qDetail.setLatency(queryDetail.getLatency());
+ qDetail.setState(queryDetail.getState());
+ }
+ if (queryNumber > queryCapacity) {
+ QueryDetail qDetail = totalQueries.remove();
+ runningQueries.remove(qDetail.getQueryId());
Review comment:
Forget to update the queryNumber?
And I think queryNumber is useless, you can just use `totalQueries.size()`
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> runningQueries = Maps.newHashMap();
+ private static LinkedList<QueryDetail> totalQueries = new LinkedList<QueryDetail>();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (runningQueries.get(queryDetail.getQueryId()) == null) {
+ queryNumber++;
+ runningQueries.put(queryDetail.getQueryId(), queryDetail);
+ totalQueries.add(queryDetail);
+ } else {
+ QueryDetail qDetail = runningQueries.remove(queryDetail.getQueryId());
Review comment:
Are you sure to "remove" it here? I think it should be "get"?
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> runningQueries = Maps.newHashMap();
+ private static LinkedList<QueryDetail> totalQueries = new LinkedList<QueryDetail>();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (runningQueries.get(queryDetail.getQueryId()) == null) {
+ queryNumber++;
+ runningQueries.put(queryDetail.getQueryId(), queryDetail);
+ totalQueries.add(queryDetail);
+ } else {
+ QueryDetail qDetail = runningQueries.remove(queryDetail.getQueryId());
+ qDetail.setLatency(queryDetail.getLatency());
+ qDetail.setState(queryDetail.getState());
+ }
+ if (queryNumber > queryCapacity) {
+ QueryDetail qDetail = totalQueries.remove();
+ runningQueries.remove(qDetail.getQueryId());
+ }
+ }
+
+ public static synchronized List<QueryDetail> getQueryDetails(long eventTime) {
+ List<QueryDetail> results = Lists.newArrayList();
+ int pos = 0;
+ Iterator<QueryDetail> it = totalQueries.iterator();
+ while(it.hasNext()) {
+ QueryDetail queryDetail = it.next();
+ if (queryDetail.getEventTime() > eventTime) {
+ results.add(queryDetail);
+ } else {
+ pos++;
+ }
+ }
+ for (int i = 0; i < pos; ++i) {
Review comment:
Why removing the qDetails? This method is not reentrant?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433284738
##########
File path: fe/src/main/java/org/apache/doris/http/rest/ConnectionAction.java
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.rest;
+
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.service.ExecuteEnv;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+// This class is used to get current query_id of connection_id.
+// Every connection holds at most one query at every point.
+// Some we can get query_id firstly, and get query by query_id.
+public class ConnectionAction extends RestBaseAction {
+ public ConnectionAction(ActionController controller) {
+ super(controller);
+ }
+
+ public static void registerAction (ActionController controller) throws IllegalArgException {
+ controller.registerHandler(HttpMethod.GET, "/connection", new ConnectionAction(controller));
+ }
+
+ @Override
+ public void execute(BaseRequest request, BaseResponse response) {
+ String connStr = request.getSingleParameter("connection_id");
+ if (connStr == null) {
+ response.getContent().append("not valid parameter");
+ sendResult(request, response, HttpResponseStatus.BAD_REQUEST);
+ return;
+ }
+ long connectionId = Long.valueOf(connStr.trim());
+ ConnectContext context = ExecuteEnv.getInstance().getScheduler().getContext(connectionId);
+ if (context == null || context.queryId() == null) {
+ response.getContent().append("connection id " + connectionId + " not found.");
+ sendResult(request, response, HttpResponseStatus.NOT_FOUND);
+ return;
+ }
+ String queryId = DebugUtil.printId(context.queryId());
+ response.getContent().append(queryId);
+ sendResult(request, response);
Review comment:
prefer response by json
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433282652
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetail.java
##########
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+public class QueryDetail {
+ public enum QueryMemState {
+ RUNNING,
+ FINISHED,
+ FAILED,
+ CANCELLED
+ };
+
+ // When query received, FE will construct a QueryDtail
+ // object. This object will set queryId, startTime, sql
+ // fields. As well state is be set as RUNNING.
+ // After query finished, endTime and latency will
+ // be set and state will be updated to be FINISHED/FAILED/CANCELLED
+ // according to the query execution results.
+ // So, one query will be inserted into as a item and
+ // be udpated upon finished. To indicate the two event,
+ // a extra field named eventTime is added.
+ private long eventTime;
Review comment:
I still don't understand what's the meaning of eventTime? and why do we need it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433589126
##########
File path: fe/src/main/java/org/apache/doris/qe/ConnectProcessor.java
##########
@@ -126,6 +129,11 @@ private void auditAfterExec(String origStmt, StatementBase parsedStmt, PQuerySta
MetricRepo.HISTO_QUERY_LATENCY.update(elapseMs);
}
ctx.getAuditEventBuilder().setIsQuery(true);
+ ctx.getQueryDetail().setEventTime(endTime);
Review comment:
Because one query will be set when started, and then update when finished.
If query monitor collect between the two point. The query monitor can use eventTime to distinguish the two event.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433281766
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
+ private static Map<String, QueryDetail> queryDetails = Maps.newHashMap();
+ private static int queryNumber = 0;
+ private static int queryCapacity = 10000;
+
+ public static synchronized void addOrUpdateQueryDetail(QueryDetail queryDetail) {
+ if (queryDetails.get(queryDetail.getQueryId()) == null) {
+ if (queryNumber > queryCapacity) {
+ return;
+ }
+ queryNumber++;
+ queryDetails.put(queryDetail.getQueryId(), queryDetail);
+ } else {
+ queryDetails.replace(queryDetail.getQueryId(), queryDetail);
+ }
+ }
+
+ public static synchronized List<QueryDetail> getQueryDetails(long eventTime) {
+ List<QueryDetail> results = Lists.newArrayList();
+ for (QueryDetail queryDetail : queryDetails.values()) {
+ if (queryDetail.getEventTime() > eventTime) {
+ results.add(queryDetail);
+ }
+ }
+ queryDetails.clear();
Review comment:
Why clear when get?
##########
File path: fe/src/main/java/org/apache/doris/qe/SessionVariable.java
##########
@@ -109,7 +109,7 @@
// if true, need report to coordinator when plan fragment execute successfully.
@VariableMgr.VarAttr(name = IS_REPORT_SUCCESS)
- private boolean isReportSucc = false;
+ private boolean isReportSucc = true;
Review comment:
Shouldn't change the default value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli closed pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli closed pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
morningman commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r439830474
##########
File path: docs/zh-CN/administrator-guide/http-actions/connection-action.md
##########
@@ -0,0 +1,36 @@
+---
+{
+ "title": "CONNECTION",
+ "language": "en"
Review comment:
```suggestion
"language": "zh-CN"
```
##########
File path: docs/zh-CN/administrator-guide/http-actions/connection-action.md
##########
@@ -0,0 +1,36 @@
+---
+{
+ "title": "CONNECTION",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# CONNECTION
+## description
Review comment:
中文文档使用中文标题
##########
File path: fe/src/main/java/org/apache/doris/common/Log4jConfig.java
##########
@@ -179,10 +179,6 @@ private static void reconfig() throws IOException {
strSub = new StrSubstitutor(new Interpolator(properties));
newXmlConfTemplate = strSub.replace(newXmlConfTemplate);
- System.out.println("=====");
Review comment:
Why removing this?
##########
File path: docs/zh-CN/administrator-guide/http-actions/query-detail-action.md
##########
@@ -0,0 +1,37 @@
+---
+{
+ "title": "QUERY DETAIL",
+ "language": "en"
Review comment:
```suggestion
"language": "zh-CN"
```
##########
File path: fe/src/main/java/org/apache/doris/PaloFe.java
##########
@@ -73,7 +73,6 @@ public static void start(String dorisHomeDir, String pidDir, String[] args) {
}
CommandLineOptions cmdLineOpts = parseArgs(args);
- System.out.println(cmdLineOpts.toString());
Review comment:
Why removing this?
##########
File path: docs/zh-CN/administrator-guide/http-actions/connection-action.md
##########
@@ -0,0 +1,36 @@
+---
+{
+ "title": "CONNECTION",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# CONNECTION
+## description
+
+ 通过connection_id, 获取当前连接的query_id
+
+ URI: http://fe_host:fe_http_port/api/connection?connection_id=123
+
+## example
Review comment:
需要给出api的返回结果示例
##########
File path: fe/src/main/java/org/apache/doris/qe/QueryDetailQueue.java
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import org.apache.doris.qe.QueryDetail;
+
+// Queue of QueryDetail.
+// It's used to collect queries for monitor.
+// The default copacity is 10000.
+public class QueryDetailQueue {
Review comment:
Add unit test for this class
##########
File path: docs/zh-CN/administrator-guide/http-actions/query-detail-action.md
##########
@@ -0,0 +1,37 @@
+---
+{
+ "title": "QUERY DETAIL",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# QUERY DETAIL
+## description
+
+ 从FE获取所有的query细节，获取关于查询执行的相关信息。
+ event_time的单位是毫秒。
Review comment:
需要说明 event_time 的含义，比如是获取从 event_time 之后的所有 query 的执行情况
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r439169970
##########
File path: be/src/service/http_service.cpp
##########
@@ -91,7 +91,7 @@ Status HttpService::start() {
// register metrics
{
auto action = new MetricsAction(DorisMetrics::instance()->metrics());
- _ev_http_server->register_handler(HttpMethod::GET, "/metrics", action);
+ _ev_http_server->register_handler(HttpMethod::GET, "/api/metrics", action);
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #3744: [Feature] Add QueryDetail to store query statistics.
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on a change in pull request #3744:
URL: https://github.com/apache/incubator-doris/pull/3744#discussion_r433283472
##########
File path: fe/src/main/java/org/apache/doris/http/rest/ConnectionAction.java
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.http.rest;
+
+import io.netty.handler.codec.http.HttpMethod;
+import io.netty.handler.codec.http.HttpResponseStatus;
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.http.ActionController;
+import org.apache.doris.http.BaseRequest;
+import org.apache.doris.http.BaseResponse;
+import org.apache.doris.http.IllegalArgException;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.service.ExecuteEnv;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+// This class is used to get current query_id of connection_id.
+// Every connection holds at most one query at every point.
+// Some we can get query_id firstly, and get query by query_id.
+public class ConnectionAction extends RestBaseAction {
+ public ConnectionAction(ActionController controller) {
+ super(controller);
+ }
+
+ public static void registerAction (ActionController controller) throws IllegalArgException {
+ controller.registerHandler(HttpMethod.GET, "/connection", new ConnectionAction(controller));
Review comment:
prefer to "/api/connection"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org