You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Hudson (JIRA)" <ji...@apache.org> on 2015/09/03 21:07:45 UTC
[jira] [Commented] (AMBARI-12983) Optimize aggregator queries by
performing GROUP BY on server
[ https://issues.apache.org/jira/browse/AMBARI-12983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729601#comment-14729601 ]
Hudson commented on AMBARI-12983:
---------------------------------
FAILURE: Integrated in Ambari-trunk-Commit #3383 (See [https://builds.apache.org/job/Ambari-trunk-Commit/3383/])
AMBARI-12983. Optimize aggregator queries by performing GROUP BY on server. (swagle) (swagle: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=3e0b8f07c388ad2bd67ef23b6dc313f3fef4d117)
* ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITMetricAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITClusterAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
* ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
* ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
* ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java
* ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java
* ambari-metrics/ambari-metrics-timelineservice/pom.xml
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java
* ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml
* ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java
> Optimize aggregator queries by performing GROUP BY on server
> ------------------------------------------------------------
>
> Key: AMBARI-12983
> URL: https://issues.apache.org/jira/browse/AMBARI-12983
> Project: Ambari
> Issue Type: Task
> Components: ambari-metrics
> Affects Versions: 2.1.1
> Reporter: Siddharth Wagle
> Assignee: Siddharth Wagle
> Priority: Critical
> Fix For: 2.1.2
>
>
> Query to do calculate host level time aggregates.
> *Sample*
> {code}
> select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE METRIC_NAME IN ('bytes_in','rpc.rpc.RpcAuthorizationSuccesses') SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID limit 50;
> 0: jdbc:phoenix:localhost:61181:/hbase> select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE METRIC_NAME IN ('bytes_in','rpc.rpc.RpcAuthorizationSuccesses') AND SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID limit 50;
> +-----------------+-------------------+-----------------+-----------------+-------------+------------+------------+
> | SUM(METRIC_SUM) | SUM(METRIC_COUNT) | MAX(METRIC_MAX) | MIN(METRIC_MIN) | METRIC_NAME | HOSTNAME | APP_ID |
> +-----------------+-------------------+-----------------+-----------------+-------------+------------+------------+
> | 1546242.5521201263 | 24 | 564689.2744108599 | 616.7147803365921 | bytes_in | ams-test-1.c.pramod-thangali.internal | HOST |
> | 1521023.3303401857 | 24 | 825825.3247578086 | 220.66636815664341 | bytes_in | ams-test-2.c.pramod-thangali.internal | HOST |
> | 153685.92452883872 | 24 | 25389.11630078307 | 448.15683323834327 | bytes_in | ams-test-3.c.pramod-thangali.internal | HOST |
> | 0.0 | 14 | 0.0 | 0.0 | rpc.rpc.RpcAuthorizationSuccesses | ams-test-1.c.pramod-thangali.internal | datanode |
> | 17.0 | 14 | 2.0 | 0.0 | rpc.rpc.RpcAuthorizationSuccesses | ams-test-1.c.pramod-thangali.internal | namenode |
> | 0.0 | 14 | 0.0 | 0.0 | rpc.rpc.RpcAuthorizationSuccesses | ams-test-2.c.pramod-thangali.internal | datanode |
> +-----------------+-------------------+-----------------+-----------------+-------------+------------+------------+
> {code}
> *Query plan*:
> {code}
> 0: jdbc:phoenix:localhost:61181:/hbase> explain select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID;
> +------------+
> | PLAN |
> +------------+
> | CLIENT PARALLEL 1-WAY FULL SCAN OVER METRIC_RECORD |
> | SERVER FILTER BY (SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000) |
> | SERVER AGGREGATE INTO DISTINCT ROWS BY [METRIC_NAME, HOSTNAME, APP_ID] |
> | CLIENT MERGE SORT |
> +------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)