You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Sid Wagle <sw...@hortonworks.com> on 2015/09/02 23:30:19 UTC
Review Request 38070: Optimize aggregator queries by performing GROUP
BY on server
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38070/
-----------------------------------------------------------
Review request for Ambari, Dmytro Sen, Mahadev Konar, Sumit Mohanty, and Srimanth Gunturi.
Bugs: AMBARI-12983
https://issues.apache.org/jira/browse/AMBARI-12983
Repository: ambari
Description
-------
Use GroupBy aggregators wherever possible.
Query plan:
0: jdbc:phoenix:localhost:61181:/hbase> explain select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID;
+------------+
| PLAN |
+------------+
| CLIENT PARALLEL 1-WAY FULL SCAN OVER METRIC_RECORD |
| SERVER FILTER BY (SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000) |
| SERVER AGGREGATE INTO DISTINCT ROWS BY [METRIC_NAME, HOSTNAME, APP_ID] |
| CLIENT MERGE SORT |
+------------+
Diffs
-----
ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector bdf00b5
ambari-metrics/ambari-metrics-timelineservice/pom.xml d994d52
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java 8a37a57
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java c615804
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java d4f919e
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java 37fb088
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java 642fcfe
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java e0cb3d0
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java 462c8d9
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java ef0f4ce
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java c8b8709
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITClusterAggregator.java 13fa348
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITMetricAggregator.java b480b7a
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 06e153b
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java ef6fc58
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml 2d11412
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml c716bea
ambari-server/src/test/resources/TestAmbaryServer.samples/blueprint_hosts.json 5c4ded0
Diff: https://reviews.apache.org/r/38070/diff/
Testing
-------
Thanks,
Sid Wagle
Re: Review Request 38070: Optimize aggregator queries by performing
GROUP BY on server
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38070/
-----------------------------------------------------------
(Updated Sept. 2, 2015, 9:48 p.m.)
Review request for Ambari, Dmytro Sen, Mahadev Konar, Sumit Mohanty, and Srimanth Gunturi.
Bugs: AMBARI-12983
https://issues.apache.org/jira/browse/AMBARI-12983
Repository: ambari
Description (updated)
-------
Use GroupBy aggregators wherever possible.
Query plan:
0: jdbc:phoenix:localhost:61181:/hbase> explain select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID;
+------------+
| PLAN |
+------------+
| CLIENT PARALLEL 1-WAY FULL SCAN OVER METRIC_RECORD |
| SERVER FILTER BY (SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000) |
| SERVER AGGREGATE INTO DISTINCT ROWS BY [METRIC_NAME, HOSTNAME, APP_ID] |
| CLIENT MERGE SORT |
+------------+
Scan depends on number of Regions, so it would by lets say 4 way parallel scan with 4 Regions.
Next effort is to come up with a good split policy for the precision table.
Diffs
-----
ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector bdf00b5
ambari-metrics/ambari-metrics-timelineservice/pom.xml d994d52
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java 8a37a57
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java c615804
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java d4f919e
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java 37fb088
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java 642fcfe
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java e0cb3d0
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java 462c8d9
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java ef0f4ce
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java c8b8709
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITClusterAggregator.java 13fa348
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITMetricAggregator.java b480b7a
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 06e153b
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java ef6fc58
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml 2d11412
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml c716bea
Diff: https://reviews.apache.org/r/38070/diff/
Testing
-------
All ams unit test pass.
Thanks,
Sid Wagle
Re: Review Request 38070: Optimize aggregator queries by performing
GROUP BY on server
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38070/
-----------------------------------------------------------
(Updated Sept. 2, 2015, 9:33 p.m.)
Review request for Ambari, Dmytro Sen, Mahadev Konar, Sumit Mohanty, and Srimanth Gunturi.
Bugs: AMBARI-12983
https://issues.apache.org/jira/browse/AMBARI-12983
Repository: ambari
Description
-------
Use GroupBy aggregators wherever possible.
Query plan:
0: jdbc:phoenix:localhost:61181:/hbase> explain select SUM(METRIC_SUM), SUM(METRIC_COUNT), MAX(METRIC_MAX), MIN(METRIC_MIN), METRIC_NAME, HOSTNAME, APP_ID from METRIC_RECORD WHERE SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000 GROUP BY METRIC_NAME, HOSTNAME, APP_ID;
+------------+
| PLAN |
+------------+
| CLIENT PARALLEL 1-WAY FULL SCAN OVER METRIC_RECORD |
| SERVER FILTER BY (SERVER_TIME > 1440106830000 AND SERVER_TIME < 1440106950000) |
| SERVER AGGREGATE INTO DISTINCT ROWS BY [METRIC_NAME, HOSTNAME, APP_ID] |
| CLIENT MERGE SORT |
+------------+
Diffs (updated)
-----
ambari-metrics/ambari-metrics-timelineservice/conf/unix/ambari-metrics-collector bdf00b5
ambari-metrics/ambari-metrics-timelineservice/pom.xml d994d52
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryClientService.java 8a37a57
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java c615804
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java d4f919e
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java 37fb088
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java 642fcfe
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java e0cb3d0
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java 462c8d9
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java ef0f4ce
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java c8b8709
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITClusterAggregator.java 13fa348
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITMetricAggregator.java b480b7a
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java 06e153b
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java ef6fc58
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml 2d11412
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml c716bea
Diff: https://reviews.apache.org/r/38070/diff/
Testing (updated)
-------
All ams unit test pass.
Thanks,
Sid Wagle