You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Aravindan Vijayan <av...@hortonworks.com> on 2015/10/23 22:04:08 UTC
Re: Review Request 39543: AMBARI-13411 Problem in precision handling
of metrics returned by AMS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39543/
-----------------------------------------------------------
(Updated Oct. 23, 2015, 8:04 p.m.)
Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-13411
https://issues.apache.org/jira/browse/AMBARI-13411
Repository: ambari
Description
-------
PROBLEM
The AMS cache layer running on the Ambari server requests the metrics data only for the delta period (period for which the data is not present in the cache) to the AMS. The bug was that the precision of the data requested (or computed) was not taken into account during this delta request. Hence, when the AMS gets the delta time range request, it calculates the precision that might not match the precision level of the data in the cache.
SOLUTION
Compute and store precision in the cache layer. Request the data to the AMS based on the requested precision and the current precision of the data in the cache.
Same precision -> request only delta data along with precision.
Different precision -> Flush the cache and request the AMS for metrics in entire time range with precision. Store new data in the cache along with precision.
This patch also fixes bugs in handling DAYS precision level.
Added a Unit test to cover one such case.
Diffs
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/Precision.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java 52cef59
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java 06ae292
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/Precision.java eb95bdb
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java e1da289
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java 06a3d79
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java 98af2aa
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java cf4395f
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java 5bb8149
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java 969215b
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java f738f16
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITPhoenixHBaseAccessor.java 09f1584
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java 4900388
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java b21bcae
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java 4b5bfe0
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheEntryFactory.java b7b081d
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricsCacheValue.java f9f1f54
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml 996c566
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheSizingTest.java 4dbf27a
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheTest.java 32ce1e4
Diff: https://reviews.apache.org/r/39543/diff/
Testing
-------
mvn clean test
Thanks,
Aravindan Vijayan
Re: Review Request 39543: AMBARI-13411 Problem in precision handling
of metrics returned by AMS
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39543/#review103865
-----------------------------------------------------------
Ship it!
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheTest.java (line 484)
<https://reviews.apache.org/r/39543/#comment161965>
This should always work since it is a treemap but CollectionUtils would probably be a better for comparing
- Sid Wagle
On Oct. 23, 2015, 8:51 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39543/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2015, 8:51 p.m.)
>
>
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-13411
> https://issues.apache.org/jira/browse/AMBARI-13411
>
>
> Repository: ambari
>
>
> Description
> -------
>
> PROBLEM
> The AMS cache layer running on the Ambari server requests the metrics data only for the delta period (period for which the data is not present in the cache) to the AMS. The bug was that the precision of the data requested (or computed) was not taken into account during this delta request. Hence, when the AMS gets the delta time range request, it calculates the precision that might not match the precision level of the data in the cache.
>
> SOLUTION
> Compute and store precision in the cache layer. Request the data to the AMS based on the requested precision and the current precision of the data in the cache.
>
> Same precision -> request only delta data along with precision.
> Different precision -> Flush the cache and request the AMS for metrics in entire time range with precision. Store new data in the cache along with precision.
>
> This patch also fixes bugs in handling DAYS precision level.
>
> Added a Unit test to cover one such case.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/Precision.java PRE-CREATION
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java 9c0b94d
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java 9488316
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/Precision.java eb95bdb
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java e1da289
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java 06a3d79
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java 98af2aa
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java cf4395f
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java 5bb8149
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java 969215b
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java f738f16
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITPhoenixHBaseAccessor.java 09f1584
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java 4900388
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java b21bcae
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java b0aad57
> ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheEntryFactory.java b7b081d
> ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricsCacheValue.java f9f1f54
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml f2970f1
> ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheTest.java 3432e1d
>
> Diff: https://reviews.apache.org/r/39543/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Aravindan Vijayan
>
>
Re: Review Request 39543: AMBARI-13411 Problem in precision handling
of metrics returned by AMS
Posted by Aravindan Vijayan <av...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39543/
-----------------------------------------------------------
(Updated Oct. 23, 2015, 8:51 p.m.)
Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
Changes
-------
Fixing conflicts in 2.1.2 branch.
Bugs: AMBARI-13411
https://issues.apache.org/jira/browse/AMBARI-13411
Repository: ambari
Description
-------
PROBLEM
The AMS cache layer running on the Ambari server requests the metrics data only for the delta period (period for which the data is not present in the cache) to the AMS. The bug was that the precision of the data requested (or computed) was not taken into account during this delta request. Hence, when the AMS gets the delta time range request, it calculates the precision that might not match the precision level of the data in the cache.
SOLUTION
Compute and store precision in the cache layer. Request the data to the AMS based on the requested precision and the current precision of the data in the cache.
Same precision -> request only delta data along with precision.
Different precision -> Flush the cache and request the AMS for metrics in entire time range with precision. Store new data in the cache along with precision.
This patch also fixes bugs in handling DAYS precision level.
Added a Unit test to cover one such case.
Diffs (updated)
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/Precision.java PRE-CREATION
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricStore.java 9c0b94d
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java 9488316
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/Precision.java eb95bdb
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java e1da289
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/Condition.java 06a3d79
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/DefaultCondition.java 98af2aa
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/EmptyCondition.java cf4395f
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java 5bb8149
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/SplitByMetricNamesCondition.java 969215b
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/TimelineWebServices.java f738f16
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/ITPhoenixHBaseAccessor.java 09f1584
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessorTest.java 4900388
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestPhoenixTransactSQL.java b21bcae
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TestTimelineMetricStore.java b0aad57
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheEntryFactory.java b7b081d
ambari-server/src/main/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricsCacheValue.java f9f1f54
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml f2970f1
ambari-server/src/test/java/org/apache/ambari/server/controller/metrics/timeline/cache/TimelineMetricCacheTest.java 3432e1d
Diff: https://reviews.apache.org/r/39543/diff/
Testing
-------
mvn clean test
Thanks,
Aravindan Vijayan