You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Sid Wagle <sw...@hortonworks.com> on 2016/04/15 01:54:50 UTC
Review Request 46238: Refactor Metadata manager for supporting
distributed collector
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46238/
-----------------------------------------------------------
Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.
Bugs: AMBARI-15902
https://issues.apache.org/jira/browse/AMBARI-15902
Repository: ambari
Description
-------
Presently metadata is served from a write through cache, we need a bounded cache that gets refreshed periodically to allow sharded writes to succeed.
Additionally the metadata cache needs ability to skip certain metrics, example: YARN Container metrics with too many unique names using a config.
Diffs
-----
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java 5b517ef
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java 8e58203
ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataSync.java 54ea200
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataManager.java bf887be
ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataSync.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java ddaff84
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml e5758bf
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2 6f32000
Diff: https://reviews.apache.org/r/46238/diff/
Testing
-------
All metrics Unit tests passed.
Thanks,
Sid Wagle
Re: Review Request 46238: Refactor Metadata manager for supporting
distributed collector
Posted by Aravindan Vijayan <av...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46238/#review129135
-----------------------------------------------------------
Ship it!
Ship It!
- Aravindan Vijayan
On April 14, 2016, 11:54 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46238/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 11:54 p.m.)
>
>
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15902
> https://issues.apache.org/jira/browse/AMBARI-15902
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Presently metadata is served from a write through cache, we need a bounded cache that gets refreshed periodically to allow sharded writes to succeed.
>
> Additionally the metadata cache needs ability to skip certain metrics, example: YARN Container metrics with too many unique names using a config.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java 5b517ef
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java 8e58203
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataSync.java 54ea200
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataManager.java bf887be
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataSync.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java ddaff84
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml e5758bf
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2 6f32000
>
> Diff: https://reviews.apache.org/r/46238/diff/
>
>
> Testing
> -------
>
> All metrics Unit tests passed.
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 46238: Refactor Metadata manager for supporting
distributed collector
Posted by Sid Wagle <sw...@hortonworks.com>.
> On April 15, 2016, 8:19 a.m., Dmytro Sen wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2, line 104
> > <https://reviews.apache.org/r/46238/diff/1/?file=1345966#file1345966line104>
> >
> > It'll increase number of metrics sent from sinks to ams collector, so increase the collector load
The ContainerMetrics will be syphoned off by HadoopMetricsSync into separate tables using API. This is currently scheduled for drop into 2.4.0, hence this is related change that we need to make so that the ContainerMetrics are not dropped.
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46238/#review129079
-----------------------------------------------------------
On April 14, 2016, 11:54 p.m., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46238/
> -----------------------------------------------------------
>
> (Updated April 14, 2016, 11:54 p.m.)
>
>
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15902
> https://issues.apache.org/jira/browse/AMBARI-15902
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Presently metadata is served from a write through cache, we need a bounded cache that gets refreshed periodically to allow sharded writes to succeed.
>
> Additionally the metadata cache needs ability to skip certain metrics, example: YARN Container metrics with too many unique names using a config.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java 5b517ef
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java 8e58203
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataSync.java 54ea200
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataManager.java bf887be
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataSync.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java ddaff84
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml e5758bf
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2 6f32000
>
> Diff: https://reviews.apache.org/r/46238/diff/
>
>
> Testing
> -------
>
> All metrics Unit tests passed.
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 46238: Refactor Metadata manager for supporting
distributed collector
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46238/#review129079
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2
<https://reviews.apache.org/r/46238/#comment192537>
It'll increase number of metrics sent from sinks to ams collector, so increase the collector load
- Dmytro Sen
On Апрель 14, 2016, 11:54 п.п., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46238/
> -----------------------------------------------------------
>
> (Updated Апрель 14, 2016, 11:54 п.п.)
>
>
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15902
> https://issues.apache.org/jira/browse/AMBARI-15902
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Presently metadata is served from a write through cache, we need a bounded cache that gets refreshed periodically to allow sharded writes to succeed.
>
> Additionally the metadata cache needs ability to skip certain metrics, example: YARN Container metrics with too many unique names using a config.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java 5b517ef
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java 8e58203
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataSync.java 54ea200
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataManager.java bf887be
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataSync.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java ddaff84
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml e5758bf
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2 6f32000
>
> Diff: https://reviews.apache.org/r/46238/diff/
>
>
> Testing
> -------
>
> All metrics Unit tests passed.
>
>
> Thanks,
>
> Sid Wagle
>
>
Re: Review Request 46238: Refactor Metadata manager for supporting
distributed collector
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46238/#review129138
-----------------------------------------------------------
Ship it!
Ship It!
- Dmytro Sen
On Апрель 14, 2016, 11:54 п.п., Sid Wagle wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46238/
> -----------------------------------------------------------
>
> (Updated Апрель 14, 2016, 11:54 п.п.)
>
>
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.
>
>
> Bugs: AMBARI-15902
> https://issues.apache.org/jira/browse/AMBARI-15902
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Presently metadata is served from a write through cache, we need a bounded cache that gets refreshed periodically to allow sharded writes to succeed.
>
> Additionally the metadata cache needs ability to skip certain metrics, example: YARN Container metrics with too many unique names using a config.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java 5b517ef
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataManager.java 8e58203
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TimelineMetricMetadataSync.java 54ea200
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataManager.java bf887be
> ambari-metrics/ambari-metrics-timelineservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/discovery/TestMetadataSync.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java ddaff84
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml e5758bf
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2 6f32000
>
> Diff: https://reviews.apache.org/r/46238/diff/
>
>
> Testing
> -------
>
> All metrics Unit tests passed.
>
>
> Thanks,
>
> Sid Wagle
>
>