You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Aravindan Vijayan <av...@hortonworks.com> on 2017/03/10 23:51:21 UTC
Review Request 57523: AMBARI-20403 : Yarn Container metrics are not
being collected by AMS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57523/
-----------------------------------------------------------
Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
Bugs: AMBARI-20403
https://issues.apache.org/jira/browse/AMBARI-20403
Repository: ambari
Description
-------
Exception in NM logs :
2017-03-07 19:14:06,143 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:emitMetricsJson(197)) - Unable to connect to collector, null
This exceptions will be ignored for next 100 times
2017-03-07 19:14:06,144 WARN timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(344)) - Unable to send metrics to collector by address:null
0jdbc:phoenix:localhost:2181:/ams-hbase-uns> select * from CONTAINER_METRICS order by START_TIME desc limit 10;
+------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
|?? APP_ID?? | CONTAINER_ID | START_TIME | FINISH_TIME |??DURATION??|??HOSTNAME??| EXIT_CODE??| LOCALIZATION_DURATION | LAUNCH_DURATION | MEM_REQUESTED_GB | MEM_REQUESTED_GB_ |
+------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
+------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
No rows selected (0.505 seconds)
PROBLEM
The collector URI used for yarn container metrics is initially seeded and not refreshed based on collector availability (HA changes). A couple of other places where the Container metrics URL was wrongly invoked.
FIX
Changed container metrics path to use collector host HA changes.
Diffs
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AbstractTimelineMetricsSink.java a1fd008
ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java 988f15d
ambari-server/src/main/java/org/apache/ambari/server/metrics/system/impl/AmbariMetricSinkImpl.java 275ad16
ambari-server/src/test/java/org/apache/ambari/server/metric/system/impl/TestAmbariMetricsSinkImpl.java 36d8fee
Diff: https://reviews.apache.org/r/57523/diff/1/
Testing
-------
Manually tested.
[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 5 licence.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Metrics Common ............................. SUCCESS [7.125s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.492s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.830s
[INFO] Finished at: Fri Mar 10 14:42:08 PST 2017
[INFO] Final Memory: 35M/981M
[INFO] ------------------------------------------------------------------------
Thanks,
Aravindan Vijayan
Re: Review Request 57523: AMBARI-20403 : Yarn Container metrics are
not being collected by AMS
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57523/#review168697
-----------------------------------------------------------
Ship it!
Ship It!
- Sid Wagle
On March 10, 2017, 11:51 p.m., Aravindan Vijayan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57523/
> -----------------------------------------------------------
>
> (Updated March 10, 2017, 11:51 p.m.)
>
>
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
>
>
> Bugs: AMBARI-20403
> https://issues.apache.org/jira/browse/AMBARI-20403
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Exception in NM logs :
>
>
> 2017-03-07 19:14:06,143 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:emitMetricsJson(197)) - Unable to connect to collector, null
> This exceptions will be ignored for next 100 times
>
> 2017-03-07 19:14:06,144 WARN timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(344)) - Unable to send metrics to collector by address:null
>
>
> 0jdbc:phoenix:localhost:2181:/ams-hbase-uns> select * from CONTAINER_METRICS order by START_TIME desc limit 10;
> +------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
> |?? APP_ID?? | CONTAINER_ID | START_TIME | FINISH_TIME |??DURATION??|??HOSTNAME??| EXIT_CODE??| LOCALIZATION_DURATION | LAUNCH_DURATION | MEM_REQUESTED_GB | MEM_REQUESTED_GB_ |
> +------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
> +------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
> No rows selected (0.505 seconds)
>
>
> PROBLEM
> The collector URI used for yarn container metrics is initially seeded and not refreshed based on collector availability (HA changes). A couple of other places where the Container metrics URL was wrongly invoked.
>
> FIX
> Changed container metrics path to use collector host HA changes.
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AbstractTimelineMetricsSink.java a1fd008
> ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java 988f15d
> ambari-server/src/main/java/org/apache/ambari/server/metrics/system/impl/AmbariMetricSinkImpl.java 275ad16
> ambari-server/src/test/java/org/apache/ambari/server/metric/system/impl/TestAmbariMetricsSinkImpl.java 36d8fee
>
>
> Diff: https://reviews.apache.org/r/57523/diff/1/
>
>
> Testing
> -------
>
> Manually tested.
>
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 5 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Metrics Common ............................. SUCCESS [7.125s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.492s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12.830s
> [INFO] Finished at: Fri Mar 10 14:42:08 PST 2017
> [INFO] Final Memory: 35M/981M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Aravindan Vijayan
>
>