You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Aravindan Vijayan <av...@hortonworks.com> on 2017/03/10 23:51:21 UTC

Review Request 57523: AMBARI-20403 : Yarn Container metrics are not being collected by AMS

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57523/
-----------------------------------------------------------

Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-20403
    https://issues.apache.org/jira/browse/AMBARI-20403


Repository: ambari


Description
-------

Exception in NM logs :


    2017-03-07 19:14:06,143 INFO  timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:emitMetricsJson(197)) - Unable to connect to collector, null
    This exceptions will be ignored for next 100 times
    
    2017-03-07 19:14:06,144 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(344)) - Unable to send metrics to collector by address:null


0jdbc:phoenix:localhost:2181:/ams-hbase-uns> select * from CONTAINER_METRICS order by START_TIME desc limit 10;
+------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
|?? APP_ID?? | CONTAINER_ID | START_TIME | FINISH_TIME |??DURATION??|??HOSTNAME??| EXIT_CODE??| LOCALIZATION_DURATION | LAUNCH_DURATION | MEM_REQUESTED_GB | MEM_REQUESTED_GB_ |
+------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
+------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
No rows selected (0.505 seconds)


PROBLEM
The collector URI used for yarn container metrics is initially seeded and not refreshed based on collector availability (HA changes). A couple of other places where the Container metrics URL was wrongly invoked.

FIX
Changed container metrics path to use collector host HA changes.


Diffs
-----

  ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AbstractTimelineMetricsSink.java a1fd008 
  ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java 988f15d 
  ambari-server/src/main/java/org/apache/ambari/server/metrics/system/impl/AmbariMetricSinkImpl.java 275ad16 
  ambari-server/src/test/java/org/apache/ambari/server/metric/system/impl/TestAmbariMetricsSinkImpl.java 36d8fee 


Diff: https://reviews.apache.org/r/57523/diff/1/


Testing
-------

Manually tested.

[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 5 licence.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Metrics Common ............................. SUCCESS [7.125s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.492s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.830s
[INFO] Finished at: Fri Mar 10 14:42:08 PST 2017
[INFO] Final Memory: 35M/981M
[INFO] ------------------------------------------------------------------------


Thanks,

Aravindan Vijayan


Re: Review Request 57523: AMBARI-20403 : Yarn Container metrics are not being collected by AMS

Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57523/#review168697
-----------------------------------------------------------


Ship it!




Ship It!

- Sid Wagle


On March 10, 2017, 11:51 p.m., Aravindan Vijayan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57523/
> -----------------------------------------------------------
> 
> (Updated March 10, 2017, 11:51 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-20403
>     https://issues.apache.org/jira/browse/AMBARI-20403
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Exception in NM logs :
> 
> 
>     2017-03-07 19:14:06,143 INFO  timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:emitMetricsJson(197)) - Unable to connect to collector, null
>     This exceptions will be ignored for next 100 times
>     
>     2017-03-07 19:14:06,144 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(344)) - Unable to send metrics to collector by address:null
> 
> 
> 0jdbc:phoenix:localhost:2181:/ams-hbase-uns> select * from CONTAINER_METRICS order by START_TIME desc limit 10;
> +------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
> |?? APP_ID?? | CONTAINER_ID | START_TIME | FINISH_TIME |??DURATION??|??HOSTNAME??| EXIT_CODE??| LOCALIZATION_DURATION | LAUNCH_DURATION | MEM_REQUESTED_GB | MEM_REQUESTED_GB_ |
> +------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
> +------------+--------------+------------+-------------+------------+------------+------------+-----------------------+-----------------+------------------+-------------------+
> No rows selected (0.505 seconds)
> 
> 
> PROBLEM
> The collector URI used for yarn container metrics is initially seeded and not refreshed based on collector availability (HA changes). A couple of other places where the Container metrics URL was wrongly invoked.
> 
> FIX
> Changed container metrics path to use collector host HA changes.
> 
> 
> Diffs
> -----
> 
>   ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/AbstractTimelineMetricsSink.java a1fd008 
>   ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java 988f15d 
>   ambari-server/src/main/java/org/apache/ambari/server/metrics/system/impl/AmbariMetricSinkImpl.java 275ad16 
>   ambari-server/src/test/java/org/apache/ambari/server/metric/system/impl/TestAmbariMetricsSinkImpl.java 36d8fee 
> 
> 
> Diff: https://reviews.apache.org/r/57523/diff/1/
> 
> 
> Testing
> -------
> 
> Manually tested.
> 
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 5 licence.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Metrics Common ............................. SUCCESS [7.125s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [5.492s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 12.830s
> [INFO] Finished at: Fri Mar 10 14:42:08 PST 2017
> [INFO] Final Memory: 35M/981M
> [INFO] ------------------------------------------------------------------------
> 
> 
> Thanks,
> 
> Aravindan Vijayan
> 
>