You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2016/02/24 19:17:48 UTC
Review Request 43955: OOM on TimelineMetricCache in Nodemanager
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/
-----------------------------------------------------------
Review request for Ambari, Aravindan Vijayan and Sid Wagle.
Bugs: AMBARI-15100
https://issues.apache.org/jira/browse/AMBARI-15100
Repository: ambari
Description
-------
Reported on ambari user group:
2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
Diffs
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 91c63b9
ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 3f139da
Diff: https://reviews.apache.org/r/43955/diff/
Testing
-------
Unit tests passed
Thanks,
Dmytro Sen
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Sid Wagle <sw...@hortonworks.com>.
> On Feb. 26, 2016, 8:03 a.m., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java, line 74
> > <https://reviews.apache.org/r/43955/diff/3-4/?file=1270988#file1270988line74>
> >
> > What extra locking if using ConcurrentMap, is this necessary?
>
> Dmytro Sen wrote:
> TimelineMetricHolder extends ConcurrentSkipListMap<String, TimelineMetricWrapper>, so it's thread-safe only for read/assign TimelineMetricWrapper by metric name, but not for metric values wrapped by TimelineMetricWrapper. Multiple threads are still allowed to get reference to the same TimelineMetricWrapper and simultaneously call TimelineMetricWrapper.putMetric() or (TimelineMetricWrapper.putMetric() + TimelineMetricWrapper.getTimelineMetric())
>
> Dmytro Sen wrote:
> TimelineMetricHolder is thread-safe by ConcurrentSkipListMap and TimelineMetricWrapper is thread-safe by using synchronized methods
+1
- Sid
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120846
-----------------------------------------------------------
On Feb. 26, 2016, 6:55 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2016, 6:55 a.m.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Фев. 26, 2016, 8:03 д.п., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java, line 74
> > <https://reviews.apache.org/r/43955/diff/3-4/?file=1270988#file1270988line74>
> >
> > What extra locking if using ConcurrentMap, is this necessary?
>
> Dmytro Sen wrote:
> TimelineMetricHolder extends ConcurrentSkipListMap<String, TimelineMetricWrapper>, so it's thread-safe only for read/assign TimelineMetricWrapper by metric name, but not for metric values wrapped by TimelineMetricWrapper. Multiple threads are still allowed to get reference to the same TimelineMetricWrapper and simultaneously call TimelineMetricWrapper.putMetric() or (TimelineMetricWrapper.putMetric() + TimelineMetricWrapper.getTimelineMetric())
TimelineMetricHolder is thread-safe by ConcurrentSkipListMap and TimelineMetricWrapper is thread-safe by using synchronized methods
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120846
-----------------------------------------------------------
On Фев. 26, 2016, 6:55 д.п., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Фев. 26, 2016, 6:55 д.п.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Фев. 26, 2016, 8:03 д.п., Sid Wagle wrote:
> > ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java, line 74
> > <https://reviews.apache.org/r/43955/diff/3-4/?file=1270988#file1270988line74>
> >
> > What extra locking if using ConcurrentMap, is this necessary?
TimelineMetricHolder extends ConcurrentSkipListMap<String, TimelineMetricWrapper>, so it's thread-safe only for read/assign TimelineMetricWrapper by metric name, but not for metric values wrapped by TimelineMetricWrapper. Multiple threads are still allowed to get reference to the same TimelineMetricWrapper and simultaneously call TimelineMetricWrapper.putMetric() or (TimelineMetricWrapper.putMetric() + TimelineMetricWrapper.getTimelineMetric())
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120846
-----------------------------------------------------------
On Фев. 26, 2016, 6:55 д.п., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Фев. 26, 2016, 6:55 д.п.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120846
-----------------------------------------------------------
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java (line 73)
<https://reviews.apache.org/r/43955/#comment182362>
What extra locking if using ConcurrentMap, is this necessary?
- Sid Wagle
On Feb. 26, 2016, 6:55 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2016, 6:55 a.m.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/
-----------------------------------------------------------
(Updated Фев. 26, 2016, 6:55 д.п.)
Review request for Ambari, Aravindan Vijayan and Sid Wagle.
Bugs: AMBARI-15100
https://issues.apache.org/jira/browse/AMBARI-15100
Repository: ambari
Description
-------
Reported on ambari user group:
2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
Diffs (updated)
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
Diff: https://reviews.apache.org/r/43955/diff/
Testing
-------
Unit tests passed
mvn clean test
[INFO] Reactor Summary:
[INFO]
[INFO] ambari-metrics .................................... SUCCESS [0.845s]
[INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
Thanks,
Dmytro Sen
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120722
-----------------------------------------------------------
Fix it, then Ship it!
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java (line 107)
<https://reviews.apache.org/r/43955/#comment182193>
Can we synchrnoze access to the collection to avoid CME or use a java.util.concurrent collection, I believe ConcurrentSkipListMap is equivalent of a TreeMap.
I would think seconds choice will be more performant.
- Sid Wagle
On Feb. 25, 2016, 3:08 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Feb. 25, 2016, 3:08 p.m.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
> ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/
-----------------------------------------------------------
(Updated Фев. 25, 2016, 3:08 п.п.)
Review request for Ambari, Aravindan Vijayan and Sid Wagle.
Bugs: AMBARI-15100
https://issues.apache.org/jira/browse/AMBARI-15100
Repository: ambari
Description
-------
Reported on ambari user group:
2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
Diffs (updated)
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
ambari-metrics/ambari-metrics-common/src/test/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCacheTest.java ad98525
Diff: https://reviews.apache.org/r/43955/diff/
Testing
-------
Unit tests passed
mvn clean test
[INFO] Reactor Summary:
[INFO]
[INFO] ambari-metrics .................................... SUCCESS [0.845s]
[INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
Thanks,
Dmytro Sen
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Фев. 24, 2016, 10:25 п.п., Sid Wagle wrote:
> > Patch looks good for OOM.
> > Did you look into possiblity of java.util.ConcurrentModificationException
> > From the Jira it seems we might have a problem on update.
In this patch I don't modify any collection, just re-assign - this.timelineMetric.setMetricValues(metricsSubSet);
But in general, yes. It's better to use some thread-safe collection for cache implementation instead of LinkedHashMap or Guava cache
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120591
-----------------------------------------------------------
On Фев. 24, 2016, 6:24 п.п., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Фев. 24, 2016, 6:24 п.п.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/#review120591
-----------------------------------------------------------
Patch looks good for OOM.
Did you look into possiblity of java.util.ConcurrentModificationException
>From the Jira it seems we might have a problem on update.
- Sid Wagle
On Feb. 24, 2016, 6:24 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43955/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2016, 6:24 p.m.)
>
>
> Review request for Ambari, Aravindan Vijayan and Sid Wagle.
>
>
> Bugs: AMBARI-15100
> https://issues.apache.org/jira/browse/AMBARI-15100
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Reported on ambari user group:
>
> 2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
> 2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:703)
> at java.util.HashMap.putVal(HashMap.java:662)
> at java.util.HashMap.put(HashMap.java:611)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
> at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
> at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
> at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
> at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
> 2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
>
>
> Diffs
> -----
>
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
>
> Diff: https://reviews.apache.org/r/43955/diff/
>
>
> Testing
> -------
>
> Unit tests passed
>
> mvn clean test
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] ambari-metrics .................................... SUCCESS [0.845s]
> [INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
> [INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
> [INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
> [INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
> [INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
> [INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
> [INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
> [INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
> [INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/
-----------------------------------------------------------
(Updated Фев. 24, 2016, 6:24 п.п.)
Review request for Ambari, Aravindan Vijayan and Sid Wagle.
Bugs: AMBARI-15100
https://issues.apache.org/jira/browse/AMBARI-15100
Repository: ambari
Description
-------
Reported on ambari user group:
2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
Diffs (updated)
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
Diff: https://reviews.apache.org/r/43955/diff/
Testing
-------
Unit tests passed
mvn clean test
[INFO] Reactor Summary:
[INFO]
[INFO] ambari-metrics .................................... SUCCESS [0.845s]
[INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
Thanks,
Dmytro Sen
Re: Review Request 43955: OOM on TimelineMetricCache in Nodemanager
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43955/
-----------------------------------------------------------
(Updated Фев. 24, 2016, 6:22 п.п.)
Review request for Ambari, Aravindan Vijayan and Sid Wagle.
Bugs: AMBARI-15100
https://issues.apache.org/jira/browse/AMBARI-15100
Repository: ambari
Description
-------
Reported on ambari user group:
2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[timeline,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
at org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
at org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
at org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
at org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt with status -1 Message: HaltException
Diffs
-----
ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsCache.java 15bd5f4
ambari-metrics/ambari-metrics-storm-sink/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSink.java 91c63b9
ambari-metrics/ambari-metrics-storm-sink/src/test/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsSinkTest.java 3f139da
Diff: https://reviews.apache.org/r/43955/diff/
Testing (updated)
-------
Unit tests passed
mvn clean test
[INFO] Reactor Summary:
[INFO]
[INFO] ambari-metrics .................................... SUCCESS [0.845s]
[INFO] Ambari Metrics Common ............................. SUCCESS [2.570s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [4.165s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [2.147s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [2.900s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [0.472s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [4:40.041s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.116s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [6.003s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [34.107s]
Thanks,
Dmytro Sen