You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2021/02/03 00:20:15 UTC
[GitHub] [samza] ZitingShen opened a new pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
ZitingShen opened a new pull request #1459:
URL: https://github.com/apache/samza/pull/1459
Feature: Add a metric to measure the fraction of physical memory used out of the total memory of the container. During autosizing, the container-count and size can change as traffic changes, so only monitoring the phsical-memory-mb does not provide enough information for alerting. We need a fraction or % metric to allow for easy monitoring and alerting
Changes: Add a metric to measure the fraction of physical memory used out of the total memory of the container.
Tests: unit tests & integration tests (TODO: check how to do integration tests and verify this in debug log)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] mynameborat commented on a change in pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
Posted by GitBox <gi...@apache.org>.
mynameborat commented on a change in pull request #1459:
URL: https://github.com/apache/samza/pull/1459#discussion_r569032738
##########
File path: samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala
##########
@@ -47,6 +47,7 @@ class SamzaContainerMetrics(
val diskQuotaBytes = newGauge("disk-quota-bytes", Long.MaxValue)
val executorWorkFactor = newGauge("executor-work-factor", 1.0)
val physicalMemoryMb = newGauge[Double]("physical-memory-mb", 0.0F)
+ val containerMemoryUtilization = newGauge("container-memory-utilization", 0.0F)
Review comment:
1. Can we be more specific and call this `physical-memory-utilization`?
2. Can you update the [monitoring documentation](https://github.com/apache/samza/blob/master/docs/learn/documentation/versioned/operations/monitoring.md) as well
3. Do we need to explicitly pass the metric value type here? I see we have inconsistencies above where some pass the type `Double` and some don't. Will be good to maintain consistency one way or the other.
##########
File path: samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
##########
@@ -617,13 +617,18 @@ object SamzaContainer extends Logging {
taskConfig,
clock)
+ val containerMemoryMb : Int = new ClusterManagerConfig(config).getContainerMemoryMb
+
val memoryStatisticsMonitor : SystemStatisticsMonitor = new StatisticsMonitorImpl()
memoryStatisticsMonitor.registerListener(new SystemStatisticsMonitor.Listener {
override def onUpdate(sample: SystemMemoryStatistics): Unit = {
val physicalMemoryBytes : Long = sample.getPhysicalMemoryBytes
val physicalMemoryMb : Double = physicalMemoryBytes / (1024.0 * 1024.0)
+ val memoryUtilization : Float = physicalMemoryMb.toFloat / containerMemoryMb
logger.debug("Container physical memory utilization (mb): " + physicalMemoryMb)
+ logger.debug("Container memory utilization: " + memoryUtilization)
Review comment:
nit: be explicit about physical memory utilization
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] mynameborat merged pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
Posted by GitBox <gi...@apache.org>.
mynameborat merged pull request #1459:
URL: https://github.com/apache/samza/pull/1459
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] ZitingShen commented on a change in pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
Posted by GitBox <gi...@apache.org>.
ZitingShen commented on a change in pull request #1459:
URL: https://github.com/apache/samza/pull/1459#discussion_r569041889
##########
File path: samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala
##########
@@ -47,6 +47,7 @@ class SamzaContainerMetrics(
val diskQuotaBytes = newGauge("disk-quota-bytes", Long.MaxValue)
val executorWorkFactor = newGauge("executor-work-factor", 1.0)
val physicalMemoryMb = newGauge[Double]("physical-memory-mb", 0.0F)
+ val containerMemoryUtilization = newGauge("container-memory-utilization", 0.0F)
Review comment:
1. Sure
2. Done
3. Fixed the `Double` ones to be like the others, i.e. use `Float` as the type instead
##########
File path: samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
##########
@@ -617,13 +617,18 @@ object SamzaContainer extends Logging {
taskConfig,
clock)
+ val containerMemoryMb : Int = new ClusterManagerConfig(config).getContainerMemoryMb
+
val memoryStatisticsMonitor : SystemStatisticsMonitor = new StatisticsMonitorImpl()
memoryStatisticsMonitor.registerListener(new SystemStatisticsMonitor.Listener {
override def onUpdate(sample: SystemMemoryStatistics): Unit = {
val physicalMemoryBytes : Long = sample.getPhysicalMemoryBytes
val physicalMemoryMb : Double = physicalMemoryBytes / (1024.0 * 1024.0)
+ val memoryUtilization : Float = physicalMemoryMb.toFloat / containerMemoryMb
logger.debug("Container physical memory utilization (mb): " + physicalMemoryMb)
+ logger.debug("Container memory utilization: " + memoryUtilization)
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] ZitingShen commented on pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
Posted by GitBox <gi...@apache.org>.
ZitingShen commented on pull request #1459:
URL: https://github.com/apache/samza/pull/1459#issuecomment-772908354
> @ZitingShen With travis being slow, would you mind hooking up your repository to travis-ci.com and adding the CI build report to the PR. I can then merge it.
@mynameborat It seems the CI build already passed https://travis-ci.org/github/apache/samza/builds/757470170
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] mynameborat commented on a change in pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
Posted by GitBox <gi...@apache.org>.
mynameborat commented on a change in pull request #1459:
URL: https://github.com/apache/samza/pull/1459#discussion_r569032738
##########
File path: samza-core/src/main/scala/org/apache/samza/container/SamzaContainerMetrics.scala
##########
@@ -47,6 +47,7 @@ class SamzaContainerMetrics(
val diskQuotaBytes = newGauge("disk-quota-bytes", Long.MaxValue)
val executorWorkFactor = newGauge("executor-work-factor", 1.0)
val physicalMemoryMb = newGauge[Double]("physical-memory-mb", 0.0F)
+ val containerMemoryUtilization = newGauge("container-memory-utilization", 0.0F)
Review comment:
1. Can we be more specific and call this `physical-memory-utilization`?
2. Can you update the [monitoring documentation](https://github.com/apache/samza/blob/master/docs/learn/documentation/versioned/operations/monitoring.md) as well
3. Do we need to explicitly pass the metric value type here? I see we have inconsistencies above where some pass the type `Double` and some don't. Will be good to maintain consistency one way or the other.
##########
File path: samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala
##########
@@ -617,13 +617,18 @@ object SamzaContainer extends Logging {
taskConfig,
clock)
+ val containerMemoryMb : Int = new ClusterManagerConfig(config).getContainerMemoryMb
+
val memoryStatisticsMonitor : SystemStatisticsMonitor = new StatisticsMonitorImpl()
memoryStatisticsMonitor.registerListener(new SystemStatisticsMonitor.Listener {
override def onUpdate(sample: SystemMemoryStatistics): Unit = {
val physicalMemoryBytes : Long = sample.getPhysicalMemoryBytes
val physicalMemoryMb : Double = physicalMemoryBytes / (1024.0 * 1024.0)
+ val memoryUtilization : Float = physicalMemoryMb.toFloat / containerMemoryMb
logger.debug("Container physical memory utilization (mb): " + physicalMemoryMb)
+ logger.debug("Container memory utilization: " + memoryUtilization)
Review comment:
nit: be explicit about physical memory utilization
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [samza] mynameborat commented on pull request #1459: SAMZA-2618: Add per-container metric for memory-utilization
Posted by GitBox <gi...@apache.org>.
mynameborat commented on pull request #1459:
URL: https://github.com/apache/samza/pull/1459#issuecomment-772907205
@ZitingShen With travis being slow, would you mind hooking up your repository to travis-ci.com and adding the CI build report to the PR. I can then merge it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org