You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "FMX (via GitHub)" <gi...@apache.org> on 2025/06/24 06:45:43 UTC

[PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

FMX opened a new pull request, #3346:
URL: https://github.com/apache/celeborn/pull/3346

   ### What changes were proposed in this pull request?
   1. To limit the max size of `timerMetrics` in abstractSource.
   2. Fix a concurrent issue that may cause elements to be left in the timer metrics.
   3. To fix an issue that will cause the worker to run out of memory if the metrics are not captured for a long time.
   
   
   ### Why are the changes needed?
   A long-running worker ran out of memory and found out that the metrics are huge in the heap dump.
   As you can see below, the biggest object is the time metric queue, and I got 1.6 million records.
   <img width="1516" alt="Screenshot 2025-06-24 at 09 59 30" src="https://github.com/user-attachments/assets/691c7bc2-b974-4cc0-8d5a-bf626ab903c0" />
   <img width="1239" alt="Screenshot 2025-06-24 at 14 45 10" src="https://github.com/user-attachments/assets/ebdf5a4d-c941-4f1e-911f-647aa156b37a" />
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   NO.
   
   
   ### How was this patch tested?
   Cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167729203


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Seems the loggerSink is a safety measures to help cleanup the timeMetrics, if the `getMetrics` is not called for long time.
   
   For our use case, we have a sidecar agent to collect celeborn metrics periodically, and we do not need to enable this feature.
   
   could you highlight it in the ConfigEntry docs or `metrics.properties.template`, etc?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167729203


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Seems the loggerSink is a workaround to help cleanup the timeMetrics, if the `getMetrics` is not called for long time.



##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Seems the loggerSink is a safety measures to help cleanup the timeMetrics, if the `getMetrics` is not called for long time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#issuecomment-3004117761

   > BTW, could you add the new sink into testing resource metrics.properties?
   > 
   > and provide example output for testing verification?
   
   I've added some UT.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168092297


##########
service/src/main/scala/org/apache/celeborn/common/metrics/sink/LoggerSink.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.metrics.sink
+
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.metrics.source.Source
+import org.apache.celeborn.common.util.ThreadUtils
+
+/**
+ * This sink is not follow the strandard sink interface. It has the duty to clean internal state.
+ * @param sources
+ * @param conf
+ */
+class LoggerSink(sources: Seq[Source], conf: CelebornConf) extends Sink with Logging {
+  val metricsLoggerSinkScrapeOutputEnabled = conf.metricsLoggerSinkScrapeOutputEnabled
+  val metricsLoggerSinkScrapeInterval = conf.metricsLoggerSinkScrapeInterval
+  val metricScrapeThread: ScheduledExecutorService =
+    ThreadUtils.newDaemonSingleThreadScheduledExecutor(s"metrics-scrape-thread")
+  override def start(): Unit = {

Review Comment:
   seems expected, loggerSink always invoke `source.getMetrics` to scrape metrics.
   
   metricsLoggerSinkScrapeOutputEnabled only determine whether log it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2165966597


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -224,13 +224,15 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
 
   def getAndClearTimerMetrics(): List[String] = {
     timerMetrics.synchronized {
-      var timerMetricsSize = timerMetrics.size()
-      val timerMetricsList = ArrayBuffer[String]()
-      while (timerMetricsSize > 0) {
-        timerMetricsList.append(timerMetrics.poll())
-        timerMetricsSize = timerMetricsSize - 1
+      timerMetrics.synchronized {

Review Comment:
   It's an unintentional change. I'll revert it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168231783


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   I have updated the configuration doc and migration guides.
   The logger sink is the safety valve for the metrics data. Some users enabled metrics but do not have metrics collectors. So I think we should enable the logger sink by default. And there is no side effect to enable logger sink, `getMetrics` will not remove any gauges or timers, it just removes the outdated timer outputs.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167729203


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Seems the loggerSink is a safety measures to help cleanup the timeMetrics, if the `getMetrics` is not called for long time.
   
   For our use case, we have a sidecar agent to collect celeborn metrics periodically, and we do not need to enable this feature.
   
   could you highlight it in the ConfigEntry docs or migration guide?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2165899102


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -224,13 +224,15 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
 
   def getAndClearTimerMetrics(): List[String] = {
     timerMetrics.synchronized {
-      var timerMetricsSize = timerMetrics.size()
-      val timerMetricsList = ArrayBuffer[String]()
-      while (timerMetricsSize > 0) {
-        timerMetricsList.append(timerMetrics.poll())
-        timerMetricsSize = timerMetricsSize - 1
+      timerMetrics.synchronized {

Review Comment:
   is the duplicate `timerMetrics.synchronized` by design?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2166109092


##########
service/src/main/scala/org/apache/celeborn/common/metrics/sink/LoggerSink.scala:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.metrics.sink
+
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.metrics.source.Source
+import org.apache.celeborn.common.util.ThreadUtils
+
+class LoggerSink(sources: Seq[Source], conf: CelebornConf) extends Sink with Logging {
+  private val metricsLoggerSinkScrapeOutputEnabled = conf.metricsLoggerSinkScrapeSaveEnabled
+  private val metricsLoggerSinkScrapeInterval = conf.metricsLoggerSinkScrapeInterval
+  var metricScrapeThread: ScheduledExecutorService = null
+  override def start(): Unit = {
+    metricScrapeThread =
+      ThreadUtils.newDaemonSingleThreadScheduledExecutor(s"metrics-scrape-thread")
+    metricScrapeThread.scheduleWithFixedDelay(
+      new Runnable {
+        override def run(): Unit = {
+          if (metricsLoggerSinkScrapeOutputEnabled) {
+            sources.foreach { source =>
+              logInfo(s"Source ${source.sourceName} scraped metrics: ${source.getMetrics}")
+            }
+          }
+        }
+      },
+      metricsLoggerSinkScrapeInterval,
+      metricsLoggerSinkScrapeInterval,
+      TimeUnit.MILLISECONDS)
+  }
+
+  override def stop(): Unit = {
+    if (metricScrapeThread != null) {
+      metricScrapeThread.shutdown()

Review Comment:
   ```suggestion
         ThreadUtils.shutdown(metricScrapeThread)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168299547


##########
docs/migration.md:
##########
@@ -100,9 +100,14 @@ license: |
 - Since 0.6.0, the client respects the spark.celeborn.storage.availableTypes configuration, 
     ensuring revived partition locations no longer default to memory storage. In contrast, clients prior 
     to 0.6.0 default to memory storage for revived partitions. This means that if memory storage is enabled in 
-    worker nodes, pre-0.6.0 clients may inadvertently utilize memory storage for an application even when memory 
+    worker nodes, clients prior to 0.6.0 may inadvertently utilize memory storage for an application even when memory 
     storage is not enabled for that app.
 
+- Since 0.6.0, we have added a new sink `org.apache.celeborn.common.metrics.sink.LoggerSink` to make sure that Celeborn 
+    metrics will be scraped periodically. It's recommend to enable this sink to make sure that worker's metrics data won't

Review Comment:
   ```suggestion
       metrics will be scraped periodically. It's recommended to enable this sink to make sure that worker's metrics data won't
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167283184


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Does it mean that, if we enable LoggerSink, the prometheus can not retrieve timeMetrics then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167979262


##########
service/src/main/scala/org/apache/celeborn/common/metrics/sink/LoggerSink.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.metrics.sink
+
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.metrics.source.Source
+import org.apache.celeborn.common.util.ThreadUtils
+
+/**
+ * This sink is not follow the strandard sink interface. It has the duty to clean internal state.
+ * @param sources
+ * @param conf
+ */
+class LoggerSink(sources: Seq[Source], conf: CelebornConf) extends Sink with Logging {
+  val metricsLoggerSinkScrapeOutputEnabled = conf.metricsLoggerSinkScrapeOutputEnabled
+  val metricsLoggerSinkScrapeInterval = conf.metricsLoggerSinkScrapeInterval
+  val metricScrapeThread: ScheduledExecutorService =
+    ThreadUtils.newDaemonSingleThreadScheduledExecutor(s"metrics-scrape-thread")
+  override def start(): Unit = {

Review Comment:
   Should schedule scrape-thread only when `metricsLoggerSinkScrapeOutputEnabled` is true



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167729203


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Seems the loggerSink is a safety measures to help cleanup the timeMetrics, if the `getMetrics` is not called for long time.
   
   For our use case, we have a sidecar agent to collect celeborn metrics periodically, and we do not need to enable this feature.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168301622


##########
docs/migration.md:
##########
@@ -100,9 +100,14 @@ license: |
 - Since 0.6.0, the client respects the spark.celeborn.storage.availableTypes configuration, 
     ensuring revived partition locations no longer default to memory storage. In contrast, clients prior 
     to 0.6.0 default to memory storage for revived partitions. This means that if memory storage is enabled in 
-    worker nodes, pre-0.6.0 clients may inadvertently utilize memory storage for an application even when memory 
+    worker nodes, clients prior to 0.6.0 may inadvertently utilize memory storage for an application even when memory 
     storage is not enabled for that app.
 
+- Since 0.6.0, we have added a new sink `org.apache.celeborn.common.metrics.sink.LoggerSink` to make sure that Celeborn 
+    metrics will be scraped periodically. It's recommend to enable this sink to make sure that worker's metrics data won't
+    be too large to cause worker OOM if you don't have a collector to scrape metrics periodically. Don't forget to update 
+    the `metrics.conf`.

Review Comment:
   ```suggestion
       the `celeborn.metrics.loggerSink.scrape.interval` and `celeborn.metrics.loggerSink.output.enabled`.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167283184


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Does it mean that, if we enable LoggerSink, the prometheus can not retrieve timeMetrics then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#issuecomment-3002721766

   ## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3346?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `82.92683%` with `7 lines` in your changes missing coverage. Please review.
   > Project coverage is 63.60%. Comparing base [(`676beca`)](https://app.codecov.io/gh/apache/celeborn/commit/676beca616c8b5cb0e8ba72450d92fa9bc9cca42?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`737d20e`)](https://app.codecov.io/gh/apache/celeborn/commit/737d20e4248985fa2cee18a4ed00138c2e677bd2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 2 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/celeborn/pull/3346?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...eleborn/common/metrics/source/AbstractSource.scala](https://app.codecov.io/gh/apache/celeborn/pull/3346?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fceleborn%2Fcommon%2Fmetrics%2Fsource%2FAbstractSource.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvY2VsZWJvcm4vY29tbW9uL21ldHJpY3Mvc291cmNlL0Fic3RyYWN0U291cmNlLnNjYWxh) | 63.16% | [4 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/celeborn/pull/3346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #3346      +/-   ##
   ==========================================
   + Coverage   63.56%   63.60%   +0.05%     
   ==========================================
     Files         348      348              
     Lines       21300    21335      +35     
     Branches     1879     1882       +3     
   ==========================================
   + Hits        13537    13569      +32     
     Misses       6784     6784              
   - Partials      979      982       +3     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/3346?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2165259173


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -224,11 +227,11 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
 
   def getAndClearTimerMetrics(): List[String] = {
     timerMetrics.synchronized {
-      var timerMetricsSize = timerMetrics.size()
       val timerMetricsList = ArrayBuffer[String]()
-      while (timerMetricsSize > 0) {
-        timerMetricsList.append(timerMetrics.poll())
-        timerMetricsSize = timerMetricsSize - 1
+      var elem = timerMetrics.poll()
+      while (elem != null) {

Review Comment:
   IMO the while loop can be terminated when `timerMetrics.size() > 0 && elem !=null`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Fix OOM caused by worker source if the worker's metrics is not captured for a long time [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#issuecomment-3003650912

   Gently ping @SteNicholas @turboFei
   I have updated this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168225954


##########
service/src/main/scala/org/apache/celeborn/common/metrics/sink/LoggerSink.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.metrics.sink
+
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.metrics.source.Source
+import org.apache.celeborn.common.util.ThreadUtils
+
+/**
+ * This sink is not follow the strandard sink interface. It has the duty to clean internal state.
+ * @param sources
+ * @param conf
+ */
+class LoggerSink(sources: Seq[Source], conf: CelebornConf) extends Sink with Logging {
+  val metricsLoggerSinkScrapeOutputEnabled = conf.metricsLoggerSinkScrapeOutputEnabled
+  val metricsLoggerSinkScrapeInterval = conf.metricsLoggerSinkScrapeInterval
+  val metricScrapeThread: ScheduledExecutorService =
+    ThreadUtils.newDaemonSingleThreadScheduledExecutor(s"metrics-scrape-thread")
+  override def start(): Unit = {

Review Comment:
   If you have enabled the logger sink, it will need to create a thread, regardless of whether `metricsLoggerSinkScrapeOutputEnabled` is true or not. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168305136


##########
conf/metrics.properties.template:
##########
@@ -17,3 +17,4 @@
 
 *.sink.prometheusServlet.class=org.apache.celeborn.common.metrics.sink.PrometheusServlet
 *.sink.jsonServlet.class=org.apache.celeborn.common.metrics.sink.JsonServlet
+*.sink.loggerSink.class=org.apache.celeborn.common.metrics.sink.LoggerSink

Review Comment:
   Please also update the `monitoring.md` for `LoggerSink`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167282435


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   <img width="806" alt="image" src="https://github.com/user-attachments/assets/7bbb95e2-544f-4ce4-89d5-833e132105c1" />
   
   I saw that, the `AbstractServlet`(Prometheus or Json) will call the `getMetricsSnapshot` and underlying `getMetrics` method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167286598


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   So, if we enable PrometheusServlet and the metrics are collected periodically, we won't meet the OOM issue caused by timeMetrics?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168092297


##########
service/src/main/scala/org/apache/celeborn/common/metrics/sink/LoggerSink.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.metrics.sink
+
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.metrics.source.Source
+import org.apache.celeborn.common.util.ThreadUtils
+
+/**
+ * This sink is not follow the strandard sink interface. It has the duty to clean internal state.
+ * @param sources
+ * @param conf
+ */
+class LoggerSink(sources: Seq[Source], conf: CelebornConf) extends Sink with Logging {
+  val metricsLoggerSinkScrapeOutputEnabled = conf.metricsLoggerSinkScrapeOutputEnabled
+  val metricsLoggerSinkScrapeInterval = conf.metricsLoggerSinkScrapeInterval
+  val metricScrapeThread: ScheduledExecutorService =
+    ThreadUtils.newDaemonSingleThreadScheduledExecutor(s"metrics-scrape-thread")
+  override def start(): Unit = {

Review Comment:
   seems expected, loggerSink always invoke `source.getMetrics` to scrape.
   
   metricsLoggerSinkScrapeOutputEnabled only determine whether log it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2167729203


##########
common/src/main/scala/org/apache/celeborn/common/metrics/source/AbstractSource.scala:
##########
@@ -627,6 +627,8 @@ abstract class AbstractSource(conf: CelebornConf, role: String)
     sum
   }
 
+  // Do use this method to get metrics, because it will clear the timeMetrics queue
+  // Do not use the LogReporter to report metrics

Review Comment:
   Seems the loggerSink is a safety measures to help cleanup the timeMetrics, if the `getMetrics` is not called for long time.
   
   For our use case, we have a sidecar agent to collect celeborn metrics periodically, and seems not necessary to enable this feature.
   
   could you highlight it in the ConfigEntry docs or migration guide?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add logger sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on code in PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#discussion_r2168092297


##########
service/src/main/scala/org/apache/celeborn/common/metrics/sink/LoggerSink.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.celeborn.common.metrics.sink
+
+import java.util.concurrent.{ScheduledExecutorService, TimeUnit}
+
+import org.apache.celeborn.common.CelebornConf
+import org.apache.celeborn.common.internal.Logging
+import org.apache.celeborn.common.metrics.source.Source
+import org.apache.celeborn.common.util.ThreadUtils
+
+/**
+ * This sink is not follow the strandard sink interface. It has the duty to clean internal state.
+ * @param sources
+ * @param conf
+ */
+class LoggerSink(sources: Seq[Source], conf: CelebornConf) extends Sink with Logging {
+  val metricsLoggerSinkScrapeOutputEnabled = conf.metricsLoggerSinkScrapeOutputEnabled
+  val metricsLoggerSinkScrapeInterval = conf.metricsLoggerSinkScrapeInterval
+  val metricScrapeThread: ScheduledExecutorService =
+    ThreadUtils.newDaemonSingleThreadScheduledExecutor(s"metrics-scrape-thread")
+  override def start(): Unit = {

Review Comment:
   seems expected, loggerSink always invoke `source.getMetrics` to scrape metrics.
   
   metricsLoggerSinkScrapeOutputEnabled only determine whether to log metrics.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2045] Add user sinks to allow persist metrics data and avoid possible worker OOM [celeborn]

Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #3346:
URL: https://github.com/apache/celeborn/pull/3346#issuecomment-3003985223

   BTW, could you add the new sink into testing metrics template?
   
   and provide example output for testing verification?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org