[GitHub] [spark] tgravescs commented on a diff in pull request #40637: [SPARK-43002][YARN] Modify yarn client application report logging frequency to reduce noise

["tgravescs (via GitHub)" <gi...@apache.org>]

tgravescs commented on code in PR #40637:
URL: https://github.com/apache/spark/pull/40637#discussion_r1159925977


##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala:
##########
@@ -225,6 +225,18 @@ package object config extends Logging {
     .timeConf(TimeUnit.MILLISECONDS)
     .createWithDefaultString("1s")
 
+  private[spark] val REPORT_LOG_FREQUENCY = {
+    ConfigBuilder("spark.yarn.report.logging.frequency")
+      .doc("Maximum number of application reports processed " +
+        "until the next application status is logged. " +
+        "If there is a change of state, the application status will be logged " +
+        "regardless of the number of application reports processed. " +
+        "This property is dependent on the spark.yarn.report.interval")

Review Comment:
   this property isn't dependent upon it, the overall frequency is dependent upon it.  this * frequency = time when logged when no status change.



##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:
##########
@@ -1160,9 +1162,13 @@ private[spark] class Client(
               Some(msg))
         }
       val state = report.getYarnApplicationState
-
+      reportsSinceLastLog += 1
       if (logApplicationReport) {
-        logInfo(s"Application report for $appId (state: $state)")
+        if (reportsSinceLastLog >= reportsTillNextLog || lastState != state) {
+          logInfo(s"Application report for $appId (state: $state)")
+          reportsSinceLastLog = 0
+        }
+

Review Comment:
   nit remove extra newline



##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala:
##########
@@ -225,6 +225,18 @@ package object config extends Logging {
     .timeConf(TimeUnit.MILLISECONDS)
     .createWithDefaultString("1s")
 
+  private[spark] val REPORT_LOG_FREQUENCY = {
+    ConfigBuilder("spark.yarn.report.logging.frequency")
+      .doc("Maximum number of application reports processed " +
+        "until the next application status is logged. " +
+        "If there is a change of state, the application status will be logged " +
+        "regardless of the number of application reports processed. " +
+        "This property is dependent on the spark.yarn.report.interval")

Review Comment:
   there is also a sark.yarn.clientLaunchMonitorInterval that is used, that is why the interval is passed into that function.



##########
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala:
##########
@@ -225,6 +225,18 @@ package object config extends Logging {
     .timeConf(TimeUnit.MILLISECONDS)
     .createWithDefaultString("1s")
 
+  private[spark] val REPORT_LOG_FREQUENCY = {
+    ConfigBuilder("spark.yarn.report.logging.frequency")

Review Comment:
   this needs to be documented in the running_on_yarn.md



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org