[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #38357: [SPARK-40887][K8S] Allow Spark on K8s to integrate w/ Log Service

[GitBox <gi...@apache.org>]

dongjoon-hyun commented on code in PR #38357:
URL: https://github.com/apache/spark/pull/38357#discussion_r1051265233


##########
core/src/main/scala/org/apache/spark/scheduler/SchedulerBackend.scala:
##########
@@ -74,14 +76,24 @@ private[spark] trait SchedulerBackend {
    * Executors tab for the driver.
    * @return Map containing the log names and their respective URLs
    */
-  def getDriverLogUrls: Option[Map[String, String]] = None
+  def getDriverLogUrls: Option[Map[String, String]] = {
+    val prefix = "SPARK_DRIVER_LOG_URL_"
+    val driverLogUrls = sys.env.filterKeys(_.startsWith(prefix))
+      .map(e => (e._1.substring(prefix.length).toLowerCase(Locale.ROOT), e._2)).toMap
+    if (driverLogUrls.nonEmpty) Some(driverLogUrls) else None
+  }
 
   /**
    * Get the attributes on driver. These attributes are used to replace log URLs when
    * custom log url pattern is specified.
    * @return Map containing attributes on driver.
    */
-  def getDriverAttributes: Option[Map[String, String]] = None
+  def getDriverAttributes: Option[Map[String, String]] = {
+    val prefix = "SPARK_DRIVER_ATTRIBUTE_"
+    val driverAttributes = sys.env.filterKeys(_.startsWith(prefix))
+      .map(e => (e._1.substring(prefix.length).toUpperCase(Locale.ROOT), e._2)).toMap
+    if (driverAttributes.nonEmpty) Some(driverAttributes) else None

Review Comment:
   This looks like logically duplicated in both `getDriverLogUrls` and `getDriverAttributes` except the variable names. Could you try to refactor more?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org