You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/11 07:26:12 UTC

[GitHub] [spark] zhouyifan279 opened a new pull request #35162: [SPARK-37863] Add submitTime for Spark Application

zhouyifan279 opened a new pull request #35162:
URL: https://github.com/apache/spark/pull/35162


   ### What changes were proposed in this pull request?
   Add submitTime for Spark Application
   
   
   ### Why are the changes needed?
   Spark Driver may take a long time to startup when running with cluster mode, if YARN temporarily does not have enough resources, or HDFS's performance is poor due to high pressure.
   
   After`submitTime` is added, we can get `submitTIme` from Spark UI or Spark REST API and detect such situation by comparing it with the `spark.app.startTime`(already exists) .
   ![image](https://user-images.githubusercontent.com/88070094/148897057-be0d1367-2d97-4e3d-9d76-c1020c76e67a.png)
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   Use existing tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #35162:
URL: https://github.com/apache/spark/pull/35162#issuecomment-1009977730


   cc @Ngone51 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35162:
URL: https://github.com/apache/spark/pull/35162#discussion_r782489674



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -856,6 +856,9 @@ private[spark] class SparkSubmit extends Logging {
     if (args.verbose && isSqlShell(childMainClass)) {
       childArgs ++= Seq("--verbose")
     }
+
+    sparkConf.set("spark.app.submitTime", System.currentTimeMillis().toString)

Review comment:
       This seems specifically only for `Environment`'s `Spark Properties` table.
   
   https://github.com/apache/spark/blob/122a59986f6ab358452a293256ad45be0fe96a1c/core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala#L77-L79




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35162:
URL: https://github.com/apache/spark/pull/35162#discussion_r782491809



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -856,6 +856,9 @@ private[spark] class SparkSubmit extends Logging {
     if (args.verbose && isSqlShell(childMainClass)) {
       childArgs ++= Seq("--verbose")
     }
+
+    sparkConf.set("spark.app.submitTime", System.currentTimeMillis().toString)

Review comment:
       `spark.app.startTime` is also setting like this, but `startTime` is propagated to more classes differently because it's used in non-UI usage.
   - The same usage.
   ```
   _conf.set("spark.app.startTime", startTime.toString)
   ```
   ```
   core/src/main/scala/org/apache/spark/executor/Executor.scala: 
   private val appStartTime = conf.getLong("spark.app.startTime", 0)
   ```
   
   - The following is another usage pattern of `startTime`.
   ```
   Some(SparkUI.create(Some(this), _statusStore, _conf, _env.securityManager, appName, "", startTime))
   ```
   
   ```
   listenerBus.post(SparkListenerApplicationStart(appName, Some(applicationId),
         startTime, sparkUser, applicationAttemptId, schedulerBackend.getDriverLogUrls,
         schedulerBackend.getDriverAttributes))
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] zhouyifan279 commented on a change in pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
zhouyifan279 commented on a change in pull request #35162:
URL: https://github.com/apache/spark/pull/35162#discussion_r782657405



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -856,6 +856,9 @@ private[spark] class SparkSubmit extends Logging {
     if (args.verbose && isSqlShell(childMainClass)) {
       childArgs ++= Seq("--verbose")
     }
+
+    sparkConf.set("spark.app.submitTime", System.currentTimeMillis().toString)

Review comment:
       Yes. It is set in [SparkContext.scala#393](https://github.com/apache/spark/blob/4edc5d3263c6be782a48a667e252a27ed622b363/core/src/main/scala/org/apache/spark/SparkContext.scala#L393)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] yaooqinn commented on pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #35162:
URL: https://github.com/apache/spark/pull/35162#issuecomment-1010557092


   thanks @zhouyifan279, @dongjoon-hyun @cloud-fan , merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35162:
URL: https://github.com/apache/spark/pull/35162#discussion_r782491809



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -856,6 +856,9 @@ private[spark] class SparkSubmit extends Logging {
     if (args.verbose && isSqlShell(childMainClass)) {
       childArgs ++= Seq("--verbose")
     }
+
+    sparkConf.set("spark.app.submitTime", System.currentTimeMillis().toString)

Review comment:
       `spark.app.startTime` is also setting like this, but `startTime` is propagated to more classes differently because it's used in non-UI usage.
   ```
   _conf.set("spark.app.startTime", startTime.toString)
   ```
   
   The following is another usage pattern of `startTime`.
   ```
   Some(SparkUI.create(Some(this), _statusStore, _conf, _env.securityManager, appName, "", startTime))
   ```
   
   ```
   listenerBus.post(SparkListenerApplicationStart(appName, Some(applicationId),
         startTime, sparkUser, applicationAttemptId, schedulerBackend.getDriverLogUrls,
         schedulerBackend.getDriverAttributes))
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] yaooqinn commented on pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #35162:
URL: https://github.com/apache/spark/pull/35162#issuecomment-1009904828


   cc @cloud-fan @HyukjinKwon 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] yaooqinn closed pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #35162:
URL: https://github.com/apache/spark/pull/35162


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] yaooqinn commented on pull request #35162: [SPARK-37863] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #35162:
URL: https://github.com/apache/spark/pull/35162#issuecomment-1009675129


   https://github.com/apache/spark/pull/35162/checks?check_run_id=4772072355 please follow the instructions to renew your folk


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35162:
URL: https://github.com/apache/spark/pull/35162#issuecomment-1010386899


   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #35162:
URL: https://github.com/apache/spark/pull/35162#discussion_r782154358



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -856,6 +856,9 @@ private[spark] class SparkSubmit extends Logging {
     if (args.verbose && isSqlShell(childMainClass)) {
       childArgs ++= Seq("--verbose")
     }
+
+    sparkConf.set("spark.app.submitTime", System.currentTimeMillis().toString)

Review comment:
       do we also report the start time using this way (set a config) ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #35162: [SPARK-37863][CORE] Add submitTime for Spark Application

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #35162:
URL: https://github.com/apache/spark/pull/35162#discussion_r782491809



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -856,6 +856,9 @@ private[spark] class SparkSubmit extends Logging {
     if (args.verbose && isSqlShell(childMainClass)) {
       childArgs ++= Seq("--verbose")
     }
+
+    sparkConf.set("spark.app.submitTime", System.currentTimeMillis().toString)

Review comment:
       `spark.app.startTime` is also setting this, but `startTime` is propagated to more classes differently because it's used in non-UI usage.
   ```
   _conf.set("spark.app.startTime", startTime.toString)
   ```
   
   The following is another usage pattern of `startTime`.
   ```
   Some(SparkUI.create(Some(this), _statusStore, _conf, _env.securityManager, appName, "", startTime))
   ```
   
   ```
   listenerBus.post(SparkListenerApplicationStart(appName, Some(applicationId),
         startTime, sparkUser, applicationAttemptId, schedulerBackend.getDriverLogUrls,
         schedulerBackend.getDriverAttributes))
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org