[GitHub] [incubator-livy] jerryshao commented on a change in pull request #242: [LIVY-336][SERVER] Livy should not spawn one thread per job to track the job on Yarn

[GitBox <gi...@apache.org>]

jerryshao commented on a change in pull request #242: [LIVY-336][SERVER] Livy should not spawn one thread per job to track the job on Yarn
URL: https://github.com/apache/incubator-livy/pull/242#discussion_r352973535
 
 

 ##########
 File path: server/src/main/scala/org/apache/livy/LivyConf.scala
 ##########
 @@ -216,11 +213,27 @@ object LivyConf {
   // RSC related jars separated with comma.
   val RSC_JARS = Entry("livy.rsc.jars", null)
 
-  // How long to check livy session leakage
+  // How long to check livy session leakage.
   val YARN_APP_LEAKAGE_CHECK_TIMEOUT = Entry("livy.server.yarn.app-leakage.check-timeout", "600s")
-  // how often to check livy session leakage
+  // How often to check livy session leakage.
   val YARN_APP_LEAKAGE_CHECK_INTERVAL = Entry("livy.server.yarn.app-leakage.check-interval", "60s")
 
+  // The size of thread pool to monitor all yarn apps.
+  val YARN_APP_MONITOR_THREAD_POOL_SIZE =
+    Entry("livy.server.yarn.app-monitor.thread-pool.size", 4)
+  // How often thread monitor the YARN app.
+  val YARN_APP_MONITOR_THREAD_INTERVAL = Entry("livy.server.yarn.app-monitor.thread-interval", "5s")
+  // How often to check monitor thread block
+  val YARN_APP_MONITOR_THREAD_BLOCK_CHECK_INTERVAL =
+    Entry("livy.server.yarn.app-monitor.thread-block.check-interval", "10s")
+  // If some thread cost more than this config to monitor one app,
+  // stop the monitor of the app which considered blocked.
+  val YARN_APP_MONITOR_THREAD_BLOCK_TIMEOUT =
+    Entry("livy.server.yarn.app-monitor.thread-block-timeout", "60s")
+  // If Livy can't monitor the yarn app successfully within this max times, consider the app failed.
+  val YARN_APP_MONITOR_MAX_FAILED_TIMES =
+    Entry("livy.server.yarn.app-monitor.max-failed.times", 12)
 
 Review comment:
   Can you please simplify the configurations, there're some many related configurations, which makes user hard to configure.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services