[GitHub] caneGuy opened a new pull request #23826: [SPARK-26914][SQL] Fix scheduler pool may be unpredictable when we only want to use default pool and do not set spark.scheduler.pool for the session

[GitBox <gi...@apache.org>]

caneGuy opened a new pull request #23826: [SPARK-26914][SQL] Fix scheduler pool may be unpredictable when we only want to use default pool and do not set spark.scheduler.pool for the session
URL: https://github.com/apache/spark/pull/23826
 
 
   ## What changes were proposed in this pull request?
   
   When using fair scheduler mode for thrift server, we may have unpredictable result.
   `val pool = sessionToActivePool.get(parentSession.getSessionHandle)
       if (pool != null) {
         sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL, pool)
       }`
   The cause is we use thread pool to execute queries for thriftserver, and when we call setLocalProperty we may have unpredictab behavior.
   `/**
      * Set a local property that affects jobs submitted from this thread, such as the Spark fair
      * scheduler pool. User-defined properties may also be set here. These properties are propagated
      * through to worker tasks and can be accessed there via
      * [[org.apache.spark.TaskContext#getLocalProperty]].
      *
      * These properties are inherited by child threads spawned from this thread. This
      * may have unexpected consequences when working with thread pools. The standard java
      * implementation of thread pools have worker threads spawn other worker threads.
      * As a result, local properties may propagate unpredictably.
      */
     def setLocalProperty(key: String, value: String) {
       if (value == null) {
         localProperties.get.remove(key)
       } else {
         localProperties.get.setProperty(key, value)
       }
     }`
   I post an example on https://jira.apache.org/jira/browse/SPARK-26914 .
   
   ## How was this patch tested?
   UT
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org