You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zhoukang (JIRA)" <ji...@apache.org> on 2019/02/18 11:41:00 UTC

[jira] [Updated] (SPARK-26914) ThriftServer scheduler pool may be unpredictably when using fair schedule mode

     [ https://issues.apache.org/jira/browse/SPARK-26914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

zhoukang updated SPARK-26914:
-----------------------------
    Description: 
When using fair scheduler mode for thrift server, we may have unpredictable result.

{code:java}
val pool = sessionToActivePool.get(parentSession.getSessionHandle)
    if (pool != null) {
      sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL, pool)
    }
{code}

Here is an example:
We have some query will use default pool, however it submit to 'normal' pool.
 !26914-02.png! 

I changed code and add some log.Got some strange result.
 !26914-01.png! 
 !26914-03.png! 

Then i found out that the localProperties of SparkContext may has unpredictable result when call setLocalProperty. And since thriftserver use thread pool to execute queries, it will trigger this bug sometimes.

{code:java}
/**
   * Set a local property that affects jobs submitted from this thread, such as the Spark fair
   * scheduler pool. User-defined properties may also be set here. These properties are propagated
   * through to worker tasks and can be accessed there via
   * [[org.apache.spark.TaskContext#getLocalProperty]].
   *
   * These properties are inherited by child threads spawned from this thread. This
   * may have unexpected consequences when working with thread pools. The standard java
   * implementation of thread pools have worker threads spawn other worker threads.
   * As a result, local properties may propagate unpredictably.
   */
  def setLocalProperty(key: String, value: String) {
    if (value == null) {
      localProperties.get.remove(key)
    } else {
      localProperties.get.setProperty(key, value)
    }
  }
{code}
   


  was:
When using fair scheduler mode for thrift server, we may have unpredictable result.

{code:java}
val pool = sessionToActivePool.get(parentSession.getSessionHandle)
    if (pool != null) {
      sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL, pool)
    }
{code}

Here is an example:
We have some query will use default pool, however it submit to 'normal' pool

I changed code and add some log.Got some strange result.

Then i found out that the localProperties of SparkContext may has unpredictable result when call setLocalProperty. And since thriftserver use thread pool to execute queries, it will trigger this bug sometimes.

{code:java}
/**
   * Set a local property that affects jobs submitted from this thread, such as the Spark fair
   * scheduler pool. User-defined properties may also be set here. These properties are propagated
   * through to worker tasks and can be accessed there via
   * [[org.apache.spark.TaskContext#getLocalProperty]].
   *
   * These properties are inherited by child threads spawned from this thread. This
   * may have unexpected consequences when working with thread pools. The standard java
   * implementation of thread pools have worker threads spawn other worker threads.
   * As a result, local properties may propagate unpredictably.
   */
  def setLocalProperty(key: String, value: String) {
    if (value == null) {
      localProperties.get.remove(key)
    } else {
      localProperties.get.setProperty(key, value)
    }
  }
{code}
   



> ThriftServer scheduler pool may be unpredictably when using fair schedule mode
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-26914
>                 URL: https://issues.apache.org/jira/browse/SPARK-26914
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: zhoukang
>            Priority: Major
>         Attachments: 26914-01.png, 26914-02.png, 26914-03.png
>
>
> When using fair scheduler mode for thrift server, we may have unpredictable result.
> {code:java}
> val pool = sessionToActivePool.get(parentSession.getSessionHandle)
>     if (pool != null) {
>       sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL, pool)
>     }
> {code}
> Here is an example:
> We have some query will use default pool, however it submit to 'normal' pool.
>  !26914-02.png! 
> I changed code and add some log.Got some strange result.
>  !26914-01.png! 
>  !26914-03.png! 
> Then i found out that the localProperties of SparkContext may has unpredictable result when call setLocalProperty. And since thriftserver use thread pool to execute queries, it will trigger this bug sometimes.
> {code:java}
> /**
>    * Set a local property that affects jobs submitted from this thread, such as the Spark fair
>    * scheduler pool. User-defined properties may also be set here. These properties are propagated
>    * through to worker tasks and can be accessed there via
>    * [[org.apache.spark.TaskContext#getLocalProperty]].
>    *
>    * These properties are inherited by child threads spawned from this thread. This
>    * may have unexpected consequences when working with thread pools. The standard java
>    * implementation of thread pools have worker threads spawn other worker threads.
>    * As a result, local properties may propagate unpredictably.
>    */
>   def setLocalProperty(key: String, value: String) {
>     if (value == null) {
>       localProperties.get.remove(key)
>     } else {
>       localProperties.get.setProperty(key, value)
>     }
>   }
> {code}
>    



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org