You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "WangJianfei (JIRA)" <ji...@apache.org> on 2016/09/14 14:01:21 UTC
[jira] [Updated] (SPARK-17535) Performance Improvement of Signleton pattern in SparkContext

     [ https://issues.apache.org/jira/browse/SPARK-17535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

WangJianfei updated SPARK-17535:
--------------------------------
    Description: 
I think the singleton pattern of SparkContext is inefficient if there are many request to get the SparkContext,
So we can write the singleton pattern as below:
{code}
 // the current version
  def getOrCreate(): SparkContext = {
    SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
      if (activeContext.get() == null) {
        setActiveContext(new SparkContext(), allowMultipleContexts = false)
      }
      activeContext.get()
    }
  }
  // by myself
  def getOrCreate(): SparkContext = {
    if (activeContext.get() == null) {
      SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
        if(activeContext.get == null) {
          @volatile val sparkContext = new SparkContext()
          setActiveContext(sparkContext, allowMultipleContexts = false)
        }
      }
      activeContext.get()
    }
  }
{code}

  was:
I think the singleton pattern of SparkContext is inefficient if there are many request to get the SparkContext,
So we can write the singleton pattern as below:
```
 // the current version
  def getOrCreate(): SparkContext = {
    SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
      if (activeContext.get() == null) {
        setActiveContext(new SparkContext(), allowMultipleContexts = false)
      }
      activeContext.get()
    }
  }
  // by myself
  def getOrCreate(): SparkContext = {
    if (activeContext.get() == null) {
      SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
        if(activeContext.get == null) {
          @volatile val sparkContext = new SparkContext()
          setActiveContext(sparkContext, allowMultipleContexts = false)
        }
      }
      activeContext.get()
    }
  }
```



> Performance Improvement of Signleton pattern in SparkContext
> ------------------------------------------------------------
>
>                 Key: SPARK-17535
>                 URL: https://issues.apache.org/jira/browse/SPARK-17535
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.0.0
>            Reporter: WangJianfei
>              Labels: easyfix, performance
>
> I think the singleton pattern of SparkContext is inefficient if there are many request to get the SparkContext,
> So we can write the singleton pattern as below:
> {code}
>  // the current version
>   def getOrCreate(): SparkContext = {
>     SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
>       if (activeContext.get() == null) {
>         setActiveContext(new SparkContext(), allowMultipleContexts = false)
>       }
>       activeContext.get()
>     }
>   }
>   // by myself
>   def getOrCreate(): SparkContext = {
>     if (activeContext.get() == null) {
>       SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
>         if(activeContext.get == null) {
>           @volatile val sparkContext = new SparkContext()
>           setActiveContext(sparkContext, allowMultipleContexts = false)
>         }
>       }
>       activeContext.get()
>     }
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org