[jira] [Comment Edited] (TINKERPOP3-911) Allow setting Thread Specific Spark JobGroup/Custom Properties based on hadoop conf

["Russell Alexander Spitzer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TINKERPOP3-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970011#comment-14970011 ] 

Russell Alexander Spitzer edited comment on TINKERPOP3-911 at 10/22/15 10:26 PM:
---------------------------------------------------------------------------------

Properties that you may want to change on a per thread basis
{code}
  private[spark] val SPARK_JOB_DESCRIPTION = "spark.job.description"
  private[spark] val SPARK_JOB_GROUP_ID = "spark.jobGroup.id"
  private[spark] val SPARK_JOB_INTERRUPT_ON_CANCEL = "spark.job.interruptOnCancel"
  private[spark] val RDD_SCOPE_KEY = "spark.rdd.scope"
  private[spark] val RDD_SCOPE_NO_OVERRIDE_KEY = "spark.rdd.scope.noOverride"
{code}

And 
{code}
"spark.scheduler.pool"
{code}



was (Author: rspitzer):
Properties that you may want to change on a per thread basis
{code}
  private[spark] val SPARK_JOB_DESCRIPTION = "spark.job.description"
  private[spark] val SPARK_JOB_GROUP_ID = "spark.jobGroup.id"
  private[spark] val SPARK_JOB_INTERRUPT_ON_CANCEL = "spark.job.interruptOnCancel"
  private[spark] val RDD_SCOPE_KEY = "spark.rdd.scope"
  private[spark] val RDD_SCOPE_NO_OVERRIDE_KEY = "spark.rdd.scope.noOverride"
{code}


> Allow setting Thread Specific Spark JobGroup/Custom Properties based on hadoop conf
> -----------------------------------------------------------------------------------
>
>                 Key: TINKERPOP3-911
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP3-911
>             Project: TinkerPop 3
>          Issue Type: Improvement
>          Components: hadoop
>            Reporter: Russell Alexander Spitzer
>            Assignee: Marko A. Rodriguez
>
> When using a Persistant Spark context it can be beneficial to pass in new configuration options for new users/ GraphComputers. Currently the .getOrCreate call will always use the configuration from the initial construction. To work around this we should iterate over all of the properties passed into the graph computer and set them as local context properties on the thread we are operating on.
> See
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L630-L640
> This would let different graph computers set different spark properties for use with things like the Spark Fair Scheduler. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)