[jira] [Created] (SPARK-19369) SparkConf not getting properly initialized in PySpark 2.1.0

["Sidney Feiner (JIRA)" <ji...@apache.org>]

Sidney Feiner created SPARK-19369:
-------------------------------------

             Summary: SparkConf not getting properly initialized in PySpark 2.1.0
                 Key: SPARK-19369
                 URL: https://issues.apache.org/jira/browse/SPARK-19369
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.1.0
         Environment: Windows/Linux
            Reporter: Sidney Feiner


Trying to migrate from Spark 1.6 to 2.1, I've stumbled upon a small problem - my SparkContext doesn't get its configurations from the SparkConf object. Before passing them onto to the SparkContext constructor, I've made sure my configuration are set.

I've done some digging and this is what I've found:

When I initialize the SparkContext, the following code is executed:

def _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
             conf, jsc, profiler_cls):
    self.environment = environment or {}
    # java gateway must have been launched at this point.
    if conf is not None and conf._jconf is not None:
        # conf has been initialized in JVM properly, so use conf directly. This represent the
        # scenario that JVM has been launched before SparkConf is created (e.g. SparkContext is
        # created and then stopped, and we create a new SparkConf and new SparkContext again)
       self._conf = conf
    else:
        self._conf = SparkConf(_jvm=SparkContext._jvm)


So I can see that the only way that my SparkConf will be used is if it also has a _jvm object.
I've used spark-submit to submit my job and printed the _jvm object but it is null, which explains why my SparkConf object is ignored.
I've tried running exactly the same on Spark 2.0.1 and it worked! My SparkConf object had a valid _jvm object.

Am i doing something wrong or is this a bug?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org