[jira] [Created] (SPARK-19307) SPARK-17387 caused ignorance of conf object passed to SparkContext:

["yuriy_hupalo (JIRA)" <ji...@apache.org>]

yuriy_hupalo created SPARK-19307:
------------------------------------

             Summary: SPARK-17387 caused ignorance of conf object passed to SparkContext:
                 Key: SPARK-19307
                 URL: https://issues.apache.org/jira/browse/SPARK-19307
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.1.0
            Reporter: yuriy_hupalo


after patch SPARK-17387 was applied -- Sparkconf object is ignored when launching SparkContext programmatically via python from spark-submit:

https://github.com/apache/spark/blob/master/python/pyspark/context.py#L128:

in case when we are running python SparkContext(conf=xxx) from spark-submit:
    conf is set, conf._jconf is None ()

    passed as arg  conf object is ignored (and used only when we are launching java_gateway).

how to fix:
python/pyspark/context.py:132
===================
        if conf is not None and conf._jconf is not None:
            # conf has been initialized in JVM properly, so use conf directly. This represent the
            # scenario that JVM has been launched before SparkConf is created (e.g. SparkContext is
            # created and then stopped, and we create a new SparkConf and new SparkContext again)
            self._conf = conf
        else:
            self._conf = SparkConf(_jvm=SparkContext._jvm)
+             if conf:
+                 for key, value in conf.getAll():
+                     self._conf.set(key,value)
+                     print(key,value)
===================





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org