[How To] Using Spark Session in internal called classes

[Aakash Basu <aa...@gmail.com>]

Hi,

I have created my own Model Tuner class which I want to use to tune models
and return a Model object if the user expects. This Model Tuner is in a
file which I would ideally import into another file and call the class and
use it.

Outer file {from where I'd be calling the Model Tuner): I am using
SparkSession to start my Spark job and then read a file, do some basic
operations to prepare it as a Dense Vector and then send it into the Model
to tune.

While I send it into the model, I get this *error*:

*AttributeError: Cannot load _jvm from SparkContext. Is SparkContext
initialized?*


The *full traceback*:

Traceback (most recent call last):
  File "/home/aakashbasu/PycharmProjects/bb_lite/2.0
backend/model_tuning/aakash_test.py", line 9, in <module>
    from model_tuning.model_tuning_newer import ModelTuner
  File "/home/aakashbasu/PycharmProjects/bb_lite/2.0
backend/model_tuning/model_tuning.py", line 13, in <module>
    labelCol="label", predictionCol="prediction", metricName="accuracy")
  File "/usr/local/lib/python3.5/dist-packages/pyspark/__init__.py", line
105, in wrapper
    return func(self, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/evaluation.py",
line 298, in __init__
    "org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator",
self.uid)
  File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/wrapper.py", line
59, in _new_java_obj
    java_obj = _jvm()
  File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/util.py", line
44, in _jvm
    raise AttributeError("Cannot load _jvm from SparkContext. Is
SparkContext initialized?")
AttributeError: Cannot load _jvm from SparkContext. Is SparkContext
initialized?



How to ensure that the inner file also uses the same Spark Context which is
initiated by the outer file using SparkSession builder?

Any help?

Thanks,
Aakash.