[PYSPARK Word2Vec] Error when loading Word2Vec before calling SparkSession

[tgiordan <tg...@skapane.com>]

Hi,
I apparently found a bug with pyspark v2.3.1 when loading a Word2VecModel
before using SparkSession (for example : spark.read.parquet("df_tmp")). 
This bug append on notebook and pyspark-shell (not with spark-submit when
you need to declare 'spark =
SparkSession.builder.master("local").getOrCreate()')
Here is some code in order to reproduce the error :

#Generate Data (in a first shell):
from pyspark.ml.feature import Word2Vec
documentDF = spark.createDataFrame([
    ("Hi I heard about Spark".split(" "), ),
    ("I wish Java could use case classes".split(" "), ),
    ("Logistic regression models are neat".split(" "), )
], ["text"])

word2Vec = Word2Vec(vectorSize=3, minCount=0, inputCol="text",
outputCol="result")
model = word2Vec.fit(documentDF)
model.save("w2v_tmp")
documentDF.write.parquet("df_tmp")

----------------

#Loading Data(in a second shell):
from pyspark.ml.feature import Word2VecModel
Word2VecModel.load("w2v_tmp")
df_w2v = spark.read.parquet("df_tmp")

Throw an error : "Caused by: ERROR XSDB6: Another instance of Derby may have
already booted the database /path/to/metastore_db"


#Loading Data without error (in a second shell):
from pyspark.ml.feature import Word2VecModel
df_w2v = spark.read.parquet("df_tmp")
Word2VecModel.load("w2v_tmp")

Doesn't throw any error.

I think it's a bug and I haven't found any information about it on internet

Maybe some things need to be added when initializing pyspark-shell/notebook

Sincerely yours,
Thibault Giordan




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org