[jira] [Created] (SPARK-27287) PCAModel.load() does not honor spark configs

["Dharmesh Kakadia (JIRA)" <ji...@apache.org>]

Dharmesh Kakadia created SPARK-27287:
----------------------------------------

             Summary: PCAModel.load() does not honor spark configs
                 Key: SPARK-27287
                 URL: https://issues.apache.org/jira/browse/SPARK-27287
             Project: Spark
          Issue Type: Bug
          Components: ML
    Affects Versions: 2.4.0
            Reporter: Dharmesh Kakadia


PCAModel.load() does not seem to be using the configurations set on the current spark session. 

Repro:

 

The following will fail to read the data because the storage account credentials config used/propagated. 

conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")

spark = SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()

model = PCAModel.load('wasb://test@test.blob.core.windows.net/model')

 

The following however works:

conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")

spark = SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()

blah = spark.read.json('wasb://test@test.blob.core.windows.net/somethingelse/')

blah.show()

model = PCAModel.load('wasb://test@test.blob.core.windows.net/model')

 

It looks like spark.read...() does force the use of the config once and then PCAModel.load() will work correctly. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org