You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dharmesh Kakadia (JIRA)" <ji...@apache.org> on 2019/04/23 22:28:00 UTC
[jira] [Comment Edited] (SPARK-27287) PCAModel.load() does not
honor spark configs
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824604#comment-16824604 ]
Dharmesh Kakadia edited comment on SPARK-27287 at 4/23/19 10:27 PM:
--------------------------------------------------------------------
Understood. I do not know how to "use sparkSession" when reading ML model, because PCAModel.load() for example has no way to specify/use a given sparksession.
if you think it is relatively easy to fix (copying the configs to context), I can send a patch. I would need help on that though.
was (Author: dharmesh.kakadia):
Understood. I do not know how to "use sparkSession" when reading ML model, because PCAModel.load() for example has know way to specify/use a given sparksession.
if you think it is relatively easy to fix (copying the configs to context), I can send a patch. I would need help on that though.
> PCAModel.load() does not honor spark configs
> --------------------------------------------
>
> Key: SPARK-27287
> URL: https://issues.apache.org/jira/browse/SPARK-27287
> Project: Spark
> Issue Type: Bug
> Components: ML
> Affects Versions: 2.4.0
> Reporter: Dharmesh Kakadia
> Priority: Major
>
> PCAModel.load() does not seem to be using the configurations set on the current spark session.
> Repro:
>
> The following will fail to read the data because the storage account credentials config used/propagated.
> conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
> spark = SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
> model = PCAModel.load('wasb://test@test.blob.core.windows.net/model')
>
> The following however works:
> conf.set("fs.azure.account.key.test.blob.core.windows.net","Xosad==")
> spark = SparkSession.builder.appName("dharmesh").config(conf=conf).master('spark://spark-master:7077').getOrCreate()
> blah = spark.read.json('wasb://test@test.blob.core.windows.net/somethingelse/')
> blah.show()
> model = PCAModel.load('wasb://test@test.blob.core.windows.net/model')
>
> It looks like spark.read...() does force the use of the config once and then PCAModel.load() will work correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org