You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:25:33 UTC
[jira] [Updated] (SPARK-18469) Cannot make MLlib model predictions
in Spark streaming with checkpointing
[ https://issues.apache.org/jira/browse/SPARK-18469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-18469:
---------------------------------
Labels: bulk-closed (was: )
> Cannot make MLlib model predictions in Spark streaming with checkpointing
> -------------------------------------------------------------------------
>
> Key: SPARK-18469
> URL: https://issues.apache.org/jira/browse/SPARK-18469
> Project: Spark
> Issue Type: Bug
> Components: DStreams, MLlib
> Affects Versions: 1.6.2
> Reporter: Alex Jarman
> Priority: Major
> Labels: bulk-closed
>
> Enabling checkpointing whilst trying to produce predictions with an offline MLlib model in Spark Streaming throws up the following error:
> "Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063."
> The line in my code that appears to cause this is when calling the transform transformation on the DStream with model.predictAll i.e. :
> predictions = ratings.map(lambda r: (int(r[1]),int(r[2]))).transform(lambda rdd: model.predictAll(rdd)).map(lambda r: (r[0], r[1], r[2]))
> See http://stackoverflow.com/questions/40566492/spark-streaming-model-predictions-with-checkpointing-reference-sparkcontext-fr for a fuller description..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org