You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by vijuks <vi...@gmail.com> on 2015/10/30 21:33:04 UTC
Using model saved by MLlib with out creating spark context
I want to load a model saved by a spark machine learning job, in a web
application.
model.save(jsc.sc(), "myModelPath");
LogisticRegressionModel model = LogisticRegressionModel.load(
jsc.sc(), "myModelPath");
When I do that, I need to pass a spark context for loading the model. The
model is small and can be saved to local file system, so is there any way to
use it with out the spark context? Looks like creating spark context is an
expensive step that starts http server that listens on multiple ports.
15/10/30 10:53:39 INFO HttpServer: Starting HTTP Server
15/10/30 10:53:39 INFO Utils: Successfully started service 'HTTP file
server' on port 63341.
15/10/30 10:53:39 INFO SparkEnv: Registering OutputCommitCoordinator
15/10/30 10:53:40 INFO Utils: Successfully started service 'SparkUI' on port
4040.
15/10/30 10:53:40 INFO SparkUI: Started SparkUI at http://171.142.49.18:4040
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-model-saved-by-MLlib-with-out-creating-spark-context-tp25239.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Using model saved by MLlib with out creating spark context
Posted by Viju K <vi...@gmail.com>.
Thank you for your suggestion, Stefano. I was hoping for an easier solution :)
Sent from my iPhone
> On Oct 30, 2015, at 2:11 PM, Stefano Baghino <st...@radicalbit.io> wrote:
>
> One possibility would be to export the model as a PMML (Predictive Model Markup Language, an XML-based standard to describe predictive models) and then use it in your Web app (using something like JPMML, for example). You can directly export (some) models (including LinReg) since Spark 1.4: https://databricks.com/blog/2015/07/02/pmml-support-in-spark-mllib.html
>
> For more info on PMML support on MLlib (including model support): https://spark.apache.org/docs/latest/mllib-pmml-model-export.html
> For more info on the PMML standard: http://dmg.org/pmml/v4-2-1/GeneralStructure.html
>
>> On Fri, Oct 30, 2015 at 9:33 PM, vijuks <vi...@gmail.com> wrote:
>> I want to load a model saved by a spark machine learning job, in a web
>> application.
>>
>> model.save(jsc.sc(), "myModelPath");
>>
>> LogisticRegressionModel model = LogisticRegressionModel.load(
>> jsc.sc(), "myModelPath");
>>
>> When I do that, I need to pass a spark context for loading the model. The
>> model is small and can be saved to local file system, so is there any way to
>> use it with out the spark context? Looks like creating spark context is an
>> expensive step that starts http server that listens on multiple ports.
>>
>> 15/10/30 10:53:39 INFO HttpServer: Starting HTTP Server
>> 15/10/30 10:53:39 INFO Utils: Successfully started service 'HTTP file
>> server' on port 63341.
>> 15/10/30 10:53:39 INFO SparkEnv: Registering OutputCommitCoordinator
>> 15/10/30 10:53:40 INFO Utils: Successfully started service 'SparkUI' on port
>> 4040.
>> 15/10/30 10:53:40 INFO SparkUI: Started SparkUI at http://171.142.49.18:4040
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-model-saved-by-MLlib-with-out-creating-spark-context-tp25239.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit