You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by bowen zhang <bo...@yahoo.com.INVALID> on 2015/11/22 00:38:30 UTC

Using spark MLlib without installing Spark

Hi folks,I am a big fan of Spark's Mllib package. I have a java web app where I want to run some ml jobs inside the web app. My question is: is there a way to just import spark-core and spark-mllib jars to invoke my ML jobs without installing the entire Spark package? All the tutorials related Spark seems to indicate installing Spark is a pre-condition for this.
Thanks,Bowen

Re: Using spark MLlib without installing Spark

Posted by Stavros Kontopoulos <st...@typesafe.com>.
You can even use it without spark as well (besides local). For example i
have used the following algo in some web app:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala

Essentially some algorithms (i havent checked them all) they will have to
run the same steps in each partition so if you overlook the distributed
oriented parts (spark specific) of the code there is a lot of resuable
stuff.
You have just to use the api where that is public and conform to the
input/output contract of it.

There used to be some dependencies like Breeze for example in the api
hidden now (eg.
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala),
but of course this is a hint not a list of what is available for your use
case. Mind though that this may not be the cleanest way to implement your
use case or might sound a hack ;)

As an alternative choice besides spark local you could use a job server (
https://github.com/spark-jobserver/spark-jobserver) to integrate with your
app as a proxy for spark or have a spark service there to respond back with
results. From a design point of you its best to separate the concerns for
several reasons: scaling , utilization etc.

On Sun, Nov 22, 2015 at 5:03 AM, Reynold Xin <rx...@databricks.com> wrote:

> You can use MLlib and Spark directly without "installing anything". Just
> run Spark in local mode.
>
>
> On Sat, Nov 21, 2015 at 4:05 PM, Rad Gruchalski <ra...@gruchalski.com>
> wrote:
>
>> Bowen,
>>
>> What Andy is doing in the notebook is a slightly different thing. He’s
>> using sbt to bring all spark jars (core, mllib, repl, what have you). You
>> could use maven for that. He then creates a repl and submits all the spark
>> code into it.
>> Pretty sure spark unit tests cover similar uses cases. Maybe not mllib
>> per se but this kind of submission.
>>
>> Kind regards,
>> Radek Gruchalski
>> radek@gruchalski.com <ra...@gruchalski.com>
>> de.linkedin.com/in/radgruchalski/
>>
>>
>> *Confidentiality:*This communication is intended for the above-named
>> person and may be confidential and/or legally privileged.
>> If it has come to you in error you must take no action based on it, nor
>> must you copy or show it to anyone; please delete/destroy and inform the
>> sender immediately.
>>
>> On Sunday, 22 November 2015 at 01:01, bowen zhang wrote:
>>
>> Thanks Rad for info. I looked into the repo and see some .snb file using
>> spark mllib. Can you give me a more specific place to look for when
>> invoking the mllib functions? What if I just want to invoke some of the ML
>> functions in my HelloWorld.java?
>>
>> ------------------------------
>> *From:* Rad Gruchalski <ra...@gruchalski.com>
>> *To:* bowen zhang <bo...@yahoo.com>
>> *Cc:* "dev@spark.apache.org" <de...@spark.apache.org>
>> *Sent:* Saturday, November 21, 2015 3:43 PM
>> *Subject:* Re: Using spark MLlib without installing Spark
>>
>> Bowen,
>>
>> One project to look at could be spark-notebook:
>> https://github.com/andypetrella/spark-notebook
>> It uses Spark you in the way you intend to use it.
>> Kind regards,
>> Radek Gruchalski
>> radek@gruchalski.com <ra...@gruchalski.com>
>> de.linkedin.com/in/radgruchalski/
>>
>>
>> *Confidentiality:*This communication is intended for the above-named
>> person and may be confidential and/or legally privileged.
>> If it has come to you in error you must take no action based on it, nor
>> must you copy or show it to anyone; please delete/destroy and inform the
>> sender immediately.
>>
>>
>> On Sunday, 22 November 2015 at 00:38, bowen zhang wrote:
>>
>> Hi folks,
>> I am a big fan of Spark's Mllib package. I have a java web app where I
>> want to run some ml jobs inside the web app. My question is: is there a way
>> to just import spark-core and spark-mllib jars to invoke my ML jobs without
>> installing the entire Spark package? All the tutorials related Spark seems
>> to indicate installing Spark is a pre-condition for this.
>>
>> Thanks,
>> Bowen
>>
>>
>>
>>
>>
>>
>


-- 

Stavros Kontopoulos

<http://www.typesafe.com>


<http://www.typesafe.com>

Re: Using spark MLlib without installing Spark

Posted by Debasish Das <de...@gmail.com>.
Decoupling mlllib and core is difficult...it is not intended to run spark
core 1.5 with spark mllib 1.6 snapshot...core is more stabilized due to new
algorithms getting added to mllib and sometimes you might be tempted to do
that but its not recommend.
On Nov 21, 2015 8:04 PM, "Reynold Xin" <rx...@databricks.com> wrote:

> You can use MLlib and Spark directly without "installing anything". Just
> run Spark in local mode.
>
>
> On Sat, Nov 21, 2015 at 4:05 PM, Rad Gruchalski <ra...@gruchalski.com>
> wrote:
>
>> Bowen,
>>
>> What Andy is doing in the notebook is a slightly different thing. He’s
>> using sbt to bring all spark jars (core, mllib, repl, what have you). You
>> could use maven for that. He then creates a repl and submits all the spark
>> code into it.
>> Pretty sure spark unit tests cover similar uses cases. Maybe not mllib
>> per se but this kind of submission.
>>
>> Kind regards,
>> Radek Gruchalski
>> radek@gruchalski.com <ra...@gruchalski.com>
>> de.linkedin.com/in/radgruchalski/
>>
>>
>> *Confidentiality:*This communication is intended for the above-named
>> person and may be confidential and/or legally privileged.
>> If it has come to you in error you must take no action based on it, nor
>> must you copy or show it to anyone; please delete/destroy and inform the
>> sender immediately.
>>
>> On Sunday, 22 November 2015 at 01:01, bowen zhang wrote:
>>
>> Thanks Rad for info. I looked into the repo and see some .snb file using
>> spark mllib. Can you give me a more specific place to look for when
>> invoking the mllib functions? What if I just want to invoke some of the ML
>> functions in my HelloWorld.java?
>>
>> ------------------------------
>> *From:* Rad Gruchalski <ra...@gruchalski.com>
>> *To:* bowen zhang <bo...@yahoo.com>
>> *Cc:* "dev@spark.apache.org" <de...@spark.apache.org>
>> *Sent:* Saturday, November 21, 2015 3:43 PM
>> *Subject:* Re: Using spark MLlib without installing Spark
>>
>> Bowen,
>>
>> One project to look at could be spark-notebook:
>> https://github.com/andypetrella/spark-notebook
>> It uses Spark you in the way you intend to use it.
>> Kind regards,
>> Radek Gruchalski
>> radek@gruchalski.com <ra...@gruchalski.com>
>> de.linkedin.com/in/radgruchalski/
>>
>>
>> *Confidentiality:*This communication is intended for the above-named
>> person and may be confidential and/or legally privileged.
>> If it has come to you in error you must take no action based on it, nor
>> must you copy or show it to anyone; please delete/destroy and inform the
>> sender immediately.
>>
>>
>> On Sunday, 22 November 2015 at 00:38, bowen zhang wrote:
>>
>> Hi folks,
>> I am a big fan of Spark's Mllib package. I have a java web app where I
>> want to run some ml jobs inside the web app. My question is: is there a way
>> to just import spark-core and spark-mllib jars to invoke my ML jobs without
>> installing the entire Spark package? All the tutorials related Spark seems
>> to indicate installing Spark is a pre-condition for this.
>>
>> Thanks,
>> Bowen
>>
>>
>>
>>
>>
>>
>

Re: Using spark MLlib without installing Spark

Posted by Reynold Xin <rx...@databricks.com>.
You can use MLlib and Spark directly without "installing anything". Just
run Spark in local mode.


On Sat, Nov 21, 2015 at 4:05 PM, Rad Gruchalski <ra...@gruchalski.com>
wrote:

> Bowen,
>
> What Andy is doing in the notebook is a slightly different thing. He’s
> using sbt to bring all spark jars (core, mllib, repl, what have you). You
> could use maven for that. He then creates a repl and submits all the spark
> code into it.
> Pretty sure spark unit tests cover similar uses cases. Maybe not mllib per
> se but this kind of submission.
>
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
> On Sunday, 22 November 2015 at 01:01, bowen zhang wrote:
>
> Thanks Rad for info. I looked into the repo and see some .snb file using
> spark mllib. Can you give me a more specific place to look for when
> invoking the mllib functions? What if I just want to invoke some of the ML
> functions in my HelloWorld.java?
>
> ------------------------------
> *From:* Rad Gruchalski <ra...@gruchalski.com>
> *To:* bowen zhang <bo...@yahoo.com>
> *Cc:* "dev@spark.apache.org" <de...@spark.apache.org>
> *Sent:* Saturday, November 21, 2015 3:43 PM
> *Subject:* Re: Using spark MLlib without installing Spark
>
> Bowen,
>
> One project to look at could be spark-notebook:
> https://github.com/andypetrella/spark-notebook
> It uses Spark you in the way you intend to use it.
> Kind regards,
> Radek Gruchalski
> radek@gruchalski.com <ra...@gruchalski.com>
> de.linkedin.com/in/radgruchalski/
>
>
> *Confidentiality:*This communication is intended for the above-named
> person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
> On Sunday, 22 November 2015 at 00:38, bowen zhang wrote:
>
> Hi folks,
> I am a big fan of Spark's Mllib package. I have a java web app where I
> want to run some ml jobs inside the web app. My question is: is there a way
> to just import spark-core and spark-mllib jars to invoke my ML jobs without
> installing the entire Spark package? All the tutorials related Spark seems
> to indicate installing Spark is a pre-condition for this.
>
> Thanks,
> Bowen
>
>
>
>
>
>

Re: Using spark MLlib without installing Spark

Posted by Rad Gruchalski <ra...@gruchalski.com>.
Bowen,  

What Andy is doing in the notebook is a slightly different thing. He’s using sbt to bring all spark jars (core, mllib, repl, what have you). You could use maven for that. He then creates a repl and submits all the spark code into it.
Pretty sure spark unit tests cover similar uses cases. Maybe not mllib per se but this kind of submission.










Kind regards,

Radek Gruchalski

radek@gruchalski.com (mailto:radek@gruchalski.com)
 (mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.



On Sunday, 22 November 2015 at 01:01, bowen zhang wrote:

> Thanks Rad for info. I looked into the repo and see some .snb file using spark mllib. Can you give me a more specific place to look for when invoking the mllib functions? What if I just want to invoke some of the ML functions in my HelloWorld.java?
>  
> From: Rad Gruchalski <radek@gruchalski.com (mailto:radek@gruchalski.com)>
> To: bowen zhang <bowenzhangusa@yahoo.com (mailto:bowenzhangusa@yahoo.com)>  
> Cc: "dev@spark.apache.org (mailto:dev@spark.apache.org)" <dev@spark.apache.org (mailto:dev@spark.apache.org)>
> Sent: Saturday, November 21, 2015 3:43 PM
> Subject: Re: Using spark MLlib without installing Spark
>  
> Bowen,  
>  
> One project to look at could be spark-notebook: https://github.com/andypetrella/spark-notebook
> It uses Spark you in the way you intend to use it.
>  
>  
>  
> Kind regards,

> Radek Gruchalski
> 
radek@gruchalski.com (mailto:radek@gruchalski.com)
 (mailto:radek@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
>  
> Confidentiality:
> This communication is intended for the above-named person and may be confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.  
>  
>  
> On Sunday, 22 November 2015 at 00:38, bowen zhang wrote:
> > Hi folks,
> > I am a big fan of Spark's Mllib package. I have a java web app where I want to run some ml jobs inside the web app. My question is: is there a way to just import spark-core and spark-mllib jars to invoke my ML jobs without installing the entire Spark package? All the tutorials related Spark seems to indicate installing Spark is a pre-condition for this.
> >  
> > Thanks,
> > Bowen
>  
>  
>  


Re: Using spark MLlib without installing Spark

Posted by bowen zhang <bo...@yahoo.com.INVALID>.
Thanks Rad for info. I looked into the repo and see some .snb file using spark mllib. Can you give me a more specific place to look for when invoking the mllib functions? What if I just want to invoke some of the ML functions in my HelloWorld.java?

      From: Rad Gruchalski <ra...@gruchalski.com>
 To: bowen zhang <bo...@yahoo.com> 
Cc: "dev@spark.apache.org" <de...@spark.apache.org>
 Sent: Saturday, November 21, 2015 3:43 PM
 Subject: Re: Using spark MLlib without installing Spark
   
Bowen, 
One project to look at could be spark-notebook: https://github.com/andypetrella/spark-notebookIt uses Spark you in the way you intend to use it.    Kindregards,

RadekGruchalski

radek@gruchalski.com

de.linkedin.com/in/radgruchalski/

Confidentiality:
Thiscommunication is intended for the above-named person and may beconfidential and/or legally privileged.
If it has come to you inerror you must take no action based on it, nor must you copy or showit to anyone; please delete/destroy and inform the senderimmediately. 

On Sunday, 22 November 2015 at 00:38, bowen zhang wrote: 
 Hi folks,I am a big fan of Spark's Mllib package. I have a java web app where I want to run some ml jobs inside the web app. My question is: is there a way to just import spark-core and spark-mllib jars to invoke my ML jobs without installing the entire Spark package? All the tutorials related Spark seems to indicate installing Spark is a pre-condition for this.
Thanks,Bowen
 
  
 

  

Re: Using spark MLlib without installing Spark

Posted by Rad Gruchalski <ra...@gruchalski.com>.
Bowen,  

One project to look at could be spark-notebook: https://github.com/andypetrella/spark-notebook
It uses Spark you in the way you intend to use it.










Kind regards,

Radek Gruchalski

radek@gruchalski.com (mailto:radek@gruchalski.com)
 (mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)

Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.



On Sunday, 22 November 2015 at 00:38, bowen zhang wrote:

> Hi folks,
> I am a big fan of Spark's Mllib package. I have a java web app where I want to run some ml jobs inside the web app. My question is: is there a way to just import spark-core and spark-mllib jars to invoke my ML jobs without installing the entire Spark package? All the tutorials related Spark seems to indicate installing Spark is a pre-condition for this.
>  
> Thanks,
> Bowen