You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Diana Carroll <dc...@cloudera.com> on 2014/04/28 19:30:09 UTC
running SparkALS
Hi everyone. I'm trying to run some of the Spark example code, and most of
it appears to be undocumented (unless I'm missing something). Can someone
help me out?
I'm particularly interested in running SparkALS, which wants parameters:
M U F iter slices
What are these variables? They appear to be integers and the default
values are 100, 500 and 10 respectively but beyond that...huh?
Thanks!
Diana
Re: running SparkALS
Posted by Xiangrui Meng <me...@gmail.com>.
Hi Diana,
SparkALS is an example implementation of ALS. It doesn't call the ALS
algorithm implemented in MLlib. M, U, and F are used to generate
synthetic data.
I'm updating the examples. In the meantime, you can take a look at the
updated MLlib guide:
http://50.17.120.186:4000/mllib-collaborative-filtering.html and try
the example code there.
Thanks,
Xiangrui
On Mon, Apr 28, 2014 at 10:30 AM, Diana Carroll <dc...@cloudera.com> wrote:
> Hi everyone. I'm trying to run some of the Spark example code, and most of
> it appears to be undocumented (unless I'm missing something). Can someone
> help me out?
>
> I'm particularly interested in running SparkALS, which wants parameters:
> M U F iter slices
>
> What are these variables? They appear to be integers and the default values
> are 100, 500 and 10 respectively but beyond that...huh?
>
> Thanks!
>
> Diana
RE: running SparkALS
Posted by "Laird, Benjamin" <Be...@capitalone.com>.
Good clarification Sean. Diana, I was also referring to this example when setting up some of my bigger ALS runs. I don't this particular example is very helpful, as it is creating the initial matrix locally in memory before parallelizing in spark. So (unless I'm misunderstanding), it is an ok example at demonstrating basic spark functionality, but since it is limited by local memory you obviously can't demo it on bigger datasets.
The example at spark.apache.org/docs/0.9.1/mllib-guide.html I found to be much more helpful for good use cases.
-----Original Message-----
From: Sean Owen [mailto:sowen@cloudera.com]
Sent: Monday, April 28, 2014 1:41 PM
To: user@spark.apache.org
Subject: Re: running SparkALS
Yeah you'd have to look at the source code in this case; it's not explained in the scaladoc or usage message as far as I can see either.
The args refer specifically to the example of recommending Movies to Users. This example makes up a bunch of ratings and then makes recommendations using ALS.
M = number of movies in the made-up data U = number of users F = number of latent factors in the ALS model iter = number of iterations to run ALS for slices = level of parallelism to use on Spark
I'd say that could be explained in the scaladoc at least.
As an aside though, looking at the code, this is not a demo of ALS from MLlib, but a different implementation and based on the CERN linear algebra libraries. Is this still a good example?
--
Sean Owen | Director, Data Science | London
On Mon, Apr 28, 2014 at 6:30 PM, Diana Carroll <dc...@cloudera.com> wrote:
> Hi everyone. I'm trying to run some of the Spark example code, and
> most of it appears to be undocumented (unless I'm missing something).
> Can someone help me out?
>
> I'm particularly interested in running SparkALS, which wants parameters:
> M U F iter slices
>
> What are these variables? They appear to be integers and the default
> values are 100, 500 and 10 respectively but beyond that...huh?
>
> Thanks!
>
> Diana
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: running SparkALS
Posted by Sean Owen <so...@cloudera.com>.
Yeah you'd have to look at the source code in this case; it's not
explained in the scaladoc or usage message as far as I can see either.
The args refer specifically to the example of recommending Movies to
Users. This example makes up a bunch of ratings and then makes
recommendations using ALS.
M = number of movies in the made-up data
U = number of users
F = number of latent factors in the ALS model
iter = number of iterations to run ALS for
slices = level of parallelism to use on Spark
I'd say that could be explained in the scaladoc at least.
As an aside though, looking at the code, this is not a demo of ALS
from MLlib, but a different implementation and based on the CERN
linear algebra libraries. Is this still a good example?
--
Sean Owen | Director, Data Science | London
On Mon, Apr 28, 2014 at 6:30 PM, Diana Carroll <dc...@cloudera.com> wrote:
> Hi everyone. I'm trying to run some of the Spark example code, and most of
> it appears to be undocumented (unless I'm missing something). Can someone
> help me out?
>
> I'm particularly interested in running SparkALS, which wants parameters:
> M U F iter slices
>
> What are these variables? They appear to be integers and the default values
> are 100, 500 and 10 respectively but beyond that...huh?
>
> Thanks!
>
> Diana
Re: running SparkALS
Posted by Li Pu <lp...@twitter.com>.
http://spark.apache.org/docs/0.9.0/mllib-guide.html#collaborative-filtering-1
One thing which is undocumented: the integers representing users and
items have to be positive. Otherwise it throws exceptions.
Li
On 28 avr. 2014, at 10:30, Diana Carroll <dc...@cloudera.com> wrote:
> Hi everyone. I'm trying to run some of the Spark example code, and most of it appears to be undocumented (unless I'm missing something). Can someone help me out?
>
> I'm particularly interested in running SparkALS, which wants parameters:
> M U F iter slices
>
> What are these variables? They appear to be integers and the default values are 100, 500 and 10 respectively but beyond that...huh?
>
> Thanks!
>
> Diana
Re: running SparkALS
Posted by Diana Carroll <dc...@cloudera.com>.
Should I file a JIRA to remove the example? I think it is confusing to
include example code without explanation of how to run it, and it sounds
like this one isn't worth running or reviewing anyway.
On Mon, Apr 28, 2014 at 2:34 PM, Debasish Das <de...@gmail.com>wrote:
> Don't use SparkALS...that's the first version of the code and does not
> scale...
>
> Li is right...you have to do the dictionary generation on users, products
> and then generate indexed file....I wrote some utilities but looks like it
> is application dependent....the indexed netflix format is more generic...
>
>
> On Mon, Apr 28, 2014 at 10:47 AM, Diana Carroll <dc...@cloudera.com>wrote:
>
>> Thanks, Deb. But I'm looking at org.apache.spark.examples.SparkALS,
>> which is not in the mllib examples, and does not take any file parameters.
>>
>> I don't see the class you refer to in the examples ...however, if I did
>> want to run that example, where would I find the file in question?
>>
>> It would be great if this were documented, perhaps in the source code.
>> I'll add a JIRA.
>>
>> Thanks,
>> Diana
>>
>>
>> On Mon, Apr 28, 2014 at 1:41 PM, Debasish Das <de...@gmail.com>wrote:
>>
>>> Diana,
>>>
>>> Here are the parameters:
>>>
>>> ./bin/spark-class org.apache.spark.mllib.recommendation.ALS
>>> Usage: ALS <master> <ratings_file> <rank> <iterations> <output_dir>
>>> [<lambda>] [<implicitPrefs>] [<alpha>] [<blocks>]
>>>
>>> Master: Local/Deployed spark cluster master
>>> ratings_file: Netflix format data
>>> rank: Reduced dimension of the User and Product factors
>>> iterations: How many ALS iterations you would like to run
>>> output_dir: Where to generate the usera and product factors
>>>
>>> lambda: regularization for nuclear norm
>>> implicitPrefs: true will run Koren's netflix prize paper's implicit
>>> algorithm
>>> alpha: is valid for implicitPrefs
>>> blocks: How many blocks you want to partition your rating, user and
>>> product factor matrix
>>>
>>> I have run with 64 GB JVM with 20M users, 1.6M ratings and 50
>>> factors....you should be able to go even beyond that if you want to
>>> increase the JVM size...
>>>
>>> The scalability issue comes from the fact that each JVM has to collect
>>> either user/product factors before doing a BLAS posv solve....seems like
>>> that's the bottleneck step...but making double to float is one way to scale
>>> it even further...
>>>
>>> Thanks.
>>> Deb
>>>
>>>
>>>
>>> On Mon, Apr 28, 2014 at 10:30 AM, Diana Carroll <dc...@cloudera.com>wrote:
>>>
>>>> Hi everyone. I'm trying to run some of the Spark example code, and
>>>> most of it appears to be undocumented (unless I'm missing something). Can
>>>> someone help me out?
>>>>
>>>> I'm particularly interested in running SparkALS, which wants parameters:
>>>> M U F iter slices
>>>>
>>>> What are these variables? They appear to be integers and the default
>>>> values are 100, 500 and 10 respectively but beyond that...huh?
>>>>
>>>> Thanks!
>>>>
>>>> Diana
>>>>
>>>
>>>
>>
>
Re: running SparkALS
Posted by Debasish Das <de...@gmail.com>.
Don't use SparkALS...that's the first version of the code and does not
scale...
Li is right...you have to do the dictionary generation on users, products
and then generate indexed file....I wrote some utilities but looks like it
is application dependent....the indexed netflix format is more generic...
On Mon, Apr 28, 2014 at 10:47 AM, Diana Carroll <dc...@cloudera.com>wrote:
> Thanks, Deb. But I'm looking at org.apache.spark.examples.SparkALS,
> which is not in the mllib examples, and does not take any file parameters.
>
> I don't see the class you refer to in the examples ...however, if I did
> want to run that example, where would I find the file in question?
>
> It would be great if this were documented, perhaps in the source code.
> I'll add a JIRA.
>
> Thanks,
> Diana
>
>
> On Mon, Apr 28, 2014 at 1:41 PM, Debasish Das <de...@gmail.com>wrote:
>
>> Diana,
>>
>> Here are the parameters:
>>
>> ./bin/spark-class org.apache.spark.mllib.recommendation.ALS
>> Usage: ALS <master> <ratings_file> <rank> <iterations> <output_dir>
>> [<lambda>] [<implicitPrefs>] [<alpha>] [<blocks>]
>>
>> Master: Local/Deployed spark cluster master
>> ratings_file: Netflix format data
>> rank: Reduced dimension of the User and Product factors
>> iterations: How many ALS iterations you would like to run
>> output_dir: Where to generate the usera and product factors
>>
>> lambda: regularization for nuclear norm
>> implicitPrefs: true will run Koren's netflix prize paper's implicit
>> algorithm
>> alpha: is valid for implicitPrefs
>> blocks: How many blocks you want to partition your rating, user and
>> product factor matrix
>>
>> I have run with 64 GB JVM with 20M users, 1.6M ratings and 50
>> factors....you should be able to go even beyond that if you want to
>> increase the JVM size...
>>
>> The scalability issue comes from the fact that each JVM has to collect
>> either user/product factors before doing a BLAS posv solve....seems like
>> that's the bottleneck step...but making double to float is one way to scale
>> it even further...
>>
>> Thanks.
>> Deb
>>
>>
>>
>> On Mon, Apr 28, 2014 at 10:30 AM, Diana Carroll <dc...@cloudera.com>wrote:
>>
>>> Hi everyone. I'm trying to run some of the Spark example code, and most
>>> of it appears to be undocumented (unless I'm missing something). Can
>>> someone help me out?
>>>
>>> I'm particularly interested in running SparkALS, which wants parameters:
>>> M U F iter slices
>>>
>>> What are these variables? They appear to be integers and the default
>>> values are 100, 500 and 10 respectively but beyond that...huh?
>>>
>>> Thanks!
>>>
>>> Diana
>>>
>>
>>
>
Re: running SparkALS
Posted by Diana Carroll <dc...@cloudera.com>.
Thanks, Deb. But I'm looking at org.apache.spark.examples.SparkALS, which
is not in the mllib examples, and does not take any file parameters.
I don't see the class you refer to in the examples ...however, if I did
want to run that example, where would I find the file in question?
It would be great if this were documented, perhaps in the source code.
I'll add a JIRA.
Thanks,
Diana
On Mon, Apr 28, 2014 at 1:41 PM, Debasish Das <de...@gmail.com>wrote:
> Diana,
>
> Here are the parameters:
>
> ./bin/spark-class org.apache.spark.mllib.recommendation.ALS
> Usage: ALS <master> <ratings_file> <rank> <iterations> <output_dir>
> [<lambda>] [<implicitPrefs>] [<alpha>] [<blocks>]
>
> Master: Local/Deployed spark cluster master
> ratings_file: Netflix format data
> rank: Reduced dimension of the User and Product factors
> iterations: How many ALS iterations you would like to run
> output_dir: Where to generate the usera and product factors
>
> lambda: regularization for nuclear norm
> implicitPrefs: true will run Koren's netflix prize paper's implicit
> algorithm
> alpha: is valid for implicitPrefs
> blocks: How many blocks you want to partition your rating, user and
> product factor matrix
>
> I have run with 64 GB JVM with 20M users, 1.6M ratings and 50
> factors....you should be able to go even beyond that if you want to
> increase the JVM size...
>
> The scalability issue comes from the fact that each JVM has to collect
> either user/product factors before doing a BLAS posv solve....seems like
> that's the bottleneck step...but making double to float is one way to scale
> it even further...
>
> Thanks.
> Deb
>
>
>
> On Mon, Apr 28, 2014 at 10:30 AM, Diana Carroll <dc...@cloudera.com>wrote:
>
>> Hi everyone. I'm trying to run some of the Spark example code, and most
>> of it appears to be undocumented (unless I'm missing something). Can
>> someone help me out?
>>
>> I'm particularly interested in running SparkALS, which wants parameters:
>> M U F iter slices
>>
>> What are these variables? They appear to be integers and the default
>> values are 100, 500 and 10 respectively but beyond that...huh?
>>
>> Thanks!
>>
>> Diana
>>
>
>
Re: running SparkALS
Posted by Debasish Das <de...@gmail.com>.
Diana,
Here are the parameters:
./bin/spark-class org.apache.spark.mllib.recommendation.ALS
Usage: ALS <master> <ratings_file> <rank> <iterations> <output_dir>
[<lambda>] [<implicitPrefs>] [<alpha>] [<blocks>]
Master: Local/Deployed spark cluster master
ratings_file: Netflix format data
rank: Reduced dimension of the User and Product factors
iterations: How many ALS iterations you would like to run
output_dir: Where to generate the usera and product factors
lambda: regularization for nuclear norm
implicitPrefs: true will run Koren's netflix prize paper's implicit
algorithm
alpha: is valid for implicitPrefs
blocks: How many blocks you want to partition your rating, user and product
factor matrix
I have run with 64 GB JVM with 20M users, 1.6M ratings and 50
factors....you should be able to go even beyond that if you want to
increase the JVM size...
The scalability issue comes from the fact that each JVM has to collect
either user/product factors before doing a BLAS posv solve....seems like
that's the bottleneck step...but making double to float is one way to scale
it even further...
Thanks.
Deb
On Mon, Apr 28, 2014 at 10:30 AM, Diana Carroll <dc...@cloudera.com>wrote:
> Hi everyone. I'm trying to run some of the Spark example code, and most
> of it appears to be undocumented (unless I'm missing something). Can
> someone help me out?
>
> I'm particularly interested in running SparkALS, which wants parameters:
> M U F iter slices
>
> What are these variables? They appear to be integers and the default
> values are 100, 500 and 10 respectively but beyond that...huh?
>
> Thanks!
>
> Diana
>