You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Deepak Gopalakrishnan <dg...@gmail.com> on 2016/03/11 04:52:35 UTC
Running ALS on comparitively large RDD
Hello All,
I've been running Spark's ALS on a dataset of users and rated items. I
first encode my users to integers by using an auto increment function (
just like zipWithIndex), I do the same for my items. I then create an RDD
of the ratings and feed it to ALS.
My issue is that the ALS algorithm never completes. Attached is a
screenshot of the stages window.
Any help will be greatly appreciated
--
Regards,
*Deepak Gopalakrishnan*
*Mobile*:+918891509774
*Skype* : deepakgk87
http://myexps.blogspot.com
Re: Running ALS on comparitively large RDD
Posted by Deepak Gopalakrishnan <dg...@gmail.com>.
Executor memory : 45g X 4 executors , 1 Driver with 45g memory
Data Source is from S3 and I've logs that tells me the Rating objects are
loaded fine.
On Fri, Mar 11, 2016 at 2:13 PM, Nick Pentreath <ni...@gmail.com>
wrote:
> Hmmm, something else is going on there. What data source are you reading
> from? How much driver and executor memory have you provided to Spark?
>
>
>
> On Fri, 11 Mar 2016 at 09:21 Deepak Gopalakrishnan <dg...@gmail.com>
> wrote:
>
>> 1. I'm using about 1 million users against few thousand products. I
>> basically have around a million ratings
>> 2. Spark 1.6 on Amazon EMR
>>
>> On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath <
>> nick.pentreath@gmail.com> wrote:
>>
>>> Could you provide more details about:
>>> 1. Data set size (# ratings, # users and # products)
>>> 2. Spark cluster set up and version
>>>
>>> Thanks
>>>
>>> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com>
>>> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I've been running Spark's ALS on a dataset of users and rated items. I
>>>> first encode my users to integers by using an auto increment function (
>>>> just like zipWithIndex), I do the same for my items. I then create an RDD
>>>> of the ratings and feed it to ALS.
>>>>
>>>> My issue is that the ALS algorithm never completes. Attached is a
>>>> screenshot of the stages window.
>>>>
>>>> Any help will be greatly appreciated
>>>>
>>>> --
>>>> Regards,
>>>> *Deepak Gopalakrishnan*
>>>> *Mobile*:+918891509774
>>>> *Skype* : deepakgk87
>>>> http://myexps.blogspot.com
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>>
>> --
>> Regards,
>> *Deepak Gopalakrishnan*
>> *Mobile*:+918891509774
>> *Skype* : deepakgk87
>> http://myexps.blogspot.com
>>
>>
--
Regards,
*Deepak Gopalakrishnan*
*Mobile*:+918891509774
*Skype* : deepakgk87
http://myexps.blogspot.com
Re: Running ALS on comparitively large RDD
Posted by Deepak Gopalakrishnan <dg...@gmail.com>.
Executor memory : 45g X 4 executors , 1 Driver with 45g memory
Data Source is from S3 and I've logs that tells me the Rating objects are
loaded fine.
On Fri, Mar 11, 2016 at 2:13 PM, Nick Pentreath <ni...@gmail.com>
wrote:
> Hmmm, something else is going on there. What data source are you reading
> from? How much driver and executor memory have you provided to Spark?
>
>
>
> On Fri, 11 Mar 2016 at 09:21 Deepak Gopalakrishnan <dg...@gmail.com>
> wrote:
>
>> 1. I'm using about 1 million users against few thousand products. I
>> basically have around a million ratings
>> 2. Spark 1.6 on Amazon EMR
>>
>> On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath <
>> nick.pentreath@gmail.com> wrote:
>>
>>> Could you provide more details about:
>>> 1. Data set size (# ratings, # users and # products)
>>> 2. Spark cluster set up and version
>>>
>>> Thanks
>>>
>>> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com>
>>> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I've been running Spark's ALS on a dataset of users and rated items. I
>>>> first encode my users to integers by using an auto increment function (
>>>> just like zipWithIndex), I do the same for my items. I then create an RDD
>>>> of the ratings and feed it to ALS.
>>>>
>>>> My issue is that the ALS algorithm never completes. Attached is a
>>>> screenshot of the stages window.
>>>>
>>>> Any help will be greatly appreciated
>>>>
>>>> --
>>>> Regards,
>>>> *Deepak Gopalakrishnan*
>>>> *Mobile*:+918891509774
>>>> *Skype* : deepakgk87
>>>> http://myexps.blogspot.com
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>>
>> --
>> Regards,
>> *Deepak Gopalakrishnan*
>> *Mobile*:+918891509774
>> *Skype* : deepakgk87
>> http://myexps.blogspot.com
>>
>>
--
Regards,
*Deepak Gopalakrishnan*
*Mobile*:+918891509774
*Skype* : deepakgk87
http://myexps.blogspot.com
Re: Running ALS on comparitively large RDD
Posted by Nick Pentreath <ni...@gmail.com>.
Hmmm, something else is going on there. What data source are you reading
from? How much driver and executor memory have you provided to Spark?
On Fri, 11 Mar 2016 at 09:21 Deepak Gopalakrishnan <dg...@gmail.com> wrote:
> 1. I'm using about 1 million users against few thousand products. I
> basically have around a million ratings
> 2. Spark 1.6 on Amazon EMR
>
> On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath <nick.pentreath@gmail.com
> > wrote:
>
>> Could you provide more details about:
>> 1. Data set size (# ratings, # users and # products)
>> 2. Spark cluster set up and version
>>
>> Thanks
>>
>> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>>
>>> I've been running Spark's ALS on a dataset of users and rated items. I
>>> first encode my users to integers by using an auto increment function (
>>> just like zipWithIndex), I do the same for my items. I then create an RDD
>>> of the ratings and feed it to ALS.
>>>
>>> My issue is that the ALS algorithm never completes. Attached is a
>>> screenshot of the stages window.
>>>
>>> Any help will be greatly appreciated
>>>
>>> --
>>> Regards,
>>> *Deepak Gopalakrishnan*
>>> *Mobile*:+918891509774
>>> *Skype* : deepakgk87
>>> http://myexps.blogspot.com
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
> Regards,
> *Deepak Gopalakrishnan*
> *Mobile*:+918891509774
> *Skype* : deepakgk87
> http://myexps.blogspot.com
>
>
Re: Running ALS on comparitively large RDD
Posted by Nick Pentreath <ni...@gmail.com>.
Hmmm, something else is going on there. What data source are you reading
from? How much driver and executor memory have you provided to Spark?
On Fri, 11 Mar 2016 at 09:21 Deepak Gopalakrishnan <dg...@gmail.com> wrote:
> 1. I'm using about 1 million users against few thousand products. I
> basically have around a million ratings
> 2. Spark 1.6 on Amazon EMR
>
> On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath <nick.pentreath@gmail.com
> > wrote:
>
>> Could you provide more details about:
>> 1. Data set size (# ratings, # users and # products)
>> 2. Spark cluster set up and version
>>
>> Thanks
>>
>> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>>
>>> I've been running Spark's ALS on a dataset of users and rated items. I
>>> first encode my users to integers by using an auto increment function (
>>> just like zipWithIndex), I do the same for my items. I then create an RDD
>>> of the ratings and feed it to ALS.
>>>
>>> My issue is that the ALS algorithm never completes. Attached is a
>>> screenshot of the stages window.
>>>
>>> Any help will be greatly appreciated
>>>
>>> --
>>> Regards,
>>> *Deepak Gopalakrishnan*
>>> *Mobile*:+918891509774
>>> *Skype* : deepakgk87
>>> http://myexps.blogspot.com
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
> Regards,
> *Deepak Gopalakrishnan*
> *Mobile*:+918891509774
> *Skype* : deepakgk87
> http://myexps.blogspot.com
>
>
Re: Running ALS on comparitively large RDD
Posted by Deepak Gopalakrishnan <dg...@gmail.com>.
1. I'm using about 1 million users against few thousand products. I
basically have around a million ratings
2. Spark 1.6 on Amazon EMR
On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath <ni...@gmail.com>
wrote:
> Could you provide more details about:
> 1. Data set size (# ratings, # users and # products)
> 2. Spark cluster set up and version
>
> Thanks
>
> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com>
> wrote:
>
>> Hello All,
>>
>> I've been running Spark's ALS on a dataset of users and rated items. I
>> first encode my users to integers by using an auto increment function (
>> just like zipWithIndex), I do the same for my items. I then create an RDD
>> of the ratings and feed it to ALS.
>>
>> My issue is that the ALS algorithm never completes. Attached is a
>> screenshot of the stages window.
>>
>> Any help will be greatly appreciated
>>
>> --
>> Regards,
>> *Deepak Gopalakrishnan*
>> *Mobile*:+918891509774
>> *Skype* : deepakgk87
>> http://myexps.blogspot.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Regards,
*Deepak Gopalakrishnan*
*Mobile*:+918891509774
*Skype* : deepakgk87
http://myexps.blogspot.com
Re: Running ALS on comparitively large RDD
Posted by Deepak Gopalakrishnan <dg...@gmail.com>.
1. I'm using about 1 million users against few thousand products. I
basically have around a million ratings
2. Spark 1.6 on Amazon EMR
On Fri, Mar 11, 2016 at 12:46 PM, Nick Pentreath <ni...@gmail.com>
wrote:
> Could you provide more details about:
> 1. Data set size (# ratings, # users and # products)
> 2. Spark cluster set up and version
>
> Thanks
>
> On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com>
> wrote:
>
>> Hello All,
>>
>> I've been running Spark's ALS on a dataset of users and rated items. I
>> first encode my users to integers by using an auto increment function (
>> just like zipWithIndex), I do the same for my items. I then create an RDD
>> of the ratings and feed it to ALS.
>>
>> My issue is that the ALS algorithm never completes. Attached is a
>> screenshot of the stages window.
>>
>> Any help will be greatly appreciated
>>
>> --
>> Regards,
>> *Deepak Gopalakrishnan*
>> *Mobile*:+918891509774
>> *Skype* : deepakgk87
>> http://myexps.blogspot.com
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Regards,
*Deepak Gopalakrishnan*
*Mobile*:+918891509774
*Skype* : deepakgk87
http://myexps.blogspot.com
Re: Running ALS on comparitively large RDD
Posted by Nick Pentreath <ni...@gmail.com>.
Could you provide more details about:
1. Data set size (# ratings, # users and # products)
2. Spark cluster set up and version
Thanks
On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com> wrote:
> Hello All,
>
> I've been running Spark's ALS on a dataset of users and rated items. I
> first encode my users to integers by using an auto increment function (
> just like zipWithIndex), I do the same for my items. I then create an RDD
> of the ratings and feed it to ALS.
>
> My issue is that the ALS algorithm never completes. Attached is a
> screenshot of the stages window.
>
> Any help will be greatly appreciated
>
> --
> Regards,
> *Deepak Gopalakrishnan*
> *Mobile*:+918891509774
> *Skype* : deepakgk87
> http://myexps.blogspot.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
Re: Running ALS on comparitively large RDD
Posted by Nick Pentreath <ni...@gmail.com>.
Could you provide more details about:
1. Data set size (# ratings, # users and # products)
2. Spark cluster set up and version
Thanks
On Fri, 11 Mar 2016 at 05:53 Deepak Gopalakrishnan <dg...@gmail.com> wrote:
> Hello All,
>
> I've been running Spark's ALS on a dataset of users and rated items. I
> first encode my users to integers by using an auto increment function (
> just like zipWithIndex), I do the same for my items. I then create an RDD
> of the ratings and feed it to ALS.
>
> My issue is that the ALS algorithm never completes. Attached is a
> screenshot of the stages window.
>
> Any help will be greatly appreciated
>
> --
> Regards,
> *Deepak Gopalakrishnan*
> *Mobile*:+918891509774
> *Skype* : deepakgk87
> http://myexps.blogspot.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org