You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Thomas Rewig <tr...@mufin.com> on 2011/04/14 12:18:00 UTC
Hints for Best Practices for Jobs with amazon EMR
Hello
right now I'm testing Mahout (taste) Jobs on AWS EMR.
I wonder if anyone does have any experience with the best cluster size
and the best EC2 instances. Are there any best practices for mahout
(taste) jobs?
On my first test I used a small 22 MB user-item-model an compute an
ItemSimilarityJob with 3 small EC2 instances:
ruby elastic-mapreduce --create --alive --slave-instance-type m1.small
--master-instance-type m1.small --num-instances 3 --name
mahout-0.5-itemSimJob-TEST
ruby elastic-mapreduce
--jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
--main-class
org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
--arg -i --arg s3://some-uri/input/data_small_in.csv
--arg -o --arg s3://some-uri/output/data_out_small.csv
--arg -s
--arg SIMILARITY_LOGLIKELIHOOD
--arg -m
--arg 500
--arg -mo
--arg 500
-j JobId
Here everything worked well, even if it took a few minutes.
In a second test I used a bigger 200 MB user-item-model and do the same
with a cluster of large instances:
ruby elastic-mapreduce --create --alive --slave-instance-type m1.large
--master-instance-type m1.large --num-instances 8 --name
mahout-0.5-itemSimJob-TEST2
I logged in on the masterNode with ssh and looked at the syslog. First a
few hours everything looked ok and then it seems to stops at a 63%
reduce step. I wait a few hours but nothing happend and so i terminate
the job. I even couldn't find any errors in the logs.
So here my questions:
1. are the any proved best practice clustersizes and instancetypes
(Standard- or High-Memory- or High-CPU-Instances) that work fine for big
recommender jobs, or do I have to test it for every different job I use?
2. would it have some positiv effect if I split my big data_in.csv into
many small csv's?
Do anyone have any experience with it and have some hints?
Thanks in advance
Thomas
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Sean Owen <sr...@gmail.com>.
A few assorted hints --
You don't need high-memory instance. You do want to use instance with higher
I/O performance, all else equal. m1.large is a good choice.
I'd also make sure the number of mappers / reducers is at least the number
of instances, in every case. If you see a job has chosen to deploy only 10
reducers for whatever reason when you have 20 instances running, that's
suboptimal. We can talk about how to change that number. In fact for
m1.large it will be default run 2 reducers per instance, so you in theory
want 2x instances reducers.
On Thu, Apr 14, 2011 at 11:18 AM, Thomas Rewig <tr...@mufin.com> wrote:
> Hello
> right now I'm testing Mahout (taste) Jobs on AWS EMR.
> I wonder if anyone does have any experience with the best cluster size and
> the best EC2 instances. Are there any best practices for mahout (taste)
> jobs?
>
> On my first test I used a small 22 MB user-item-model an compute an
> ItemSimilarityJob with 3 small EC2 instances:
>
> ruby elastic-mapreduce --create --alive --slave-instance-type m1.small
> --master-instance-type m1.small --num-instances 3 --name
> mahout-0.5-itemSimJob-TEST
>
>
> ruby elastic-mapreduce
> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
> --main-class
> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
> --arg -i --arg s3://some-uri/input/data_small_in.csv
> --arg -o --arg s3://some-uri/output/data_out_small.csv
> --arg -s
> --arg SIMILARITY_LOGLIKELIHOOD
> --arg -m
> --arg 500
> --arg -mo
> --arg 500
> -j JobId
>
> Here everything worked well, even if it took a few minutes.
>
> In a second test I used a bigger 200 MB user-item-model and do the same
> with a cluster of large instances:
>
> ruby elastic-mapreduce --create --alive --slave-instance-type m1.large
> --master-instance-type m1.large --num-instances 8 --name
> mahout-0.5-itemSimJob-TEST2
>
> I logged in on the masterNode with ssh and looked at the syslog. First a
> few hours everything looked ok and then it seems to stops at a 63% reduce
> step. I wait a few hours but nothing happend and so i terminate the job. I
> even couldn't find any errors in the logs.
>
> So here my questions:
> 1. are the any proved best practice clustersizes and instancetypes
> (Standard- or High-Memory- or High-CPU-Instances) that work fine for big
> recommender jobs, or do I have to test it for every different job I use?
> 2. would it have some positiv effect if I split my big data_in.csv into
> many small csv's?
>
> Do anyone have any experience with it and have some hints?
>
> Thanks in advance
> Thomas
>
>
>
>
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Mat Kelcey <ma...@gmail.com>.
Hi,
I'm a bit late to this thread but anyways...
One other valuable profiling tool is Ganglia which you can install on
your cluster using a boostrap action
elastic-mapreduce --create --alive --instance-type m1.xlarge --num-instances 5 \
--bootstrap-action s3://elasticmapreduce/bootstrap-actions/install-ganglia
This tracks all the system level stuff you'd expect (cpu/ram/network
etc) as well as a bunch of hadoop level metrics
For more details see here...
http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?Ganglia_reports.html
Cheers,
Mat
(SDE on EMR)
On 14 April 2011 03:18, Thomas Rewig <tr...@mufin.com> wrote:
> Hello
> right now I'm testing Mahout (taste) Jobs on AWS EMR.
> I wonder if anyone does have any experience with the best cluster size and
> the best EC2 instances. Are there any best practices for mahout (taste)
> jobs?
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Thomas Rewig <tr...@mufin.com>.
Hi Sebastian,
It is not my primary goal to compute an inputmodel for an itembased
recommendation system, of course it would be a nice bonus. The thing I
try to do is to get an orderd list of n-similar items for every item
(could be songs, artists, playlists, users ... the usual stuff) based on
collaborative data (playhistorys and equal stuff ) mixed with static
aspect data (signalbased stuff).
For me taste seems to be the easiest way to get this orderd list and the
results on the small testsets are really nice. Maybe there is a way to
reach the same result with some clustering technique, but I havent
tested it until now.
Of course I can compute all Item-Item-Similaritys for the sparse data
and use the dense data in a next step as a filter for the
similarity-lists. But I tought it would be cool, if I drop all data in
one model and see if the results are perhaps better.
This is a thing I wanted to try for quite some time. And I was hoping
that it would work with the hadoop cluster.
Does anyone have any other ideas and techniques I can try to calculate
similarity of items based on collaborative data?
Thanks in advance
Thomas
> Hi Thomas,
>
> I'd say now the long running time comes from items from dataset two
> that have lots of user-preferences. Seems like your data is to dense
> to compare all pairs of users with ItemSimilarityJob.
>
> What exactly is the problem you're trying to solve with computing
> similar users? Do you need that as input for the computation of
> recommendations? Maybe we'll find another approach for you on this list.
>
> --sebastian
>
> On 14.04.2011 16:59, Thomas Rewig wrote:
>> Hi Sebastian
>>
>> in my datamodel are 17733658 Datapoints, there are 230116 unique
>> users (U(Ix)) and 208760 unique items I(Ux).
>> The datapoints are in some way dense and sparse because I test to
>> merge 2 Datasets and invert the result, so I can use the
>> ItemSimilarityJob:
>>
>> e.g.:
>>
>> I = Item
>> U = User
>>
>> Dataset1 (the sparse one):
>> I1 I2 I3 I4
>> U1 9 8
>> U2 7 4
>> U3 8 5
>> U5 5 9
>>
>> Dataset2 (the dense one, but has much less Items than Dataset1):
>>
>> I5 I6
>> U1 1 2
>> U2 3 2
>> U3 2
>> U4 5 3
>> U5 1 1
>>
>> Invert Dataset(1+2) so Users are Items an vice versa:
>>
>> I(U1) I(U2) I(U3) I(U4) I(U5)
>> U(I1) 9 7 5
>> U(I2) 8
>> U(I3) 4 9
>> U(I4) 8 5
>> U(I5) 1 3 2 5 1
>> U(I6) 2 2 3 1
>>
>> So Yes your right because of this invertation I have users with a
>> lots of preferences (nearly the number of users in Dataset2), and I
>> can understand why the system seems to stop.
>>
>> Maybe the Invertation of the Data isn't a good way for this purpose
>> and i have to write my own UserUserSimilartityJob. (ok in the moment
>> I have no idea how to do this, because I just started with hadoop and
>> mapreduce, but I can try ;-) ).
>>
>> Do you have some other hints I can try?
>>
>>>
>>> Can you same how many datapoints your data contains and how dense
>>> those are? 200MB doesn't seem that much, it shouldn't take hours
>>> with 8 m1.large instances.
>>>
>>> Can you give us the values of the following counters?
>>>
>>> MaybePruneRowsMapper: Elements.USED
>>> MaybePruneRowsMapper: Elements.NEGLECTED
>>>
>>> CooccurrencesMapper: Counter.COOCCURRENCES
>>
>> I'm not sure, if I find the data in the logs you want, but maybe this
>> log-sample helps:
>>
>> MaybePruneRowsMapper: Elements.NEGLECTED = 8798627
>> MaybePruneRowsMapper: Elements.USED = 6821670
>>
>> I cant find Counter.COOCCURRENCES
>>
>>
>> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 92%
>> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 100%
>> INFO org.apache.hadoop.mapred.JobClient (main): Job complete:
>> job_201104130912_0004
>> INFO org.apache.hadoop.mapred.JobClient (main): Counters: 20
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> org.apache.mahout.cf.taste.hadoop.MaybePruneRowsMapper$Elements
>> INFO org.apache.hadoop.mapred.JobClient (main): NEGLECTED=8798627
>> INFO org.apache.hadoop.mapred.JobClient (main): USED=6821670
>> INFO org.apache.hadoop.mapred.JobClient (main): Job Counters
>> INFO org.apache.hadoop.mapred.JobClient (main): Launched reduce
>> tasks=24
>> INFO org.apache.hadoop.mapred.JobClient (main): Rack-local map
>> tasks=3
>> INFO org.apache.hadoop.mapred.JobClient (main): Launched map
>> tasks=24
>> INFO org.apache.hadoop.mapred.JobClient (main): Data-local map
>> tasks=21
>> INFO org.apache.hadoop.mapred.JobClient (main): FileSystemCounters
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> FILE_BYTES_READ=45452916
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> HDFS_BYTES_READ=120672701
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> FILE_BYTES_WRITTEN=106567950
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> HDFS_BYTES_WRITTEN=51234800
>> INFO org.apache.hadoop.mapred.JobClient (main): Map-Reduce Framework
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
>> groups=208760
>> INFO org.apache.hadoop.mapred.JobClient (main): Combine output
>> records=0
>> INFO org.apache.hadoop.mapred.JobClient (main): Map input
>> records=230201
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce shuffle
>> bytes=60461985
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce output
>> records=208760
>> INFO org.apache.hadoop.mapred.JobClient (main): Spilled
>> Records=13643340
>> INFO org.apache.hadoop.mapred.JobClient (main): Map output
>> bytes=136433400
>> INFO org.apache.hadoop.mapred.JobClient (main): Combine input
>> records=0
>> INFO org.apache.hadoop.mapred.JobClient (main): Map output
>> records=6821670
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
>> records=6821670
>> INFO org.apache.mahout.common.AbstractJob (main): Command line
>> arguments: {--endPhase=2147483647, --maxSimilaritiesPerRow=501,
>> --numberOfColumns=230201,
>> --similarityClassname=SIMILARITY_LOGLIKELIHOOD, --startPhase=0,
>> --tempDir=temp}
>> INFO org.apache.hadoop.mapred.JobClient (main): Default number of map
>> tasks: 2
>> INFO org.apache.hadoop.mapred.JobClient (main): Default number of
>> reduce tasks: 24
>> INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main):
>> Total input paths to process : 24
>> INFO org.apache.hadoop.mapred.JobClient (main): Running job:
>> job_201104130912_0005
>> INFO org.apache.hadoop.mapred.JobClient (main): map 0% reduce 0%
>> INFO org.apache.hadoop.mapred.JobClient (main): map 4% reduce 0%
>>
>>>
>>> By the way, I see that you're located in Berlin, I have some free
>>> time in the next 2 weeks if you want we could meet for a coffee and
>>> you'll get some free consultation!
>>>
>> It would be really great to meet you, but only the head office is in
>> Berlin. I am in Dresden and althougth that is not far away, it does
>> not look like I can go to Berlin. Maybe it works out later when I
>> visit the headquarter. I am sure you can explain a lot to me.
>>
>>
>>
>> Thanks in advance
>> Thomas
>>
>>
>>
>>
>>
>>
>>>
>>>
>>> On 14.04.2011 12:18, Thomas Rewig wrote:
>>>> Hello
>>>> right now I'm testing Mahout (taste) Jobs on AWS EMR.
>>>> I wonder if anyone does have any experience with the best cluster
>>>> size and the best EC2 instances. Are there any best practices for
>>>> mahout (taste) jobs?
>>>>
>>>> On my first test I used a small 22 MB user-item-model an compute an
>>>> ItemSimilarityJob with 3 small EC2 instances:
>>>>
>>>> ruby elastic-mapreduce --create --alive --slave-instance-type
>>>> m1.small --master-instance-type m1.small --num-instances 3 --name
>>>> mahout-0.5-itemSimJob-TEST
>>>>
>>>>
>>>> ruby elastic-mapreduce
>>>> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
>>>> --main-class
>>>> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
>>>> --arg -i --arg s3://some-uri/input/data_small_in.csv
>>>> --arg -o --arg s3://some-uri/output/data_out_small.csv
>>>> --arg -s
>>>> --arg SIMILARITY_LOGLIKELIHOOD
>>>> --arg -m
>>>> --arg 500
>>>> --arg -mo
>>>> --arg 500
>>>> -j JobId
>>>>
>>>> Here everything worked well, even if it took a few minutes.
>>>>
>>>> In a second test I used a bigger 200 MB user-item-model and do the
>>>> same with a cluster of large instances:
>>>>
>>>> ruby elastic-mapreduce --create --alive --slave-instance-type
>>>> m1.large --master-instance-type m1.large --num-instances 8 --name
>>>> mahout-0.5-itemSimJob-TEST2
>>>>
>>>> I logged in on the masterNode with ssh and looked at the syslog.
>>>> First a few hours everything looked ok and then it seems to stops
>>>> at a 63% reduce step. I wait a few hours but nothing happend and so
>>>> i terminate the job. I even couldn't find any errors in the logs.
>>>>
>>>> So here my questions:
>>>> 1. are the any proved best practice clustersizes and instancetypes
>>>> (Standard- or High-Memory- or High-CPU-Instances) that work fine
>>>> for big recommender jobs, or do I have to test it for every
>>>> different job I use?
>>>> 2. would it have some positiv effect if I split my big data_in.csv
>>>> into many small csv's?
>>>>
>>>> Do anyone have any experience with it and have some hints?
>>>>
>>>> Thanks in advance
>>>> Thomas
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Lance Norskog <go...@gmail.com>.
Also, can you limit users to N preferences, say 50? I don't know the
Mahout job; is this possible in the job flow?
Lance
On 4/14/11, Ted Dunning <te...@gmail.com> wrote:
> Thomas,
>
> You need to avoid operations that cause dense matrices. If you can possibly
> do that, you need to.
>
> One way is to rethink what you mean by similarity.
>
> On Thu, Apr 14, 2011 at 2:43 PM, Sebastian Schelter <ss...@apache.org> wrote:
>
>> Hi Thomas,
>>
>> I'd say now the long running time comes from items from dataset two that
>> have lots of user-preferences. Seems like your data is to dense to compare
>> all pairs of users with ItemSimilarityJob.
>>
>> What exactly is the problem you're trying to solve with computing similar
>> users? Do you need that as input for the computation of recommendations?
>> Maybe we'll find another approach for you on this list.
>>
>> --sebastian
>>
>> On 14.04.2011 16:59, Thomas Rewig wrote:
>>
>>> Hi Sebastian
>>>
>>> in my datamodel are 17733658 Datapoints, there are 230116 unique users
>>> (U(Ix)) and 208760 unique items I(Ux).
>>> The datapoints are in some way dense and sparse because I test to merge 2
>>> Datasets and invert the result, so I can use the ItemSimilarityJob:
>>>
>>> e.g.:
>>>
>>> I = Item
>>> U = User
>>>
>>> Dataset1 (the sparse one):
>>> I1 I2 I3 I4
>>> U1 9 8
>>> U2 7 4
>>> U3 8 5
>>> U5 5 9
>>>
>>> Dataset2 (the dense one, but has much less Items than Dataset1):
>>>
>>> I5 I6
>>> U1 1 2
>>> U2 3 2
>>> U3 2
>>> U4 5 3
>>> U5 1 1
>>>
>>> Invert Dataset(1+2) so Users are Items an vice versa:
>>>
>>> I(U1) I(U2) I(U3) I(U4) I(U5)
>>> U(I1) 9 7 5
>>> U(I2) 8
>>> U(I3) 4 9
>>> U(I4) 8 5
>>> U(I5) 1 3 2 5 1
>>> U(I6) 2 2 3 1
>>>
>>> So Yes your right because of this invertation I have users with a lots of
>>> preferences (nearly the number of users in Dataset2), and I can
>>> understand
>>> why the system seems to stop.
>>>
>>> Maybe the Invertation of the Data isn't a good way for this purpose and i
>>> have to write my own UserUserSimilartityJob. (ok in the moment I have no
>>> idea how to do this, because I just started with hadoop and mapreduce,
>>> but I
>>> can try ;-) ).
>>>
>>> Do you have some other hints I can try?
>>>
>>>
>>>> Can you same how many datapoints your data contains and how dense those
>>>> are? 200MB doesn't seem that much, it shouldn't take hours with 8
>>>> m1.large
>>>> instances.
>>>>
>>>> Can you give us the values of the following counters?
>>>>
>>>> MaybePruneRowsMapper: Elements.USED
>>>> MaybePruneRowsMapper: Elements.NEGLECTED
>>>>
>>>> CooccurrencesMapper: Counter.COOCCURRENCES
>>>>
>>>
>>> I'm not sure, if I find the data in the logs you want, but maybe this
>>> log-sample helps:
>>>
>>> MaybePruneRowsMapper: Elements.NEGLECTED = 8798627
>>> MaybePruneRowsMapper: Elements.USED = 6821670
>>>
>>> I cant find Counter.COOCCURRENCES
>>>
>>>
>>> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 92%
>>> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 100%
>>> INFO org.apache.hadoop.mapred.JobClient (main): Job complete:
>>> job_201104130912_0004
>>> INFO org.apache.hadoop.mapred.JobClient (main): Counters: 20
>>> INFO org.apache.hadoop.mapred.JobClient (main):
>>> org.apache.mahout.cf.taste.hadoop.MaybePruneRowsMapper$Elements
>>> INFO org.apache.hadoop.mapred.JobClient (main): NEGLECTED=8798627
>>> INFO org.apache.hadoop.mapred.JobClient (main): USED=6821670
>>> INFO org.apache.hadoop.mapred.JobClient (main): Job Counters
>>> INFO org.apache.hadoop.mapred.JobClient (main): Launched reduce
>>> tasks=24
>>> INFO org.apache.hadoop.mapred.JobClient (main): Rack-local map
>>> tasks=3
>>> INFO org.apache.hadoop.mapred.JobClient (main): Launched map tasks=24
>>> INFO org.apache.hadoop.mapred.JobClient (main): Data-local map
>>> tasks=21
>>> INFO org.apache.hadoop.mapred.JobClient (main): FileSystemCounters
>>> INFO org.apache.hadoop.mapred.JobClient (main):
>>> FILE_BYTES_READ=45452916
>>> INFO org.apache.hadoop.mapred.JobClient (main):
>>> HDFS_BYTES_READ=120672701
>>> INFO org.apache.hadoop.mapred.JobClient (main):
>>> FILE_BYTES_WRITTEN=106567950
>>> INFO org.apache.hadoop.mapred.JobClient (main):
>>> HDFS_BYTES_WRITTEN=51234800
>>> INFO org.apache.hadoop.mapred.JobClient (main): Map-Reduce Framework
>>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
>>> groups=208760
>>> INFO org.apache.hadoop.mapred.JobClient (main): Combine output
>>> records=0
>>> INFO org.apache.hadoop.mapred.JobClient (main): Map input
>>> records=230201
>>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce shuffle
>>> bytes=60461985
>>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce output
>>> records=208760
>>> INFO org.apache.hadoop.mapred.JobClient (main): Spilled
>>> Records=13643340
>>> INFO org.apache.hadoop.mapred.JobClient (main): Map output
>>> bytes=136433400
>>> INFO org.apache.hadoop.mapred.JobClient (main): Combine input
>>> records=0
>>> INFO org.apache.hadoop.mapred.JobClient (main): Map output
>>> records=6821670
>>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
>>> records=6821670
>>> INFO org.apache.mahout.common.AbstractJob (main): Command line arguments:
>>> {--endPhase=2147483647, --maxSimilaritiesPerRow=501,
>>> --numberOfColumns=230201, --similarityClassname=SIMILARITY_LOGLIKELIHOOD,
>>> --startPhase=0, --tempDir=temp}
>>> INFO org.apache.hadoop.mapred.JobClient (main): Default number of map
>>> tasks: 2
>>> INFO org.apache.hadoop.mapred.JobClient (main): Default number of reduce
>>> tasks: 24
>>> INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total
>>> input paths to process : 24
>>> INFO org.apache.hadoop.mapred.JobClient (main): Running job:
>>> job_201104130912_0005
>>> INFO org.apache.hadoop.mapred.JobClient (main): map 0% reduce 0%
>>> INFO org.apache.hadoop.mapred.JobClient (main): map 4% reduce 0%
>>>
>>>
>>>> By the way, I see that you're located in Berlin, I have some free time
>>>> in
>>>> the next 2 weeks if you want we could meet for a coffee and you'll get
>>>> some
>>>> free consultation!
>>>>
>>>> It would be really great to meet you, but only the head office is in
>>> Berlin. I am in Dresden and althougth that is not far away, it does not
>>> look
>>> like I can go to Berlin. Maybe it works out later when I visit the
>>> headquarter. I am sure you can explain a lot to me.
>>>
>>>
>>>
>>> Thanks in advance
>>> Thomas
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> On 14.04.2011 12:18, Thomas Rewig wrote:
>>>>
>>>>> Hello
>>>>> right now I'm testing Mahout (taste) Jobs on AWS EMR.
>>>>> I wonder if anyone does have any experience with the best cluster size
>>>>> and the best EC2 instances. Are there any best practices for mahout
>>>>> (taste)
>>>>> jobs?
>>>>>
>>>>> On my first test I used a small 22 MB user-item-model an compute an
>>>>> ItemSimilarityJob with 3 small EC2 instances:
>>>>>
>>>>> ruby elastic-mapreduce --create --alive --slave-instance-type m1.small
>>>>> --master-instance-type m1.small --num-instances 3 --name
>>>>> mahout-0.5-itemSimJob-TEST
>>>>>
>>>>>
>>>>> ruby elastic-mapreduce
>>>>> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
>>>>> --main-class
>>>>> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
>>>>> --arg -i --arg s3://some-uri/input/data_small_in.csv
>>>>> --arg -o --arg s3://some-uri/output/data_out_small.csv
>>>>> --arg -s
>>>>> --arg SIMILARITY_LOGLIKELIHOOD
>>>>> --arg -m
>>>>> --arg 500
>>>>> --arg -mo
>>>>> --arg 500
>>>>> -j JobId
>>>>>
>>>>> Here everything worked well, even if it took a few minutes.
>>>>>
>>>>> In a second test I used a bigger 200 MB user-item-model and do the same
>>>>> with a cluster of large instances:
>>>>>
>>>>> ruby elastic-mapreduce --create --alive --slave-instance-type m1.large
>>>>> --master-instance-type m1.large --num-instances 8 --name
>>>>> mahout-0.5-itemSimJob-TEST2
>>>>>
>>>>> I logged in on the masterNode with ssh and looked at the syslog. First
>>>>> a
>>>>> few hours everything looked ok and then it seems to stops at a 63%
>>>>> reduce
>>>>> step. I wait a few hours but nothing happend and so i terminate the
>>>>> job. I
>>>>> even couldn't find any errors in the logs.
>>>>>
>>>>> So here my questions:
>>>>> 1. are the any proved best practice clustersizes and instancetypes
>>>>> (Standard- or High-Memory- or High-CPU-Instances) that work fine for
>>>>> big
>>>>> recommender jobs, or do I have to test it for every different job I
>>>>> use?
>>>>> 2. would it have some positiv effect if I split my big data_in.csv into
>>>>> many small csv's?
>>>>>
>>>>> Do anyone have any experience with it and have some hints?
>>>>>
>>>>> Thanks in advance
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Ted Dunning <te...@gmail.com>.
Thomas,
You need to avoid operations that cause dense matrices. If you can possibly
do that, you need to.
One way is to rethink what you mean by similarity.
On Thu, Apr 14, 2011 at 2:43 PM, Sebastian Schelter <ss...@apache.org> wrote:
> Hi Thomas,
>
> I'd say now the long running time comes from items from dataset two that
> have lots of user-preferences. Seems like your data is to dense to compare
> all pairs of users with ItemSimilarityJob.
>
> What exactly is the problem you're trying to solve with computing similar
> users? Do you need that as input for the computation of recommendations?
> Maybe we'll find another approach for you on this list.
>
> --sebastian
>
> On 14.04.2011 16:59, Thomas Rewig wrote:
>
>> Hi Sebastian
>>
>> in my datamodel are 17733658 Datapoints, there are 230116 unique users
>> (U(Ix)) and 208760 unique items I(Ux).
>> The datapoints are in some way dense and sparse because I test to merge 2
>> Datasets and invert the result, so I can use the ItemSimilarityJob:
>>
>> e.g.:
>>
>> I = Item
>> U = User
>>
>> Dataset1 (the sparse one):
>> I1 I2 I3 I4
>> U1 9 8
>> U2 7 4
>> U3 8 5
>> U5 5 9
>>
>> Dataset2 (the dense one, but has much less Items than Dataset1):
>>
>> I5 I6
>> U1 1 2
>> U2 3 2
>> U3 2
>> U4 5 3
>> U5 1 1
>>
>> Invert Dataset(1+2) so Users are Items an vice versa:
>>
>> I(U1) I(U2) I(U3) I(U4) I(U5)
>> U(I1) 9 7 5
>> U(I2) 8
>> U(I3) 4 9
>> U(I4) 8 5
>> U(I5) 1 3 2 5 1
>> U(I6) 2 2 3 1
>>
>> So Yes your right because of this invertation I have users with a lots of
>> preferences (nearly the number of users in Dataset2), and I can understand
>> why the system seems to stop.
>>
>> Maybe the Invertation of the Data isn't a good way for this purpose and i
>> have to write my own UserUserSimilartityJob. (ok in the moment I have no
>> idea how to do this, because I just started with hadoop and mapreduce, but I
>> can try ;-) ).
>>
>> Do you have some other hints I can try?
>>
>>
>>> Can you same how many datapoints your data contains and how dense those
>>> are? 200MB doesn't seem that much, it shouldn't take hours with 8 m1.large
>>> instances.
>>>
>>> Can you give us the values of the following counters?
>>>
>>> MaybePruneRowsMapper: Elements.USED
>>> MaybePruneRowsMapper: Elements.NEGLECTED
>>>
>>> CooccurrencesMapper: Counter.COOCCURRENCES
>>>
>>
>> I'm not sure, if I find the data in the logs you want, but maybe this
>> log-sample helps:
>>
>> MaybePruneRowsMapper: Elements.NEGLECTED = 8798627
>> MaybePruneRowsMapper: Elements.USED = 6821670
>>
>> I cant find Counter.COOCCURRENCES
>>
>>
>> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 92%
>> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 100%
>> INFO org.apache.hadoop.mapred.JobClient (main): Job complete:
>> job_201104130912_0004
>> INFO org.apache.hadoop.mapred.JobClient (main): Counters: 20
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> org.apache.mahout.cf.taste.hadoop.MaybePruneRowsMapper$Elements
>> INFO org.apache.hadoop.mapred.JobClient (main): NEGLECTED=8798627
>> INFO org.apache.hadoop.mapred.JobClient (main): USED=6821670
>> INFO org.apache.hadoop.mapred.JobClient (main): Job Counters
>> INFO org.apache.hadoop.mapred.JobClient (main): Launched reduce
>> tasks=24
>> INFO org.apache.hadoop.mapred.JobClient (main): Rack-local map tasks=3
>> INFO org.apache.hadoop.mapred.JobClient (main): Launched map tasks=24
>> INFO org.apache.hadoop.mapred.JobClient (main): Data-local map
>> tasks=21
>> INFO org.apache.hadoop.mapred.JobClient (main): FileSystemCounters
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> FILE_BYTES_READ=45452916
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> HDFS_BYTES_READ=120672701
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> FILE_BYTES_WRITTEN=106567950
>> INFO org.apache.hadoop.mapred.JobClient (main):
>> HDFS_BYTES_WRITTEN=51234800
>> INFO org.apache.hadoop.mapred.JobClient (main): Map-Reduce Framework
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
>> groups=208760
>> INFO org.apache.hadoop.mapred.JobClient (main): Combine output
>> records=0
>> INFO org.apache.hadoop.mapred.JobClient (main): Map input
>> records=230201
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce shuffle
>> bytes=60461985
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce output
>> records=208760
>> INFO org.apache.hadoop.mapred.JobClient (main): Spilled
>> Records=13643340
>> INFO org.apache.hadoop.mapred.JobClient (main): Map output
>> bytes=136433400
>> INFO org.apache.hadoop.mapred.JobClient (main): Combine input
>> records=0
>> INFO org.apache.hadoop.mapred.JobClient (main): Map output
>> records=6821670
>> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
>> records=6821670
>> INFO org.apache.mahout.common.AbstractJob (main): Command line arguments:
>> {--endPhase=2147483647, --maxSimilaritiesPerRow=501,
>> --numberOfColumns=230201, --similarityClassname=SIMILARITY_LOGLIKELIHOOD,
>> --startPhase=0, --tempDir=temp}
>> INFO org.apache.hadoop.mapred.JobClient (main): Default number of map
>> tasks: 2
>> INFO org.apache.hadoop.mapred.JobClient (main): Default number of reduce
>> tasks: 24
>> INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total
>> input paths to process : 24
>> INFO org.apache.hadoop.mapred.JobClient (main): Running job:
>> job_201104130912_0005
>> INFO org.apache.hadoop.mapred.JobClient (main): map 0% reduce 0%
>> INFO org.apache.hadoop.mapred.JobClient (main): map 4% reduce 0%
>>
>>
>>> By the way, I see that you're located in Berlin, I have some free time in
>>> the next 2 weeks if you want we could meet for a coffee and you'll get some
>>> free consultation!
>>>
>>> It would be really great to meet you, but only the head office is in
>> Berlin. I am in Dresden and althougth that is not far away, it does not look
>> like I can go to Berlin. Maybe it works out later when I visit the
>> headquarter. I am sure you can explain a lot to me.
>>
>>
>>
>> Thanks in advance
>> Thomas
>>
>>
>>
>>
>>
>>
>>
>>>
>>> On 14.04.2011 12:18, Thomas Rewig wrote:
>>>
>>>> Hello
>>>> right now I'm testing Mahout (taste) Jobs on AWS EMR.
>>>> I wonder if anyone does have any experience with the best cluster size
>>>> and the best EC2 instances. Are there any best practices for mahout (taste)
>>>> jobs?
>>>>
>>>> On my first test I used a small 22 MB user-item-model an compute an
>>>> ItemSimilarityJob with 3 small EC2 instances:
>>>>
>>>> ruby elastic-mapreduce --create --alive --slave-instance-type m1.small
>>>> --master-instance-type m1.small --num-instances 3 --name
>>>> mahout-0.5-itemSimJob-TEST
>>>>
>>>>
>>>> ruby elastic-mapreduce
>>>> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
>>>> --main-class
>>>> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
>>>> --arg -i --arg s3://some-uri/input/data_small_in.csv
>>>> --arg -o --arg s3://some-uri/output/data_out_small.csv
>>>> --arg -s
>>>> --arg SIMILARITY_LOGLIKELIHOOD
>>>> --arg -m
>>>> --arg 500
>>>> --arg -mo
>>>> --arg 500
>>>> -j JobId
>>>>
>>>> Here everything worked well, even if it took a few minutes.
>>>>
>>>> In a second test I used a bigger 200 MB user-item-model and do the same
>>>> with a cluster of large instances:
>>>>
>>>> ruby elastic-mapreduce --create --alive --slave-instance-type m1.large
>>>> --master-instance-type m1.large --num-instances 8 --name
>>>> mahout-0.5-itemSimJob-TEST2
>>>>
>>>> I logged in on the masterNode with ssh and looked at the syslog. First a
>>>> few hours everything looked ok and then it seems to stops at a 63% reduce
>>>> step. I wait a few hours but nothing happend and so i terminate the job. I
>>>> even couldn't find any errors in the logs.
>>>>
>>>> So here my questions:
>>>> 1. are the any proved best practice clustersizes and instancetypes
>>>> (Standard- or High-Memory- or High-CPU-Instances) that work fine for big
>>>> recommender jobs, or do I have to test it for every different job I use?
>>>> 2. would it have some positiv effect if I split my big data_in.csv into
>>>> many small csv's?
>>>>
>>>> Do anyone have any experience with it and have some hints?
>>>>
>>>> Thanks in advance
>>>> Thomas
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Sebastian Schelter <ss...@apache.org>.
Hi Thomas,
I'd say now the long running time comes from items from dataset two that
have lots of user-preferences. Seems like your data is to dense to
compare all pairs of users with ItemSimilarityJob.
What exactly is the problem you're trying to solve with computing
similar users? Do you need that as input for the computation of
recommendations? Maybe we'll find another approach for you on this list.
--sebastian
On 14.04.2011 16:59, Thomas Rewig wrote:
> Hi Sebastian
>
> in my datamodel are 17733658 Datapoints, there are 230116 unique users
> (U(Ix)) and 208760 unique items I(Ux).
> The datapoints are in some way dense and sparse because I test to
> merge 2 Datasets and invert the result, so I can use the
> ItemSimilarityJob:
>
> e.g.:
>
> I = Item
> U = User
>
> Dataset1 (the sparse one):
> I1 I2 I3 I4
> U1 9 8
> U2 7 4
> U3 8 5
> U5 5 9
>
> Dataset2 (the dense one, but has much less Items than Dataset1):
>
> I5 I6
> U1 1 2
> U2 3 2
> U3 2
> U4 5 3
> U5 1 1
>
> Invert Dataset(1+2) so Users are Items an vice versa:
>
> I(U1) I(U2) I(U3) I(U4) I(U5)
> U(I1) 9 7 5
> U(I2) 8
> U(I3) 4 9
> U(I4) 8 5
> U(I5) 1 3 2 5 1
> U(I6) 2 2 3 1
>
> So Yes your right because of this invertation I have users with a lots
> of preferences (nearly the number of users in Dataset2), and I can
> understand why the system seems to stop.
>
> Maybe the Invertation of the Data isn't a good way for this purpose
> and i have to write my own UserUserSimilartityJob. (ok in the moment I
> have no idea how to do this, because I just started with hadoop and
> mapreduce, but I can try ;-) ).
>
> Do you have some other hints I can try?
>
>>
>> Can you same how many datapoints your data contains and how dense
>> those are? 200MB doesn't seem that much, it shouldn't take hours with
>> 8 m1.large instances.
>>
>> Can you give us the values of the following counters?
>>
>> MaybePruneRowsMapper: Elements.USED
>> MaybePruneRowsMapper: Elements.NEGLECTED
>>
>> CooccurrencesMapper: Counter.COOCCURRENCES
>
> I'm not sure, if I find the data in the logs you want, but maybe this
> log-sample helps:
>
> MaybePruneRowsMapper: Elements.NEGLECTED = 8798627
> MaybePruneRowsMapper: Elements.USED = 6821670
>
> I cant find Counter.COOCCURRENCES
>
>
> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 92%
> INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 100%
> INFO org.apache.hadoop.mapred.JobClient (main): Job complete:
> job_201104130912_0004
> INFO org.apache.hadoop.mapred.JobClient (main): Counters: 20
> INFO org.apache.hadoop.mapred.JobClient (main):
> org.apache.mahout.cf.taste.hadoop.MaybePruneRowsMapper$Elements
> INFO org.apache.hadoop.mapred.JobClient (main): NEGLECTED=8798627
> INFO org.apache.hadoop.mapred.JobClient (main): USED=6821670
> INFO org.apache.hadoop.mapred.JobClient (main): Job Counters
> INFO org.apache.hadoop.mapred.JobClient (main): Launched reduce
> tasks=24
> INFO org.apache.hadoop.mapred.JobClient (main): Rack-local map
> tasks=3
> INFO org.apache.hadoop.mapred.JobClient (main): Launched map tasks=24
> INFO org.apache.hadoop.mapred.JobClient (main): Data-local map
> tasks=21
> INFO org.apache.hadoop.mapred.JobClient (main): FileSystemCounters
> INFO org.apache.hadoop.mapred.JobClient (main):
> FILE_BYTES_READ=45452916
> INFO org.apache.hadoop.mapred.JobClient (main):
> HDFS_BYTES_READ=120672701
> INFO org.apache.hadoop.mapred.JobClient (main):
> FILE_BYTES_WRITTEN=106567950
> INFO org.apache.hadoop.mapred.JobClient (main):
> HDFS_BYTES_WRITTEN=51234800
> INFO org.apache.hadoop.mapred.JobClient (main): Map-Reduce Framework
> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
> groups=208760
> INFO org.apache.hadoop.mapred.JobClient (main): Combine output
> records=0
> INFO org.apache.hadoop.mapred.JobClient (main): Map input
> records=230201
> INFO org.apache.hadoop.mapred.JobClient (main): Reduce shuffle
> bytes=60461985
> INFO org.apache.hadoop.mapred.JobClient (main): Reduce output
> records=208760
> INFO org.apache.hadoop.mapred.JobClient (main): Spilled
> Records=13643340
> INFO org.apache.hadoop.mapred.JobClient (main): Map output
> bytes=136433400
> INFO org.apache.hadoop.mapred.JobClient (main): Combine input
> records=0
> INFO org.apache.hadoop.mapred.JobClient (main): Map output
> records=6821670
> INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
> records=6821670
> INFO org.apache.mahout.common.AbstractJob (main): Command line
> arguments: {--endPhase=2147483647, --maxSimilaritiesPerRow=501,
> --numberOfColumns=230201,
> --similarityClassname=SIMILARITY_LOGLIKELIHOOD, --startPhase=0,
> --tempDir=temp}
> INFO org.apache.hadoop.mapred.JobClient (main): Default number of map
> tasks: 2
> INFO org.apache.hadoop.mapred.JobClient (main): Default number of
> reduce tasks: 24
> INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main):
> Total input paths to process : 24
> INFO org.apache.hadoop.mapred.JobClient (main): Running job:
> job_201104130912_0005
> INFO org.apache.hadoop.mapred.JobClient (main): map 0% reduce 0%
> INFO org.apache.hadoop.mapred.JobClient (main): map 4% reduce 0%
>
>>
>> By the way, I see that you're located in Berlin, I have some free
>> time in the next 2 weeks if you want we could meet for a coffee and
>> you'll get some free consultation!
>>
> It would be really great to meet you, but only the head office is in
> Berlin. I am in Dresden and althougth that is not far away, it does
> not look like I can go to Berlin. Maybe it works out later when I
> visit the headquarter. I am sure you can explain a lot to me.
>
>
>
> Thanks in advance
> Thomas
>
>
>
>
>
>
>>
>>
>> On 14.04.2011 12:18, Thomas Rewig wrote:
>>> Hello
>>> right now I'm testing Mahout (taste) Jobs on AWS EMR.
>>> I wonder if anyone does have any experience with the best cluster
>>> size and the best EC2 instances. Are there any best practices for
>>> mahout (taste) jobs?
>>>
>>> On my first test I used a small 22 MB user-item-model an compute an
>>> ItemSimilarityJob with 3 small EC2 instances:
>>>
>>> ruby elastic-mapreduce --create --alive --slave-instance-type
>>> m1.small --master-instance-type m1.small --num-instances 3 --name
>>> mahout-0.5-itemSimJob-TEST
>>>
>>>
>>> ruby elastic-mapreduce
>>> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
>>> --main-class
>>> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
>>> --arg -i --arg s3://some-uri/input/data_small_in.csv
>>> --arg -o --arg s3://some-uri/output/data_out_small.csv
>>> --arg -s
>>> --arg SIMILARITY_LOGLIKELIHOOD
>>> --arg -m
>>> --arg 500
>>> --arg -mo
>>> --arg 500
>>> -j JobId
>>>
>>> Here everything worked well, even if it took a few minutes.
>>>
>>> In a second test I used a bigger 200 MB user-item-model and do the
>>> same with a cluster of large instances:
>>>
>>> ruby elastic-mapreduce --create --alive --slave-instance-type
>>> m1.large --master-instance-type m1.large --num-instances 8 --name
>>> mahout-0.5-itemSimJob-TEST2
>>>
>>> I logged in on the masterNode with ssh and looked at the syslog.
>>> First a few hours everything looked ok and then it seems to stops at
>>> a 63% reduce step. I wait a few hours but nothing happend and so i
>>> terminate the job. I even couldn't find any errors in the logs.
>>>
>>> So here my questions:
>>> 1. are the any proved best practice clustersizes and instancetypes
>>> (Standard- or High-Memory- or High-CPU-Instances) that work fine for
>>> big recommender jobs, or do I have to test it for every different
>>> job I use?
>>> 2. would it have some positiv effect if I split my big data_in.csv
>>> into many small csv's?
>>>
>>> Do anyone have any experience with it and have some hints?
>>>
>>> Thanks in advance
>>> Thomas
>>>
>>>
>>>
>>
>>
>
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Thomas Rewig <tr...@mufin.com>.
Hi Sebastian
in my datamodel are 17733658 Datapoints, there are 230116 unique users
(U(Ix)) and 208760 unique items I(Ux).
The datapoints are in some way dense and sparse because I test to merge
2 Datasets and invert the result, so I can use the ItemSimilarityJob:
e.g.:
I = Item
U = User
Dataset1 (the sparse one):
I1 I2 I3 I4
U1 9 8
U2 7 4
U3 8 5
U5 5 9
Dataset2 (the dense one, but has much less Items than Dataset1):
I5 I6
U1 1 2
U2 3 2
U3 2
U4 5 3
U5 1 1
Invert Dataset(1+2) so Users are Items an vice versa:
I(U1) I(U2) I(U3) I(U4) I(U5)
U(I1) 9 7 5
U(I2) 8
U(I3) 4 9
U(I4) 8 5
U(I5) 1 3 2 5 1
U(I6) 2 2 3 1
So Yes your right because of this invertation I have users with a lots
of preferences (nearly the number of users in Dataset2), and I can
understand why the system seems to stop.
Maybe the Invertation of the Data isn't a good way for this purpose and
i have to write my own UserUserSimilartityJob. (ok in the moment I have
no idea how to do this, because I just started with hadoop and
mapreduce, but I can try ;-) ).
Do you have some other hints I can try?
>
> Can you same how many datapoints your data contains and how dense
> those are? 200MB doesn't seem that much, it shouldn't take hours with
> 8 m1.large instances.
>
> Can you give us the values of the following counters?
>
> MaybePruneRowsMapper: Elements.USED
> MaybePruneRowsMapper: Elements.NEGLECTED
>
> CooccurrencesMapper: Counter.COOCCURRENCES
I'm not sure, if I find the data in the logs you want, but maybe this
log-sample helps:
MaybePruneRowsMapper: Elements.NEGLECTED = 8798627
MaybePruneRowsMapper: Elements.USED = 6821670
I cant find Counter.COOCCURRENCES
INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 92%
INFO org.apache.hadoop.mapred.JobClient (main): map 100% reduce 100%
INFO org.apache.hadoop.mapred.JobClient (main): Job complete:
job_201104130912_0004
INFO org.apache.hadoop.mapred.JobClient (main): Counters: 20
INFO org.apache.hadoop.mapred.JobClient (main):
org.apache.mahout.cf.taste.hadoop.MaybePruneRowsMapper$Elements
INFO org.apache.hadoop.mapred.JobClient (main): NEGLECTED=8798627
INFO org.apache.hadoop.mapred.JobClient (main): USED=6821670
INFO org.apache.hadoop.mapred.JobClient (main): Job Counters
INFO org.apache.hadoop.mapred.JobClient (main): Launched reduce tasks=24
INFO org.apache.hadoop.mapred.JobClient (main): Rack-local map tasks=3
INFO org.apache.hadoop.mapred.JobClient (main): Launched map tasks=24
INFO org.apache.hadoop.mapred.JobClient (main): Data-local map tasks=21
INFO org.apache.hadoop.mapred.JobClient (main): FileSystemCounters
INFO org.apache.hadoop.mapred.JobClient (main): FILE_BYTES_READ=45452916
INFO org.apache.hadoop.mapred.JobClient (main):
HDFS_BYTES_READ=120672701
INFO org.apache.hadoop.mapred.JobClient (main):
FILE_BYTES_WRITTEN=106567950
INFO org.apache.hadoop.mapred.JobClient (main):
HDFS_BYTES_WRITTEN=51234800
INFO org.apache.hadoop.mapred.JobClient (main): Map-Reduce Framework
INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
groups=208760
INFO org.apache.hadoop.mapred.JobClient (main): Combine output records=0
INFO org.apache.hadoop.mapred.JobClient (main): Map input records=230201
INFO org.apache.hadoop.mapred.JobClient (main): Reduce shuffle
bytes=60461985
INFO org.apache.hadoop.mapred.JobClient (main): Reduce output
records=208760
INFO org.apache.hadoop.mapred.JobClient (main): Spilled Records=13643340
INFO org.apache.hadoop.mapred.JobClient (main): Map output
bytes=136433400
INFO org.apache.hadoop.mapred.JobClient (main): Combine input records=0
INFO org.apache.hadoop.mapred.JobClient (main): Map output
records=6821670
INFO org.apache.hadoop.mapred.JobClient (main): Reduce input
records=6821670
INFO org.apache.mahout.common.AbstractJob (main): Command line
arguments: {--endPhase=2147483647, --maxSimilaritiesPerRow=501,
--numberOfColumns=230201,
--similarityClassname=SIMILARITY_LOGLIKELIHOOD, --startPhase=0,
--tempDir=temp}
INFO org.apache.hadoop.mapred.JobClient (main): Default number of map
tasks: 2
INFO org.apache.hadoop.mapred.JobClient (main): Default number of reduce
tasks: 24
INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat (main): Total
input paths to process : 24
INFO org.apache.hadoop.mapred.JobClient (main): Running job:
job_201104130912_0005
INFO org.apache.hadoop.mapred.JobClient (main): map 0% reduce 0%
INFO org.apache.hadoop.mapred.JobClient (main): map 4% reduce 0%
>
> By the way, I see that you're located in Berlin, I have some free time
> in the next 2 weeks if you want we could meet for a coffee and you'll
> get some free consultation!
>
It would be really great to meet you, but only the head office is in
Berlin. I am in Dresden and althougth that is not far away, it does not
look like I can go to Berlin. Maybe it works out later when I visit the
headquarter. I am sure you can explain a lot to me.
Thanks in advance
Thomas
>
>
> On 14.04.2011 12:18, Thomas Rewig wrote:
>> Hello
>> right now I'm testing Mahout (taste) Jobs on AWS EMR.
>> I wonder if anyone does have any experience with the best cluster
>> size and the best EC2 instances. Are there any best practices for
>> mahout (taste) jobs?
>>
>> On my first test I used a small 22 MB user-item-model an compute an
>> ItemSimilarityJob with 3 small EC2 instances:
>>
>> ruby elastic-mapreduce --create --alive --slave-instance-type
>> m1.small --master-instance-type m1.small --num-instances 3 --name
>> mahout-0.5-itemSimJob-TEST
>>
>>
>> ruby elastic-mapreduce
>> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
>> --main-class
>> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
>> --arg -i --arg s3://some-uri/input/data_small_in.csv
>> --arg -o --arg s3://some-uri/output/data_out_small.csv
>> --arg -s
>> --arg SIMILARITY_LOGLIKELIHOOD
>> --arg -m
>> --arg 500
>> --arg -mo
>> --arg 500
>> -j JobId
>>
>> Here everything worked well, even if it took a few minutes.
>>
>> In a second test I used a bigger 200 MB user-item-model and do the
>> same with a cluster of large instances:
>>
>> ruby elastic-mapreduce --create --alive --slave-instance-type
>> m1.large --master-instance-type m1.large --num-instances 8 --name
>> mahout-0.5-itemSimJob-TEST2
>>
>> I logged in on the masterNode with ssh and looked at the syslog.
>> First a few hours everything looked ok and then it seems to stops at
>> a 63% reduce step. I wait a few hours but nothing happend and so i
>> terminate the job. I even couldn't find any errors in the logs.
>>
>> So here my questions:
>> 1. are the any proved best practice clustersizes and instancetypes
>> (Standard- or High-Memory- or High-CPU-Instances) that work fine for
>> big recommender jobs, or do I have to test it for every different
>> job I use?
>> 2. would it have some positiv effect if I split my big data_in.csv
>> into many small csv's?
>>
>> Do anyone have any experience with it and have some hints?
>>
>> Thanks in advance
>> Thomas
>>
>>
>>
>
>
Re: Hints for Best Practices for Jobs with amazon EMR
Posted by Sebastian Schelter <ss...@apache.org>.
Hi Thomas,
Can you same how many datapoints your data contains and how dense those
are? 200MB doesn't seem that much, it shouldn't take hours with 8
m1.large instances.
Can you give us the values of the following counters?
MaybePruneRowsMapper: Elements.USED
MaybePruneRowsMapper: Elements.NEGLECTED
CooccurrencesMapper: Counter.COOCCURRENCES
You have set "maxCooccurrencesPerItem" to a high value of 500, this
parameter controls how much the data is pruned. It is not an absolute
number but only an estimate as the pruning is run independently on each
mapper machine (I'd love to have a smarter solution for this).
There is a step in the similarity computation in which all pairs of
preferred items of each single user need to be mapped out. This number
grows quadratically so a single user with an enormous amount of items
can slow down the whole computation. Maybe that's what's happening here,
I see that you work in the music domain, it seems plausible that there
are users with lots of preferences. You could also try to run the job
with the default value for maxCooccurrencesPerItem and see how long it
takes then.
By the way, I see that you're located in Berlin, I have some free time
in the next 2 weeks if you want we could meet for a coffee and you'll
get some free consultation!
--sebastian
On 14.04.2011 12:18, Thomas Rewig wrote:
> Hello
> right now I'm testing Mahout (taste) Jobs on AWS EMR.
> I wonder if anyone does have any experience with the best cluster size
> and the best EC2 instances. Are there any best practices for mahout
> (taste) jobs?
>
> On my first test I used a small 22 MB user-item-model an compute an
> ItemSimilarityJob with 3 small EC2 instances:
>
> ruby elastic-mapreduce --create --alive --slave-instance-type m1.small
> --master-instance-type m1.small --num-instances 3 --name
> mahout-0.5-itemSimJob-TEST
>
>
> ruby elastic-mapreduce
> --jar s3://some-uri/mahout/mahout-core-0.5-Snapshot-job.jar
> --main-class
> org.apache.mahout.cf.taste.hadoop.similarity.item.ItemSimilarityJob
> --arg -i --arg s3://some-uri/input/data_small_in.csv
> --arg -o --arg s3://some-uri/output/data_out_small.csv
> --arg -s
> --arg SIMILARITY_LOGLIKELIHOOD
> --arg -m
> --arg 500
> --arg -mo
> --arg 500
> -j JobId
>
> Here everything worked well, even if it took a few minutes.
>
> In a second test I used a bigger 200 MB user-item-model and do the
> same with a cluster of large instances:
>
> ruby elastic-mapreduce --create --alive --slave-instance-type m1.large
> --master-instance-type m1.large --num-instances 8 --name
> mahout-0.5-itemSimJob-TEST2
>
> I logged in on the masterNode with ssh and looked at the syslog. First
> a few hours everything looked ok and then it seems to stops at a 63%
> reduce step. I wait a few hours but nothing happend and so i terminate
> the job. I even couldn't find any errors in the logs.
>
> So here my questions:
> 1. are the any proved best practice clustersizes and instancetypes
> (Standard- or High-Memory- or High-CPU-Instances) that work fine for
> big recommender jobs, or do I have to test it for every different job
> I use?
> 2. would it have some positiv effect if I split my big data_in.csv
> into many small csv's?
>
> Do anyone have any experience with it and have some hints?
>
> Thanks in advance
> Thomas
>
>
>