Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Hi,

I have 30.000 items and the computation takes more than 2h on a 
pseudo-cluster, which is too long in my case.

I think of some ways to reduce the execution time of RowSimilarityJob 
and I wonder if some of you have implemented them and how, or explored 
other ways.
1. tune the JVM
2. developing an in memory implementation (i.e. without hadoop)
3. reduce the size of the matrix (by removing those which have no words 
in common, for example)
4. run on real hadoop cluster with several nodes (does anyone have an 
idea of ​​the number of nodes to make it interesting)

Thanks for your help,
Yann.

Re: Need to reduce execution time of RowSimilarityJob

[Anna Lahoud <an...@gmail.com>]

It can take awhile to tune the parameters but you definitely should be able
to run the RowSimilarityJob in minutes on a set around that size. Multiple
reducers is really important, as is the density of the input matrix.
Another useful tuning parameter we used was to adjust some of the
RowSimilarityJob configuration settings like setting max similarity outputs
per row.

On Tue, Sep 18, 2012 at 12:34 PM, Sean Owen <sr...@gmail.com> wrote:

> That sounds quite slow. You're definitely computing item-item similarity?
> if users are rows, then this job is computing user-user similarity.
>
> An item-based recommender isn't necessary per se, just item similarity. The
> ItemBasedRecommender has a convenience method to just find the top N most
> similar items. If your scale is such that working in memory is feasible,
> that is by far the best answer.
>
> Sean
>
> On Tue, Sep 18, 2012 at 4:14 PM, yamo93 <ya...@gmail.com> wrote:
>
> > Hi Sean,
> >
> > My need is to compute document similarity (30.000 docs) and more
> > precisely, to find the n most similar docs.
> > As written above, i use RowSimilarityJob but it takes 2h+ to compute.
> >
> > Seb suggest to use an item-item recommender with input data (term,
> > document, tf-idf).
> >
> > Rgds,
> > Y.
> >
> >
> > On 09/18/2012 04:21 PM, Sean Owen wrote:
> >
> >> If you are computing user-user similarity, the number of items is not
> >> nearly as important as the number of users. If you have 1M users, then
> >> computing about 500 billion user-user similarities is going to take a
> long
> >> time no matter what.
> >>
> >> CSV is the input for both Hadoop-based and non-Hadoop-based
> >> implementations. The Hadoop-based implementation converts to vectors.
> You
> >> can inject vectors directly if you want, there. But you need CSV for the
> >> non-Hadoop code.
> >>
> >> There are a number of tuning params in the Hadoop implementation (and
> >> similar but different hooks in the non-Hadoop implementation) that let
> you
> >> prune data at several stages. This is the most important thing for
> speed.
> >> Yes, removing stop-words falls in that category.
> >>
> >> Tuning the JVM helps but marginally. More Hadoop nodes helps, linearly.
> >>
> >> On Tue, Sep 18, 2012 at 1:49 PM, yamo93 <ya...@gmail.com> wrote:
> >>
> >>  Hi,
> >>>
> >>> I have 30.000 items and the computation takes more than 2h on a
> >>> pseudo-cluster, which is too long in my case.
> >>>
> >>> I think of some ways to reduce the execution time of RowSimilarityJob
> and
> >>> I wonder if some of you have implemented them and how, or explored
> other
> >>> ways.
> >>> 1. tune the JVM
> >>> 2. developing an in memory implementation (i.e. without hadoop)
> >>> 3. reduce the size of the matrix (by removing those which have no words
> >>> in
> >>> common, for example)
> >>> 4. run on real hadoop cluster with several nodes (does anyone have an
> >>> idea
> >>> of the number of nodes to make it interesting)
> >>>
> >>> Thanks for your help,
> >>> Yann.
> >>>
> >>>
> >
>

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

That sounds quite slow. You're definitely computing item-item similarity?
if users are rows, then this job is computing user-user similarity.

An item-based recommender isn't necessary per se, just item similarity. The
ItemBasedRecommender has a convenience method to just find the top N most
similar items. If your scale is such that working in memory is feasible,
that is by far the best answer.

Sean

On Tue, Sep 18, 2012 at 4:14 PM, yamo93 <ya...@gmail.com> wrote:

> Hi Sean,
>
> My need is to compute document similarity (30.000 docs) and more
> precisely, to find the n most similar docs.
> As written above, i use RowSimilarityJob but it takes 2h+ to compute.
>
> Seb suggest to use an item-item recommender with input data (term,
> document, tf-idf).
>
> Rgds,
> Y.
>
>
> On 09/18/2012 04:21 PM, Sean Owen wrote:
>
>> If you are computing user-user similarity, the number of items is not
>> nearly as important as the number of users. If you have 1M users, then
>> computing about 500 billion user-user similarities is going to take a long
>> time no matter what.
>>
>> CSV is the input for both Hadoop-based and non-Hadoop-based
>> implementations. The Hadoop-based implementation converts to vectors. You
>> can inject vectors directly if you want, there. But you need CSV for the
>> non-Hadoop code.
>>
>> There are a number of tuning params in the Hadoop implementation (and
>> similar but different hooks in the non-Hadoop implementation) that let you
>> prune data at several stages. This is the most important thing for speed.
>> Yes, removing stop-words falls in that category.
>>
>> Tuning the JVM helps but marginally. More Hadoop nodes helps, linearly.
>>
>> On Tue, Sep 18, 2012 at 1:49 PM, yamo93 <ya...@gmail.com> wrote:
>>
>>  Hi,
>>>
>>> I have 30.000 items and the computation takes more than 2h on a
>>> pseudo-cluster, which is too long in my case.
>>>
>>> I think of some ways to reduce the execution time of RowSimilarityJob and
>>> I wonder if some of you have implemented them and how, or explored other
>>> ways.
>>> 1. tune the JVM
>>> 2. developing an in memory implementation (i.e. without hadoop)
>>> 3. reduce the size of the matrix (by removing those which have no words
>>> in
>>> common, for example)
>>> 4. run on real hadoop cluster with several nodes (does anyone have an
>>> idea
>>> of the number of nodes to make it interesting)
>>>
>>> Thanks for your help,
>>> Yann.
>>>
>>>
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Hi Sean,

My need is to compute document similarity (30.000 docs) and more 
precisely, to find the n most similar docs.
As written above, i use RowSimilarityJob but it takes 2h+ to compute.

Seb suggest to use an item-item recommender with input data (term, 
document, tf-idf).

Rgds,
Y.

On 09/18/2012 04:21 PM, Sean Owen wrote:
> If you are computing user-user similarity, the number of items is not
> nearly as important as the number of users. If you have 1M users, then
> computing about 500 billion user-user similarities is going to take a long
> time no matter what.
>
> CSV is the input for both Hadoop-based and non-Hadoop-based
> implementations. The Hadoop-based implementation converts to vectors. You
> can inject vectors directly if you want, there. But you need CSV for the
> non-Hadoop code.
>
> There are a number of tuning params in the Hadoop implementation (and
> similar but different hooks in the non-Hadoop implementation) that let you
> prune data at several stages. This is the most important thing for speed.
> Yes, removing stop-words falls in that category.
>
> Tuning the JVM helps but marginally. More Hadoop nodes helps, linearly.
>
> On Tue, Sep 18, 2012 at 1:49 PM, yamo93 <ya...@gmail.com> wrote:
>
>> Hi,
>>
>> I have 30.000 items and the computation takes more than 2h on a
>> pseudo-cluster, which is too long in my case.
>>
>> I think of some ways to reduce the execution time of RowSimilarityJob and
>> I wonder if some of you have implemented them and how, or explored other
>> ways.
>> 1. tune the JVM
>> 2. developing an in memory implementation (i.e. without hadoop)
>> 3. reduce the size of the matrix (by removing those which have no words in
>> common, for example)
>> 4. run on real hadoop cluster with several nodes (does anyone have an idea
>> of the number of nodes to make it interesting)
>>
>> Thanks for your help,
>> Yann.
>>

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

If you are computing user-user similarity, the number of items is not
nearly as important as the number of users. If you have 1M users, then
computing about 500 billion user-user similarities is going to take a long
time no matter what.

CSV is the input for both Hadoop-based and non-Hadoop-based
implementations. The Hadoop-based implementation converts to vectors. You
can inject vectors directly if you want, there. But you need CSV for the
non-Hadoop code.

There are a number of tuning params in the Hadoop implementation (and
similar but different hooks in the non-Hadoop implementation) that let you
prune data at several stages. This is the most important thing for speed.
Yes, removing stop-words falls in that category.

Tuning the JVM helps but marginally. More Hadoop nodes helps, linearly.

On Tue, Sep 18, 2012 at 1:49 PM, yamo93 <ya...@gmail.com> wrote:

> Hi,
>
> I have 30.000 items and the computation takes more than 2h on a
> pseudo-cluster, which is too long in my case.
>
> I think of some ways to reduce the execution time of RowSimilarityJob and
> I wonder if some of you have implemented them and how, or explored other
> ways.
> 1. tune the JVM
> 2. developing an in memory implementation (i.e. without hadoop)
> 3. reduce the size of the matrix (by removing those which have no words in
> common, for example)
> 4. run on real hadoop cluster with several nodes (does anyone have an idea
> of the number of nodes to make it interesting)
>
> Thanks for your help,
> Yann.
>

Re: Need to reduce execution time of RowSimilarityJob

[Anna Lahoud <an...@gmail.com>]

Yenn,

I have spent a great deal of time working with it recently and can offer
you some quick tips.


   1. The density of the matrix is the biggest factor. Use sparse vectors
   if you can. It will reduce the time.
   2. Set a larger number of reducers to decrease the processing time per
   node. I have had job failures when a single node cannot merge the results.
   3. If you are using the output from seq2sparse, using the tfidf vectors
   as input can be significantly less dense, depending on the parameters you
   used to run seq2sparse.

Using these suggestions, we were able to get a job that took many hours to
run in well under an hour on a large cluster.

Anna

On Tue, Sep 18, 2012 at 8:49 AM, yamo93 <ya...@gmail.com> wrote:

> Hi,
>
> I have 30.000 items and the computation takes more than 2h on a
> pseudo-cluster, which is too long in my case.
>
> I think of some ways to reduce the execution time of RowSimilarityJob and
> I wonder if some of you have implemented them and how, or explored other
> ways.
> 1. tune the JVM
> 2. developing an in memory implementation (i.e. without hadoop)
> 3. reduce the size of the matrix (by removing those which have no words in
> common, for example)
> 4. run on real hadoop cluster with several nodes (does anyone have an idea
> of the number of nodes to make it interesting)
>
> Thanks for your help,
> Yann.
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

In MIA, it seems impossible to setPreferenceInferrer for ItemSimilarity, 
isn't it ?

On 10/01/2012 12:37 PM, Sean Owen wrote:
> Yes, this is one of the weaknesses of this particular flavor of this
> particular similarity metric. The more sparse, the worse the problem
> is in general. There are some band-aid solutions like applying some
> kind of weight against similarities based on small intersection size.
> Or you can pretend that missing values are 0 (PreferenceInferrer),
> which can introduce its own problems, or perhaps some mean value.
>
> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>> Thanks for replying.
>>
>> So, documents with only one word in common have more chance to be similar
>> than documents with more words in common, right ?
>>
>>
>>
>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>> Similar items, right? You should look at the vectors that have 1.0
>>> similarity and see if they are in fact collinear. This is still by far
>>> the most likely explanation. Remember that the vector similarity is
>>> computed over elements that exist in both vectors only. They just have
>>> to have 2 identical values for this to happen.
>>>
>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>> It sounds like a bug somewhere.
>>>>
>>>>
>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>> angle between them). It's possible there are several of these, and so
>>>>> their 1.0 similarities dominate the result.
>>>>>
>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>> I saw something strange : all recommended items, returned by
>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>> Is it normal ?
>>

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

This is not true.


On 01.10.2012 21:52, bangbig wrote:
> I think it's better to understand how the RowSimilarityJob gets the result.
> For two items, 
> itemA, 0, 0,   a1, a2, a3, 0
> itemB, 0, b1, b2, b3, 0  , 0
> when computing, it just uses the blue parts of the vectors.
> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)  * sqrt(b2*b2 + b3*b3))
> 1) if itemA and itemB have just one common word, the result is 1;
> 2) if the values of the vectors are almost the same, the value would also be nearly 1;
> and for the two cases above, I think you can consider to use association rules to consider the problem.
> 
> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>> It seems that RowSimilarityJob does not have the same weakness, but i 
>> also use CosineSimilarity. Why ?
>>
>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>> Yes, this is one of the weaknesses of this particular flavor of this
>>> particular similarity metric. The more sparse, the worse the problem
>>> is in general. There are some band-aid solutions like applying some
>>> kind of weight against similarities based on small intersection size.
>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>> which can introduce its own problems, or perhaps some mean value.
>>>
>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>> Thanks for replying.
>>>>
>>>> So, documents with only one word in common have more chance to be similar
>>>> than documents with more words in common, right ?
>>>>
>>>>
>>>>
>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>> similarity and see if they are in fact collinear. This is still by far
>>>>> the most likely explanation. Remember that the vector similarity is
>>>>> computed over elements that exist in both vectors only. They just have
>>>>> to have 2 identical values for this to happen.
>>>>>
>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>> It sounds like a bug somewhere.
>>>>>>
>>>>>>
>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>> angle between them). It's possible there are several of these, and so
>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>
>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>> Is it normal ?
>>>>
>>
> 

Re:Re: Need to reduce execution time of RowSimilarityJob

[bangbig <li...@163.com>]

I think I will try to implement a version this night!
I have wrote one package that works directly on Hadoop, dealing with data extracted from HIVE.

At 2012-10-03 03:01:32,yamo93 <ya...@gmail.com> wrote:
>You'll find in attachment a class that implements cosine distance as in 
>hadoop. I've just implemented the core method : itemSimilarity.
>
>On 10/02/2012 02:59 PM, yamo93 wrote:
>> Ok, i'll try this evening.
>>
>> On 10/02/2012 02:39 PM, Sebastian Schelter wrote:
>>> Would you like to create a patch for this?
>>>
>>> On 02.10.2012 14:36, yamo93 wrote:
>>>> +1 for the implementation over all entries.
>>>>
>>>> On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>>>>> I don't see why documents with only one word in common should have a
>>>>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked 
>>>>> if you
>>>>> specify a threshold for the similarity.
>>>>>
>>>>> UncenteredCosineSimilarity works on matching entries only, which is
>>>>> problematic for documents, as empty entries have a meaning (0 term
>>>>> occurrences) as opposed to collaborative filtering data.
>>>>>
>>>>> Maybe we should remove UncenteredCosine andd create another similarity
>>>>> implementation that computes the cosine correctly over all entries.
>>>>>
>>>>> --sebastian
>>>>>
>>>>>
>>>>> On 02.10.2012 10:08, yamo93 wrote:
>>>>>> Hello Seb,
>>>>>>
>>>>>> In my comprehension, the algorithm is the same (except the 
>>>>>> normalization
>>>>>> part) as UncenteredCosine (with the drawback that vectors with 
>>>>>> only one
>>>>>> word in common have a distance of 1.0)... but the result are quite
>>>>>> different (is this just an effect of the consider() method which 
>>>>>> remove
>>>>>> irrelevant values ?) ...
>>>>>>
>>>>>> I looked at the code but there is quite nothing in
>>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity, 
>>>>>>
>>>>>>
>>>>>> the code seems to be in SimilarityReducer which is not so simple to
>>>>>> understand ...
>>>>>>
>>>>>> Thanks for helping,
>>>>>>
>>>>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>>>>> similarity between the whole vectors.
>>>>>>>
>>>>>>> see
>>>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity 
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> for details
>>>>>>>
>>>>>>> At first both vectors are scaled to unit length in normalize() and
>>>>>>> after
>>>>>>> this their dot product in similarity() (which can be computed from
>>>>>>> elements that exist in both vectors) gives the cosine between those.
>>>>>>>
>>>>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>>>>> result.
>>>>>>>> For two items,
>>>>>>>> itemA, 0, 0, a1, a2, a3, 0
>>>>>>>> itemB, 0, b1, b2, b3, 0 , 0
>>>>>>>> when computing, it just uses the blue parts of the vectors.
>>>>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>>>>> * sqrt(b2*b2 + b3*b3))
>>>>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>>>>> 2) if the values of the vectors are almost the same, the value 
>>>>>>>> would
>>>>>>>> also be nearly 1;
>>>>>>>> and for the two cases above, I think you can consider to use
>>>>>>>> association rules to consider the problem.
>>>>>>>>
>>>>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> It seems that RowSimilarityJob does not have the same weakness, 
>>>>>>>>> but i
>>>>>>>>> also use CosineSimilarity. Why ?
>>>>>>>>>
>>>>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>>>>> Yes, this is one of the weaknesses of this particular flavor 
>>>>>>>>>> of this
>>>>>>>>>> particular similarity metric. The more sparse, the worse the 
>>>>>>>>>> problem
>>>>>>>>>> is in general. There are some band-aid solutions like applying 
>>>>>>>>>> some
>>>>>>>>>> kind of weight against similarities based on small intersection
>>>>>>>>>> size.
>>>>>>>>>> Or you can pretend that missing values are 0 
>>>>>>>>>> (PreferenceInferrer),
>>>>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>> Thanks for replying.
>>>>>>>>>>>
>>>>>>>>>>> So, documents with only one word in common have more chance 
>>>>>>>>>>> to be
>>>>>>>>>>> similar
>>>>>>>>>>> than documents with more words in common, right ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>>>>> Similar items, right? You should look at the vectors that 
>>>>>>>>>>>> have 1.0
>>>>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>>>>> by far
>>>>>>>>>>>> the most likely explanation. Remember that the vector
>>>>>>>>>>>> similarity is
>>>>>>>>>>>> computed over elements that exist in both vectors only. They 
>>>>>>>>>>>> just
>>>>>>>>>>>> have
>>>>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> For each item, i have 10 recommended items with a value of 
>>>>>>>>>>>>> 1.0.
>>>>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum 
>>>>>>>>>>>>>> similarity and
>>>>>>>>>>>>>> occurs when two vector are just a scalar multiple of each
>>>>>>>>>>>>>> other (0
>>>>>>>>>>>>>> angle between them). It's possible there are several of 
>>>>>>>>>>>>>> these,
>>>>>>>>>>>>>> and so
>>>>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>>>>> Is it normal ?
>>
>

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

Please don't send patches to the mailing list. Here's a guide that tells
you how to contribute to Mahout, it involves issuing a jira ticket:

https://cwiki.apache.org/MAHOUT/how-to-contribute.html

Your patch uses Java7 while Mahout is based on Java 6. Furthermore, you
can't simply throw UnsupportedOperationExceptions for most methods.

--sebastian


On 02.10.2012 21:01, yamo93 wrote:
> You'll find in attachment a class that implements cosine distance as in
> hadoop. I've just implemented the core method : itemSimilarity.
> 
> On 10/02/2012 02:59 PM, yamo93 wrote:
>> Ok, i'll try this evening.
>>
>> On 10/02/2012 02:39 PM, Sebastian Schelter wrote:
>>> Would you like to create a patch for this?
>>>
>>> On 02.10.2012 14:36, yamo93 wrote:
>>>> +1 for the implementation over all entries.
>>>>
>>>> On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>>>>> I don't see why documents with only one word in common should have a
>>>>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked
>>>>> if you
>>>>> specify a threshold for the similarity.
>>>>>
>>>>> UncenteredCosineSimilarity works on matching entries only, which is
>>>>> problematic for documents, as empty entries have a meaning (0 term
>>>>> occurrences) as opposed to collaborative filtering data.
>>>>>
>>>>> Maybe we should remove UncenteredCosine andd create another similarity
>>>>> implementation that computes the cosine correctly over all entries.
>>>>>
>>>>> --sebastian
>>>>>
>>>>>
>>>>> On 02.10.2012 10:08, yamo93 wrote:
>>>>>> Hello Seb,
>>>>>>
>>>>>> In my comprehension, the algorithm is the same (except the
>>>>>> normalization
>>>>>> part) as UncenteredCosine (with the drawback that vectors with
>>>>>> only one
>>>>>> word in common have a distance of 1.0)... but the result are quite
>>>>>> different (is this just an effect of the consider() method which
>>>>>> remove
>>>>>> irrelevant values ?) ...
>>>>>>
>>>>>> I looked at the code but there is quite nothing in
>>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>>>>>>
>>>>>>
>>>>>> the code seems to be in SimilarityReducer which is not so simple to
>>>>>> understand ...
>>>>>>
>>>>>> Thanks for helping,
>>>>>>
>>>>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>>>>> similarity between the whole vectors.
>>>>>>>
>>>>>>> see
>>>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> for details
>>>>>>>
>>>>>>> At first both vectors are scaled to unit length in normalize() and
>>>>>>> after
>>>>>>> this their dot product in similarity() (which can be computed from
>>>>>>> elements that exist in both vectors) gives the cosine between those.
>>>>>>>
>>>>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>>>>> result.
>>>>>>>> For two items,
>>>>>>>> itemA, 0, 0, a1, a2, a3, 0
>>>>>>>> itemB, 0, b1, b2, b3, 0 , 0
>>>>>>>> when computing, it just uses the blue parts of the vectors.
>>>>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>>>>> * sqrt(b2*b2 + b3*b3))
>>>>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>>>>> 2) if the values of the vectors are almost the same, the value
>>>>>>>> would
>>>>>>>> also be nearly 1;
>>>>>>>> and for the two cases above, I think you can consider to use
>>>>>>>> association rules to consider the problem.
>>>>>>>>
>>>>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> It seems that RowSimilarityJob does not have the same weakness,
>>>>>>>>> but i
>>>>>>>>> also use CosineSimilarity. Why ?
>>>>>>>>>
>>>>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>>>>> Yes, this is one of the weaknesses of this particular flavor
>>>>>>>>>> of this
>>>>>>>>>> particular similarity metric. The more sparse, the worse the
>>>>>>>>>> problem
>>>>>>>>>> is in general. There are some band-aid solutions like applying
>>>>>>>>>> some
>>>>>>>>>> kind of weight against similarities based on small intersection
>>>>>>>>>> size.
>>>>>>>>>> Or you can pretend that missing values are 0
>>>>>>>>>> (PreferenceInferrer),
>>>>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>> Thanks for replying.
>>>>>>>>>>>
>>>>>>>>>>> So, documents with only one word in common have more chance
>>>>>>>>>>> to be
>>>>>>>>>>> similar
>>>>>>>>>>> than documents with more words in common, right ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>>>>> Similar items, right? You should look at the vectors that
>>>>>>>>>>>> have 1.0
>>>>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>>>>> by far
>>>>>>>>>>>> the most likely explanation. Remember that the vector
>>>>>>>>>>>> similarity is
>>>>>>>>>>>> computed over elements that exist in both vectors only. They
>>>>>>>>>>>> just
>>>>>>>>>>>> have
>>>>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> For each item, i have 10 recommended items with a value of
>>>>>>>>>>>>> 1.0.
>>>>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum
>>>>>>>>>>>>>> similarity and
>>>>>>>>>>>>>> occurs when two vector are just a scalar multiple of each
>>>>>>>>>>>>>> other (0
>>>>>>>>>>>>>> angle between them). It's possible there are several of
>>>>>>>>>>>>>> these,
>>>>>>>>>>>>>> and so
>>>>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>>>>> Is it normal ?
>>
> 

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

You'll find in attachment a class that implements cosine distance as in 
hadoop. I've just implemented the core method : itemSimilarity.

On 10/02/2012 02:59 PM, yamo93 wrote:
> Ok, i'll try this evening.
>
> On 10/02/2012 02:39 PM, Sebastian Schelter wrote:
>> Would you like to create a patch for this?
>>
>> On 02.10.2012 14:36, yamo93 wrote:
>>> +1 for the implementation over all entries.
>>>
>>> On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>>>> I don't see why documents with only one word in common should have a
>>>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked 
>>>> if you
>>>> specify a threshold for the similarity.
>>>>
>>>> UncenteredCosineSimilarity works on matching entries only, which is
>>>> problematic for documents, as empty entries have a meaning (0 term
>>>> occurrences) as opposed to collaborative filtering data.
>>>>
>>>> Maybe we should remove UncenteredCosine andd create another similarity
>>>> implementation that computes the cosine correctly over all entries.
>>>>
>>>> --sebastian
>>>>
>>>>
>>>> On 02.10.2012 10:08, yamo93 wrote:
>>>>> Hello Seb,
>>>>>
>>>>> In my comprehension, the algorithm is the same (except the 
>>>>> normalization
>>>>> part) as UncenteredCosine (with the drawback that vectors with 
>>>>> only one
>>>>> word in common have a distance of 1.0)... but the result are quite
>>>>> different (is this just an effect of the consider() method which 
>>>>> remove
>>>>> irrelevant values ?) ...
>>>>>
>>>>> I looked at the code but there is quite nothing in
>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity, 
>>>>>
>>>>>
>>>>> the code seems to be in SimilarityReducer which is not so simple to
>>>>> understand ...
>>>>>
>>>>> Thanks for helping,
>>>>>
>>>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>>>> similarity between the whole vectors.
>>>>>>
>>>>>> see
>>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity 
>>>>>>
>>>>>>
>>>>>>
>>>>>> for details
>>>>>>
>>>>>> At first both vectors are scaled to unit length in normalize() and
>>>>>> after
>>>>>> this their dot product in similarity() (which can be computed from
>>>>>> elements that exist in both vectors) gives the cosine between those.
>>>>>>
>>>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>>>> result.
>>>>>>> For two items,
>>>>>>> itemA, 0, 0, a1, a2, a3, 0
>>>>>>> itemB, 0, b1, b2, b3, 0 , 0
>>>>>>> when computing, it just uses the blue parts of the vectors.
>>>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>>>> * sqrt(b2*b2 + b3*b3))
>>>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>>>> 2) if the values of the vectors are almost the same, the value 
>>>>>>> would
>>>>>>> also be nearly 1;
>>>>>>> and for the two cases above, I think you can consider to use
>>>>>>> association rules to consider the problem.
>>>>>>>
>>>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>>>> It seems that RowSimilarityJob does not have the same weakness, 
>>>>>>>> but i
>>>>>>>> also use CosineSimilarity. Why ?
>>>>>>>>
>>>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>>>> Yes, this is one of the weaknesses of this particular flavor 
>>>>>>>>> of this
>>>>>>>>> particular similarity metric. The more sparse, the worse the 
>>>>>>>>> problem
>>>>>>>>> is in general. There are some band-aid solutions like applying 
>>>>>>>>> some
>>>>>>>>> kind of weight against similarities based on small intersection
>>>>>>>>> size.
>>>>>>>>> Or you can pretend that missing values are 0 
>>>>>>>>> (PreferenceInferrer),
>>>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>>>
>>>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>> Thanks for replying.
>>>>>>>>>>
>>>>>>>>>> So, documents with only one word in common have more chance 
>>>>>>>>>> to be
>>>>>>>>>> similar
>>>>>>>>>> than documents with more words in common, right ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>>>> Similar items, right? You should look at the vectors that 
>>>>>>>>>>> have 1.0
>>>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>>>> by far
>>>>>>>>>>> the most likely explanation. Remember that the vector
>>>>>>>>>>> similarity is
>>>>>>>>>>> computed over elements that exist in both vectors only. They 
>>>>>>>>>>> just
>>>>>>>>>>> have
>>>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>> For each item, i have 10 recommended items with a value of 
>>>>>>>>>>>> 1.0.
>>>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum 
>>>>>>>>>>>>> similarity and
>>>>>>>>>>>>> occurs when two vector are just a scalar multiple of each
>>>>>>>>>>>>> other (0
>>>>>>>>>>>>> angle between them). It's possible there are several of 
>>>>>>>>>>>>> these,
>>>>>>>>>>>>> and so
>>>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>>>> Is it normal ?
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Ok, i'll try this evening.

On 10/02/2012 02:39 PM, Sebastian Schelter wrote:
> Would you like to create a patch for this?
>
> On 02.10.2012 14:36, yamo93 wrote:
>> +1 for the implementation over all entries.
>>
>> On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>>> I don't see why documents with only one word in common should have a
>>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
>>> specify a threshold for the similarity.
>>>
>>> UncenteredCosineSimilarity works on matching entries only, which is
>>> problematic for documents, as empty entries have a meaning (0 term
>>> occurrences) as opposed to collaborative filtering data.
>>>
>>> Maybe we should remove UncenteredCosine andd create another similarity
>>> implementation that computes the cosine correctly over all entries.
>>>
>>> --sebastian
>>>
>>>
>>> On 02.10.2012 10:08, yamo93 wrote:
>>>> Hello Seb,
>>>>
>>>> In my comprehension, the algorithm is the same (except the normalization
>>>> part) as UncenteredCosine (with the drawback that vectors with only one
>>>> word in common have a distance of 1.0)... but the result are quite
>>>> different (is this just an effect of the consider() method which remove
>>>> irrelevant values ?) ...
>>>>
>>>> I looked at the code but there is quite nothing in
>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>>>>
>>>> the code seems to be in SimilarityReducer which is not so simple to
>>>> understand ...
>>>>
>>>> Thanks for helping,
>>>>
>>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>>> similarity between the whole vectors.
>>>>>
>>>>> see
>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>>>
>>>>>
>>>>> for details
>>>>>
>>>>> At first both vectors are scaled to unit length in normalize() and
>>>>> after
>>>>> this their dot product in similarity() (which can be computed from
>>>>> elements that exist in both vectors) gives the cosine between those.
>>>>>
>>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>>> result.
>>>>>> For two items,
>>>>>> itemA, 0, 0,   a1, a2, a3, 0
>>>>>> itemB, 0, b1, b2, b3, 0  , 0
>>>>>> when computing, it just uses the blue parts of the vectors.
>>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>>> * sqrt(b2*b2 + b3*b3))
>>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>>> 2) if the values of the vectors are almost the same, the value would
>>>>>> also be nearly 1;
>>>>>> and for the two cases above, I think you can consider to use
>>>>>> association rules to consider the problem.
>>>>>>
>>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>>>>> also use CosineSimilarity. Why ?
>>>>>>>
>>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>>>>> is in general. There are some band-aid solutions like applying some
>>>>>>>> kind of weight against similarities based on small intersection
>>>>>>>> size.
>>>>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>>
>>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> Thanks for replying.
>>>>>>>>>
>>>>>>>>> So, documents with only one word in common have more chance to be
>>>>>>>>> similar
>>>>>>>>> than documents with more words in common, right ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>>> by far
>>>>>>>>>> the most likely explanation. Remember that the vector
>>>>>>>>>> similarity is
>>>>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>>>>> have
>>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>>>>> occurs when two vector are just a scalar multiple of each
>>>>>>>>>>>> other (0
>>>>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>>>>> and so
>>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>>> Is it normal ?

Re:Re: Need to reduce execution time of RowSimilarityJob

[bangbig <li...@163.com>]

Yes, you are right!






At 2012-10-02 04:25:09,"Sebastian Schelter" <ss...@apache.org> wrote:
>The cosine similarity as computed by RowSimilarityJob is the cosine
>similarity between the whole vectors.
>
>see
>org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>for details
>
>At first both vectors are scaled to unit length in normalize() and after
>this their dot product in similarity() (which can be computed from
>elements that exist in both vectors) gives the cosine between those.
>
>On 01.10.2012 21:52, bangbig wrote:
>> I think it's better to understand how the RowSimilarityJob gets the result.
>> For two items, 
>> itemA, 0, 0,   a1, a2, a3, 0
>> itemB, 0, b1, b2, b3, 0  , 0
>> when computing, it just uses the blue parts of the vectors.
>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)  * sqrt(b2*b2 + b3*b3))
>> 1) if itemA and itemB have just one common word, the result is 1;
>> 2) if the values of the vectors are almost the same, the value would also be nearly 1;
>> and for the two cases above, I think you can consider to use association rules to consider the problem.
>> 
>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>> It seems that RowSimilarityJob does not have the same weakness, but i 
>>> also use CosineSimilarity. Why ?
>>>
>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>> particular similarity metric. The more sparse, the worse the problem
>>>> is in general. There are some band-aid solutions like applying some
>>>> kind of weight against similarities based on small intersection size.
>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>> which can introduce its own problems, or perhaps some mean value.
>>>>
>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>> Thanks for replying.
>>>>>
>>>>> So, documents with only one word in common have more chance to be similar
>>>>> than documents with more words in common, right ?
>>>>>
>>>>>
>>>>>
>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>> similarity and see if they are in fact collinear. This is still by far
>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>> computed over elements that exist in both vectors only. They just have
>>>>>> to have 2 identical values for this to happen.
>>>>>>
>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>> It sounds like a bug somewhere.
>>>>>>>
>>>>>>>
>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>> angle between them). It's possible there are several of these, and so
>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>
>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>> Is it normal ?
>>>>>
>>>
>> 
>

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

Would you like to create a patch for this?

On 02.10.2012 14:36, yamo93 wrote:
> +1 for the implementation over all entries.
> 
> On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>> I don't see why documents with only one word in common should have a
>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
>> specify a threshold for the similarity.
>>
>> UncenteredCosineSimilarity works on matching entries only, which is
>> problematic for documents, as empty entries have a meaning (0 term
>> occurrences) as opposed to collaborative filtering data.
>>
>> Maybe we should remove UncenteredCosine andd create another similarity
>> implementation that computes the cosine correctly over all entries.
>>
>> --sebastian
>>
>>
>> On 02.10.2012 10:08, yamo93 wrote:
>>> Hello Seb,
>>>
>>> In my comprehension, the algorithm is the same (except the normalization
>>> part) as UncenteredCosine (with the drawback that vectors with only one
>>> word in common have a distance of 1.0)... but the result are quite
>>> different (is this just an effect of the consider() method which remove
>>> irrelevant values ?) ...
>>>
>>> I looked at the code but there is quite nothing in
>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>>>
>>> the code seems to be in SimilarityReducer which is not so simple to
>>> understand ...
>>>
>>> Thanks for helping,
>>>
>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>> similarity between the whole vectors.
>>>>
>>>> see
>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>>
>>>>
>>>> for details
>>>>
>>>> At first both vectors are scaled to unit length in normalize() and
>>>> after
>>>> this their dot product in similarity() (which can be computed from
>>>> elements that exist in both vectors) gives the cosine between those.
>>>>
>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>> result.
>>>>> For two items,
>>>>> itemA, 0, 0,   a1, a2, a3, 0
>>>>> itemB, 0, b1, b2, b3, 0  , 0
>>>>> when computing, it just uses the blue parts of the vectors.
>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>> * sqrt(b2*b2 + b3*b3))
>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>> 2) if the values of the vectors are almost the same, the value would
>>>>> also be nearly 1;
>>>>> and for the two cases above, I think you can consider to use
>>>>> association rules to consider the problem.
>>>>>
>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>>>> also use CosineSimilarity. Why ?
>>>>>>
>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>>>> is in general. There are some band-aid solutions like applying some
>>>>>>> kind of weight against similarities based on small intersection
>>>>>>> size.
>>>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>
>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>> Thanks for replying.
>>>>>>>>
>>>>>>>> So, documents with only one word in common have more chance to be
>>>>>>>> similar
>>>>>>>> than documents with more words in common, right ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>> by far
>>>>>>>>> the most likely explanation. Remember that the vector
>>>>>>>>> similarity is
>>>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>>>> have
>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>
>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>>>> occurs when two vector are just a scalar multiple of each
>>>>>>>>>>> other (0
>>>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>>>> and so
>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>> Is it normal ?
> 

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

+1 for the implementation over all entries.

On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
> I don't see why documents with only one word in common should have a
> similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
> specify a threshold for the similarity.
>
> UncenteredCosineSimilarity works on matching entries only, which is
> problematic for documents, as empty entries have a meaning (0 term
> occurrences) as opposed to collaborative filtering data.
>
> Maybe we should remove UncenteredCosine andd create another similarity
> implementation that computes the cosine correctly over all entries.
>
> --sebastian
>
>
> On 02.10.2012 10:08, yamo93 wrote:
>> Hello Seb,
>>
>> In my comprehension, the algorithm is the same (except the normalization
>> part) as UncenteredCosine (with the drawback that vectors with only one
>> word in common have a distance of 1.0)... but the result are quite
>> different (is this just an effect of the consider() method which remove
>> irrelevant values ?) ...
>>
>> I looked at the code but there is quite nothing in
>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>> the code seems to be in SimilarityReducer which is not so simple to
>> understand ...
>>
>> Thanks for helping,
>>
>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>> similarity between the whole vectors.
>>>
>>> see
>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>
>>> for details
>>>
>>> At first both vectors are scaled to unit length in normalize() and after
>>> this their dot product in similarity() (which can be computed from
>>> elements that exist in both vectors) gives the cosine between those.
>>>
>>> On 01.10.2012 21:52, bangbig wrote:
>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>> result.
>>>> For two items,
>>>> itemA, 0, 0,   a1, a2, a3, 0
>>>> itemB, 0, b1, b2, b3, 0  , 0
>>>> when computing, it just uses the blue parts of the vectors.
>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>> * sqrt(b2*b2 + b3*b3))
>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>> 2) if the values of the vectors are almost the same, the value would
>>>> also be nearly 1;
>>>> and for the two cases above, I think you can consider to use
>>>> association rules to consider the problem.
>>>>
>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>>> also use CosineSimilarity. Why ?
>>>>>
>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>>> is in general. There are some band-aid solutions like applying some
>>>>>> kind of weight against similarities based on small intersection size.
>>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>
>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>> Thanks for replying.
>>>>>>>
>>>>>>> So, documents with only one word in common have more chance to be
>>>>>>> similar
>>>>>>> than documents with more words in common, right ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>> by far
>>>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>>> have
>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>
>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>>> and so
>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>> Is it normal ?

Re:Re:Re: Need to reduce execution time of RowSimilarityJob

[bangbig <li...@163.com>]

By the way,  note that distance is not the same thing as similarity. 

At 2012-10-02 19:43:58,bangbig <li...@163.com> wrote:
>Yes, you get it.
>I thought RowSimilarityJob was from taste when I write the previous email.
>
>At 2012-10-02 19:26:48,yamo93 <ya...@gmail.com> wrote:
>>Ok, i think i understood.
>>
>>Let's take an example with two vectors (1,1,1) and (0,1,0).
>>With UncenteredCosineSimilarity (as implemented in taste), the distance is 1
>>With Cosine (as implemented in RowSimilarityJob), the distance is 1/sqrt(3)
>>
>>Ok ?
>>
>>On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>>> I don't see why documents with only one word in common should have a
>>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
>>> specify a threshold for the similarity.
>>>
>>> UncenteredCosineSimilarity works on matching entries only, which is
>>> problematic for documents, as empty entries have a meaning (0 term
>>> occurrences) as opposed to collaborative filtering data.
>>>
>>> Maybe we should remove UncenteredCosine andd create another similarity
>>> implementation that computes the cosine correctly over all entries.
>>>
>>> --sebastian
>>>
>>>
>>> On 02.10.2012 10:08, yamo93 wrote:
>>>> Hello Seb,
>>>>
>>>> In my comprehension, the algorithm is the same (except the normalization
>>>> part) as UncenteredCosine (with the drawback that vectors with only one
>>>> word in common have a distance of 1.0)... but the result are quite
>>>> different (is this just an effect of the consider() method which remove
>>>> irrelevant values ?) ...
>>>>
>>>> I looked at the code but there is quite nothing in
>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>>>> the code seems to be in SimilarityReducer which is not so simple to
>>>> understand ...
>>>>
>>>> Thanks for helping,
>>>>
>>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>>> similarity between the whole vectors.
>>>>>
>>>>> see
>>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>>>
>>>>> for details
>>>>>
>>>>> At first both vectors are scaled to unit length in normalize() and after
>>>>> this their dot product in similarity() (which can be computed from
>>>>> elements that exist in both vectors) gives the cosine between those.
>>>>>
>>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>>> result.
>>>>>> For two items,
>>>>>> itemA, 0, 0,   a1, a2, a3, 0
>>>>>> itemB, 0, b1, b2, b3, 0  , 0
>>>>>> when computing, it just uses the blue parts of the vectors.
>>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>>> * sqrt(b2*b2 + b3*b3))
>>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>>> 2) if the values of the vectors are almost the same, the value would
>>>>>> also be nearly 1;
>>>>>> and for the two cases above, I think you can consider to use
>>>>>> association rules to consider the problem.
>>>>>>
>>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>>>>> also use CosineSimilarity. Why ?
>>>>>>>
>>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>>>>> is in general. There are some band-aid solutions like applying some
>>>>>>>> kind of weight against similarities based on small intersection size.
>>>>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>>
>>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> Thanks for replying.
>>>>>>>>>
>>>>>>>>> So, documents with only one word in common have more chance to be
>>>>>>>>> similar
>>>>>>>>> than documents with more words in common, right ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>>> by far
>>>>>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>>>>> have
>>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>>>>> and so
>>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>>> Is it normal ?
>>

Re:Re: Need to reduce execution time of RowSimilarityJob

[bangbig <li...@163.com>]

Yes, you get it.
I thought RowSimilarityJob was from taste when I write the previous email.

At 2012-10-02 19:26:48,yamo93 <ya...@gmail.com> wrote:
>Ok, i think i understood.
>
>Let's take an example with two vectors (1,1,1) and (0,1,0).
>With UncenteredCosineSimilarity (as implemented in taste), the distance is 1
>With Cosine (as implemented in RowSimilarityJob), the distance is 1/sqrt(3)
>
>Ok ?
>
>On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
>> I don't see why documents with only one word in common should have a
>> similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
>> specify a threshold for the similarity.
>>
>> UncenteredCosineSimilarity works on matching entries only, which is
>> problematic for documents, as empty entries have a meaning (0 term
>> occurrences) as opposed to collaborative filtering data.
>>
>> Maybe we should remove UncenteredCosine andd create another similarity
>> implementation that computes the cosine correctly over all entries.
>>
>> --sebastian
>>
>>
>> On 02.10.2012 10:08, yamo93 wrote:
>>> Hello Seb,
>>>
>>> In my comprehension, the algorithm is the same (except the normalization
>>> part) as UncenteredCosine (with the drawback that vectors with only one
>>> word in common have a distance of 1.0)... but the result are quite
>>> different (is this just an effect of the consider() method which remove
>>> irrelevant values ?) ...
>>>
>>> I looked at the code but there is quite nothing in
>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>>> the code seems to be in SimilarityReducer which is not so simple to
>>> understand ...
>>>
>>> Thanks for helping,
>>>
>>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>>> similarity between the whole vectors.
>>>>
>>>> see
>>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>>
>>>> for details
>>>>
>>>> At first both vectors are scaled to unit length in normalize() and after
>>>> this their dot product in similarity() (which can be computed from
>>>> elements that exist in both vectors) gives the cosine between those.
>>>>
>>>> On 01.10.2012 21:52, bangbig wrote:
>>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>>> result.
>>>>> For two items,
>>>>> itemA, 0, 0,   a1, a2, a3, 0
>>>>> itemB, 0, b1, b2, b3, 0  , 0
>>>>> when computing, it just uses the blue parts of the vectors.
>>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>>> * sqrt(b2*b2 + b3*b3))
>>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>>> 2) if the values of the vectors are almost the same, the value would
>>>>> also be nearly 1;
>>>>> and for the two cases above, I think you can consider to use
>>>>> association rules to consider the problem.
>>>>>
>>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>>>> also use CosineSimilarity. Why ?
>>>>>>
>>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>>>> is in general. There are some band-aid solutions like applying some
>>>>>>> kind of weight against similarities based on small intersection size.
>>>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>>
>>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>> Thanks for replying.
>>>>>>>>
>>>>>>>> So, documents with only one word in common have more chance to be
>>>>>>>> similar
>>>>>>>> than documents with more words in common, right ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>>> by far
>>>>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>>>> have
>>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>>
>>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>>>> and so
>>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>>> Is it normal ?
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Ok, i think i understood.

Let's take an example with two vectors (1,1,1) and (0,1,0).
With UncenteredCosineSimilarity (as implemented in taste), the distance is 1
With Cosine (as implemented in RowSimilarityJob), the distance is 1/sqrt(3)

Ok ?

On 10/02/2012 11:50 AM, Sebastian Schelter wrote:
> I don't see why documents with only one word in common should have a
> similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
> specify a threshold for the similarity.
>
> UncenteredCosineSimilarity works on matching entries only, which is
> problematic for documents, as empty entries have a meaning (0 term
> occurrences) as opposed to collaborative filtering data.
>
> Maybe we should remove UncenteredCosine andd create another similarity
> implementation that computes the cosine correctly over all entries.
>
> --sebastian
>
>
> On 02.10.2012 10:08, yamo93 wrote:
>> Hello Seb,
>>
>> In my comprehension, the algorithm is the same (except the normalization
>> part) as UncenteredCosine (with the drawback that vectors with only one
>> word in common have a distance of 1.0)... but the result are quite
>> different (is this just an effect of the consider() method which remove
>> irrelevant values ?) ...
>>
>> I looked at the code but there is quite nothing in
>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
>> the code seems to be in SimilarityReducer which is not so simple to
>> understand ...
>>
>> Thanks for helping,
>>
>> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>>> The cosine similarity as computed by RowSimilarityJob is the cosine
>>> similarity between the whole vectors.
>>>
>>> see
>>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>>
>>> for details
>>>
>>> At first both vectors are scaled to unit length in normalize() and after
>>> this their dot product in similarity() (which can be computed from
>>> elements that exist in both vectors) gives the cosine between those.
>>>
>>> On 01.10.2012 21:52, bangbig wrote:
>>>> I think it's better to understand how the RowSimilarityJob gets the
>>>> result.
>>>> For two items,
>>>> itemA, 0, 0,   a1, a2, a3, 0
>>>> itemB, 0, b1, b2, b3, 0  , 0
>>>> when computing, it just uses the blue parts of the vectors.
>>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)
>>>> * sqrt(b2*b2 + b3*b3))
>>>> 1) if itemA and itemB have just one common word, the result is 1;
>>>> 2) if the values of the vectors are almost the same, the value would
>>>> also be nearly 1;
>>>> and for the two cases above, I think you can consider to use
>>>> association rules to consider the problem.
>>>>
>>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>>> also use CosineSimilarity. Why ?
>>>>>
>>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>>> is in general. There are some band-aid solutions like applying some
>>>>>> kind of weight against similarities based on small intersection size.
>>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>>
>>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>> Thanks for replying.
>>>>>>>
>>>>>>> So, documents with only one word in common have more chance to be
>>>>>>> similar
>>>>>>> than documents with more words in common, right ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>>> by far
>>>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>>> have
>>>>>>>> to have 2 identical values for this to happen.
>>>>>>>>
>>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>>> and so
>>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>>> Is it normal ?

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

I don't see why documents with only one word in common should have a
similarity of 1.0 in RowSimilarityJob. consider() is only invoked if you
specify a threshold for the similarity.

UncenteredCosineSimilarity works on matching entries only, which is
problematic for documents, as empty entries have a meaning (0 term
occurrences) as opposed to collaborative filtering data.

Maybe we should remove UncenteredCosine andd create another similarity
implementation that computes the cosine correctly over all entries.

--sebastian


On 02.10.2012 10:08, yamo93 wrote:
> Hello Seb,
> 
> In my comprehension, the algorithm is the same (except the normalization
> part) as UncenteredCosine (with the drawback that vectors with only one
> word in common have a distance of 1.0)... but the result are quite
> different (is this just an effect of the consider() method which remove
> irrelevant values ?) ...
> 
> I looked at the code but there is quite nothing in
> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity,
> the code seems to be in SimilarityReducer which is not so simple to
> understand ...
> 
> Thanks for helping,
> 
> On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
>> The cosine similarity as computed by RowSimilarityJob is the cosine
>> similarity between the whole vectors.
>>
>> see
>> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
>>
>> for details
>>
>> At first both vectors are scaled to unit length in normalize() and after
>> this their dot product in similarity() (which can be computed from
>> elements that exist in both vectors) gives the cosine between those.
>>
>> On 01.10.2012 21:52, bangbig wrote:
>>> I think it's better to understand how the RowSimilarityJob gets the
>>> result.
>>> For two items,
>>> itemA, 0, 0,   a1, a2, a3, 0
>>> itemB, 0, b1, b2, b3, 0  , 0
>>> when computing, it just uses the blue parts of the vectors.
>>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2) 
>>> * sqrt(b2*b2 + b3*b3))
>>> 1) if itemA and itemB have just one common word, the result is 1;
>>> 2) if the values of the vectors are almost the same, the value would
>>> also be nearly 1;
>>> and for the two cases above, I think you can consider to use
>>> association rules to consider the problem.
>>>
>>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>>> also use CosineSimilarity. Why ?
>>>>
>>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>>> particular similarity metric. The more sparse, the worse the problem
>>>>> is in general. There are some band-aid solutions like applying some
>>>>> kind of weight against similarities based on small intersection size.
>>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>>> which can introduce its own problems, or perhaps some mean value.
>>>>>
>>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>> Thanks for replying.
>>>>>>
>>>>>> So, documents with only one word in common have more chance to be
>>>>>> similar
>>>>>> than documents with more words in common, right ?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>>> similarity and see if they are in fact collinear. This is still
>>>>>>> by far
>>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>>> computed over elements that exist in both vectors only. They just
>>>>>>> have
>>>>>>> to have 2 identical values for this to happen.
>>>>>>>
>>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>>> It sounds like a bug somewhere.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>>> angle between them). It's possible there are several of these,
>>>>>>>>> and so
>>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>>
>>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>>> Is it normal ?
> 

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Hello Seb,

In my comprehension, the algorithm is the same (except the normalization 
part) as UncenteredCosine (with the drawback that vectors with only one 
word in common have a distance of 1.0)... but the result are quite 
different (is this just an effect of the consider() method which remove 
irrelevant values ?) ...

I looked at the code but there is quite nothing in 
org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity, 
the code seems to be in SimilarityReducer which is not so simple to 
understand ...

Thanks for helping,

On 10/01/2012 10:25 PM, Sebastian Schelter wrote:
> The cosine similarity as computed by RowSimilarityJob is the cosine
> similarity between the whole vectors.
>
> see
> org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
> for details
>
> At first both vectors are scaled to unit length in normalize() and after
> this their dot product in similarity() (which can be computed from
> elements that exist in both vectors) gives the cosine between those.
>
> On 01.10.2012 21:52, bangbig wrote:
>> I think it's better to understand how the RowSimilarityJob gets the result.
>> For two items,
>> itemA, 0, 0,   a1, a2, a3, 0
>> itemB, 0, b1, b2, b3, 0  , 0
>> when computing, it just uses the blue parts of the vectors.
>> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)  * sqrt(b2*b2 + b3*b3))
>> 1) if itemA and itemB have just one common word, the result is 1;
>> 2) if the values of the vectors are almost the same, the value would also be nearly 1;
>> and for the two cases above, I think you can consider to use association rules to consider the problem.
>>
>> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>>> It seems that RowSimilarityJob does not have the same weakness, but i
>>> also use CosineSimilarity. Why ?
>>>
>>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>>> Yes, this is one of the weaknesses of this particular flavor of this
>>>> particular similarity metric. The more sparse, the worse the problem
>>>> is in general. There are some band-aid solutions like applying some
>>>> kind of weight against similarities based on small intersection size.
>>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>>> which can introduce its own problems, or perhaps some mean value.
>>>>
>>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>>> Thanks for replying.
>>>>>
>>>>> So, documents with only one word in common have more chance to be similar
>>>>> than documents with more words in common, right ?
>>>>>
>>>>>
>>>>>
>>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>>> similarity and see if they are in fact collinear. This is still by far
>>>>>> the most likely explanation. Remember that the vector similarity is
>>>>>> computed over elements that exist in both vectors only. They just have
>>>>>> to have 2 identical values for this to happen.
>>>>>>
>>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>>> It sounds like a bug somewhere.
>>>>>>>
>>>>>>>
>>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>>> angle between them). It's possible there are several of these, and so
>>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>>
>>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>>> Is it normal ?

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

The cosine similarity as computed by RowSimilarityJob is the cosine
similarity between the whole vectors.

see
org.apache.mahout.math.hadoop.similarity.cooccurrence.measures.CosineSimilarity
for details

At first both vectors are scaled to unit length in normalize() and after
this their dot product in similarity() (which can be computed from
elements that exist in both vectors) gives the cosine between those.

On 01.10.2012 21:52, bangbig wrote:
> I think it's better to understand how the RowSimilarityJob gets the result.
> For two items, 
> itemA, 0, 0,   a1, a2, a3, 0
> itemB, 0, b1, b2, b3, 0  , 0
> when computing, it just uses the blue parts of the vectors.
> the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)  * sqrt(b2*b2 + b3*b3))
> 1) if itemA and itemB have just one common word, the result is 1;
> 2) if the values of the vectors are almost the same, the value would also be nearly 1;
> and for the two cases above, I think you can consider to use association rules to consider the problem.
> 
> At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>> It seems that RowSimilarityJob does not have the same weakness, but i 
>> also use CosineSimilarity. Why ?
>>
>> On 10/01/2012 12:37 PM, Sean Owen wrote:
>>> Yes, this is one of the weaknesses of this particular flavor of this
>>> particular similarity metric. The more sparse, the worse the problem
>>> is in general. There are some band-aid solutions like applying some
>>> kind of weight against similarities based on small intersection size.
>>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>>> which can introduce its own problems, or perhaps some mean value.
>>>
>>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>>> Thanks for replying.
>>>>
>>>> So, documents with only one word in common have more chance to be similar
>>>> than documents with more words in common, right ?
>>>>
>>>>
>>>>
>>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>>> Similar items, right? You should look at the vectors that have 1.0
>>>>> similarity and see if they are in fact collinear. This is still by far
>>>>> the most likely explanation. Remember that the vector similarity is
>>>>> computed over elements that exist in both vectors only. They just have
>>>>> to have 2 identical values for this to happen.
>>>>>
>>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>>> It sounds like a bug somewhere.
>>>>>>
>>>>>>
>>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>>> angle between them). It's possible there are several of these, and so
>>>>>>> their 1.0 similarities dominate the result.
>>>>>>>
>>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>>> Is it normal ?
>>>>
>>
> 

Re:Re: Need to reduce execution time of RowSimilarityJob

[bangbig <li...@163.com>]

I think it's better to understand how the RowSimilarityJob gets the result.
For two items, 
itemA, 0, 0,   a1, a2, a3, 0
itemB, 0, b1, b2, b3, 0  , 0
when computing, it just uses the blue parts of the vectors.
the cosine similarity thus is (a1*b2 + a2*b3)/(sqrt(a1*a1 + a2* a2)  * sqrt(b2*b2 + b3*b3))
1) if itemA and itemB have just one common word, the result is 1;
2) if the values of the vectors are almost the same, the value would also be nearly 1;
and for the two cases above, I think you can consider to use association rules to consider the problem.

At 2012-10-01 20:53:16,yamo93 <ya...@gmail.com> wrote:
>It seems that RowSimilarityJob does not have the same weakness, but i 
>also use CosineSimilarity. Why ?
>
>On 10/01/2012 12:37 PM, Sean Owen wrote:
>> Yes, this is one of the weaknesses of this particular flavor of this
>> particular similarity metric. The more sparse, the worse the problem
>> is in general. There are some band-aid solutions like applying some
>> kind of weight against similarities based on small intersection size.
>> Or you can pretend that missing values are 0 (PreferenceInferrer),
>> which can introduce its own problems, or perhaps some mean value.
>>
>> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>>> Thanks for replying.
>>>
>>> So, documents with only one word in common have more chance to be similar
>>> than documents with more words in common, right ?
>>>
>>>
>>>
>>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>>> Similar items, right? You should look at the vectors that have 1.0
>>>> similarity and see if they are in fact collinear. This is still by far
>>>> the most likely explanation. Remember that the vector similarity is
>>>> computed over elements that exist in both vectors only. They just have
>>>> to have 2 identical values for this to happen.
>>>>
>>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>>> It sounds like a bug somewhere.
>>>>>
>>>>>
>>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>>> angle between them). It's possible there are several of these, and so
>>>>>> their 1.0 similarities dominate the result.
>>>>>>
>>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>>> I saw something strange : all recommended items, returned by
>>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>>> Is it normal ?
>>>
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

It seems that RowSimilarityJob does not have the same weakness, but i 
also use CosineSimilarity. Why ?

On 10/01/2012 12:37 PM, Sean Owen wrote:
> Yes, this is one of the weaknesses of this particular flavor of this
> particular similarity metric. The more sparse, the worse the problem
> is in general. There are some band-aid solutions like applying some
> kind of weight against similarities based on small intersection size.
> Or you can pretend that missing values are 0 (PreferenceInferrer),
> which can introduce its own problems, or perhaps some mean value.
>
> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>> Thanks for replying.
>>
>> So, documents with only one word in common have more chance to be similar
>> than documents with more words in common, right ?
>>
>>
>>
>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>> Similar items, right? You should look at the vectors that have 1.0
>>> similarity and see if they are in fact collinear. This is still by far
>>> the most likely explanation. Remember that the vector similarity is
>>> computed over elements that exist in both vectors only. They just have
>>> to have 2 identical values for this to happen.
>>>
>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>> It sounds like a bug somewhere.
>>>>
>>>>
>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>> angle between them). It's possible there are several of these, and so
>>>>> their 1.0 similarities dominate the result.
>>>>>
>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>> I saw something strange : all recommended items, returned by
>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>> Is it normal ?
>>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Ok. You're right : i have some spareness on my data.

Do you advise another similarity algorithm for text-based ?

On 10/01/2012 12:37 PM, Sean Owen wrote:
> Yes, this is one of the weaknesses of this particular flavor of this
> particular similarity metric. The more sparse, the worse the problem
> is in general. There are some band-aid solutions like applying some
> kind of weight against similarities based on small intersection size.
> Or you can pretend that missing values are 0 (PreferenceInferrer),
> which can introduce its own problems, or perhaps some mean value.
>
> On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
>> Thanks for replying.
>>
>> So, documents with only one word in common have more chance to be similar
>> than documents with more words in common, right ?
>>
>>
>>
>> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>> Similar items, right? You should look at the vectors that have 1.0
>>> similarity and see if they are in fact collinear. This is still by far
>>> the most likely explanation. Remember that the vector similarity is
>>> computed over elements that exist in both vectors only. They just have
>>> to have 2 identical values for this to happen.
>>>
>>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>> For each item, i have 10 recommended items with a value of 1.0.
>>>> It sounds like a bug somewhere.
>>>>
>>>>
>>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>>> occurs when two vector are just a scalar multiple of each other (0
>>>>> angle between them). It's possible there are several of these, and so
>>>>> their 1.0 similarities dominate the result.
>>>>>
>>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>> I saw something strange : all recommended items, returned by
>>>>>> mostSimilarItems(), have a value of 1.0.
>>>>>> Is it normal ?
>>

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

Yes, this is one of the weaknesses of this particular flavor of this
particular similarity metric. The more sparse, the worse the problem
is in general. There are some band-aid solutions like applying some
kind of weight against similarities based on small intersection size.
Or you can pretend that missing values are 0 (PreferenceInferrer),
which can introduce its own problems, or perhaps some mean value.

On Mon, Oct 1, 2012 at 11:32 AM, yamo93 <ya...@gmail.com> wrote:
> Thanks for replying.
>
> So, documents with only one word in common have more chance to be similar
> than documents with more words in common, right ?
>
>
>
> On 10/01/2012 11:28 AM, Sean Owen wrote:
>>
>> Similar items, right? You should look at the vectors that have 1.0
>> similarity and see if they are in fact collinear. This is still by far
>> the most likely explanation. Remember that the vector similarity is
>> computed over elements that exist in both vectors only. They just have
>> to have 2 identical values for this to happen.
>>
>> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>>>
>>> For each item, i have 10 recommended items with a value of 1.0.
>>> It sounds like a bug somewhere.
>>>
>>>
>>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>>>
>>>> It's possible this is correct. 1.0 is the maximum similarity and
>>>> occurs when two vector are just a scalar multiple of each other (0
>>>> angle between them). It's possible there are several of these, and so
>>>> their 1.0 similarities dominate the result.
>>>>
>>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>>>
>>>>> I saw something strange : all recommended items, returned by
>>>>> mostSimilarItems(), have a value of 1.0.
>>>>> Is it normal ?
>
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Thanks for replying.

So, documents with only one word in common have more chance to be 
similar than documents with more words in common, right ?


On 10/01/2012 11:28 AM, Sean Owen wrote:
> Similar items, right? You should look at the vectors that have 1.0
> similarity and see if they are in fact collinear. This is still by far
> the most likely explanation. Remember that the vector similarity is
> computed over elements that exist in both vectors only. They just have
> to have 2 identical values for this to happen.
>
> On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
>> For each item, i have 10 recommended items with a value of 1.0.
>> It sounds like a bug somewhere.
>>
>>
>> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>> It's possible this is correct. 1.0 is the maximum similarity and
>>> occurs when two vector are just a scalar multiple of each other (0
>>> angle between them). It's possible there are several of these, and so
>>> their 1.0 similarities dominate the result.
>>>
>>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>> I saw something strange : all recommended items, returned by
>>>> mostSimilarItems(), have a value of 1.0.
>>>> Is it normal ?

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

Similar items, right? You should look at the vectors that have 1.0
similarity and see if they are in fact collinear. This is still by far
the most likely explanation. Remember that the vector similarity is
computed over elements that exist in both vectors only. They just have
to have 2 identical values for this to happen.

On Mon, Oct 1, 2012 at 10:25 AM, yamo93 <ya...@gmail.com> wrote:
> For each item, i have 10 recommended items with a value of 1.0.
> It sounds like a bug somewhere.
>
>
> On 10/01/2012 11:06 AM, Sean Owen wrote:
>>
>> It's possible this is correct. 1.0 is the maximum similarity and
>> occurs when two vector are just a scalar multiple of each other (0
>> angle between them). It's possible there are several of these, and so
>> their 1.0 similarities dominate the result.
>>
>> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>>>
>>> I saw something strange : all recommended items, returned by
>>> mostSimilarItems(), have a value of 1.0.
>>> Is it normal ?

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

For each item, i have 10 recommended items with a value of 1.0.
It sounds like a bug somewhere.

On 10/01/2012 11:06 AM, Sean Owen wrote:
> It's possible this is correct. 1.0 is the maximum similarity and
> occurs when two vector are just a scalar multiple of each other (0
> angle between them). It's possible there are several of these, and so
> their 1.0 similarities dominate the result.
>
> On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
>> I saw something strange : all recommended items, returned by
>> mostSimilarItems(), have a value of 1.0.
>> Is it normal ?
>>
>>
>> On 10/01/2012 10:39 AM, Sean Owen wrote:
>>> This is probably because the Hadoop job does some sampling and pruning
>>> whereas the non-Hadoop generally doesn't.
>>

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

It's possible this is correct. 1.0 is the maximum similarity and
occurs when two vector are just a scalar multiple of each other (0
angle between them). It's possible there are several of these, and so
their 1.0 similarities dominate the result.

On Mon, Oct 1, 2012 at 10:03 AM, yamo93 <ya...@gmail.com> wrote:
> I saw something strange : all recommended items, returned by
> mostSimilarItems(), have a value of 1.0.
> Is it normal ?
>
>
> On 10/01/2012 10:39 AM, Sean Owen wrote:
>>
>> This is probably because the Hadoop job does some sampling and pruning
>> whereas the non-Hadoop generally doesn't.
>
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

I saw something strange : all recommended items, returned by 
mostSimilarItems(), have a value of 1.0.
Is it normal ?

On 10/01/2012 10:39 AM, Sean Owen wrote:
> This is probably because the Hadoop job does some sampling and pruning
> whereas the non-Hadoop generally doesn't.

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

This is probably because the Hadoop job does some sampling and pruning
whereas the non-Hadoop generally doesn't.

On Mon, Oct 1, 2012 at 9:38 AM, yamo93 <ya...@gmail.com> wrote:
> Because the results are not the same as with RowSimilarityJob and because
> similar documents have few words in common.
>
>
> On 10/01/2012 10:24 AM, Sean Owen wrote:
>>
>> There's not really any information here. Why do you think the result is
>> wrong?
>>
>> On Mon, Oct 1, 2012 at 9:09 AM, yamo93 <ya...@gmail.com> wrote:
>>>
>>> I tried your suggestion.
>>>
>>> I genereted a CSV file with (term, docId, distance) and I used the method
>>> mostSimilarItems with UncenteredCosineSimilarity.
>>>
>>> But this seems to produce wrong results, i don't understand why ?
>>>
>>> Any idea ?
>
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Because the results are not the same as with RowSimilarityJob and 
because similar documents have few words in common.

On 10/01/2012 10:24 AM, Sean Owen wrote:
> There's not really any information here. Why do you think the result is wrong?
>
> On Mon, Oct 1, 2012 at 9:09 AM, yamo93 <ya...@gmail.com> wrote:
>> I tried your suggestion.
>>
>> I genereted a CSV file with (term, docId, distance) and I used the method
>> mostSimilarItems with UncenteredCosineSimilarity.
>>
>> But this seems to produce wrong results, i don't understand why ?
>>
>> Any idea ?

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

There's not really any information here. Why do you think the result is wrong?

On Mon, Oct 1, 2012 at 9:09 AM, yamo93 <ya...@gmail.com> wrote:
> I tried your suggestion.
>
> I genereted a CSV file with (term, docId, distance) and I used the method
> mostSimilarItems with UncenteredCosineSimilarity.
>
> But this seems to produce wrong results, i don't understand why ?
>
> Any idea ?

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

I tried your suggestion.

I genereted a CSV file with (term, docId, distance) and I used the 
method mostSimilarItems with UncenteredCosineSimilarity.

But this seems to produce wrong results, i don't understand why ?

Any idea ?

On 09/20/2012 11:21 AM, Sean Owen wrote:
> Yes, what he means is that item-item similarity is just a subset of
> what a recommender does. You can use an in memory rec based on
> item-item similarity and just ask it for those similarities.
>
> On Thu, Sep 20, 2012 at 10:19 AM, yamo93 <ya...@gmail.com> wrote:
>> Exactly, but i need to improve perf and i thought that an in-memory impl
>> will be a solution (as mentionned in my first post).
>>
>> So Seb suggested to use recommender for that but i'm afraid he sent me on a
>> wrong way ...
>>

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

Yes, what he means is that item-item similarity is just a subset of
what a recommender does. You can use an in memory rec based on
item-item similarity and just ask it for those similarities.

On Thu, Sep 20, 2012 at 10:19 AM, yamo93 <ya...@gmail.com> wrote:
> Exactly, but i need to improve perf and i thought that an in-memory impl
> will be a solution (as mentionned in my first post).
>
> So Seb suggested to use recommender for that but i'm afraid he sent me on a
> wrong way ...
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Exactly, but i need to improve perf and i thought that an in-memory impl 
will be a solution (as mentionned in my first post).

So Seb suggested to use recommender for that but i'm afraid he sent me 
on a wrong way ...

On 09/20/2012 10:31 AM, Sean Owen wrote:
> Yes. But RowSimilarityJob is not computing recommendations, it's
> finding similar "users" (documents). It's already doing what you say
> then.
>
> On Thu, Sep 20, 2012 at 9:28 AM, yamo93 <ya...@gmail.com> wrote:
>> Hello,
>>
>> If document is second, the result will be recommendations of document to
>> terms, isn't it ?
>> My need is to find most similar documents (i.e. recommend document to
>> documents, no ?)
>>
>> Thx for your help,

Re: Need to reduce execution time of RowSimilarityJob

[Sean Owen <sr...@gmail.com>]

Yes. But RowSimilarityJob is not computing recommendations, it's
finding similar "users" (documents). It's already doing what you say
then.

On Thu, Sep 20, 2012 at 9:28 AM, yamo93 <ya...@gmail.com> wrote:
> Hello,
>
> If document is second, the result will be recommendations of document to
> terms, isn't it ?
> My need is to find most similar documents (i.e. recommend document to
> documents, no ?)
>
> Thx for your help,

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Hello,

If document is second, the result will be recommendations of document to 
terms, isn't it ?
My need is to find most similar documents (i.e. recommend document to 
documents, no ?)

Thx for your help,

On 09/18/2012 04:03 PM, Sebastian Schelter wrote:
> Another error of mine, sorry, documents need to be second :)
On 09/18/2012 03:58 PM, yamo93 wrote:
> If document is in first place, should i use user based recommender 
> instead of item based ?
>
> On 09/18/2012 03:21 PM, Sebastian Schelter wrote:
>> Oh I overlooked that, sorry. You could give it (document,term,tfidf)
>> pairs instead. If you find it awkward to use a recommender to compute
>> document similarities, than maybe it would be better to think a about a
>> custom in-memory implementation.
>>
>>
>> On 18.09.2012 15:13, yamo93 wrote:
>>> Thanks,
>>>
>>> I need some explanations :
>>> GenericItemBasedRecommender needs a FileDataModel with userId, itemId,
>>> score.
>>> But i have some text documents and today i use seq2sparse and after
>>> rowid + rowsimilarity.
>>> How to call GenericItemBasedRecommender with sparse vectors ?
>>>
>>> Y.
>>>
>>> On 09/18/2012 02:57 PM, Sebastian Schelter wrote:
>>>> You don't need to develop an in-memory implementation, we already have
>>>> that.
>>>>
>>>> Simply use a GenericItemBasedRecommender and ask it for the most 
>>>> similar
>>>> items of each item.
>>>>
>>>>
>>>> On 18.09.2012 14:49, yamo93 wrote:
>>>>> Hi,
>>>>>
>>>>> I have 30.000 items and the computation takes more than 2h on a
>>>>> pseudo-cluster, which is too long in my case.
>>>>>
>>>>> I think of some ways to reduce the execution time of RowSimilarityJob
>>>>> and I wonder if some of you have implemented them and how, or 
>>>>> explored
>>>>> other ways.
>>>>> 1. tune the JVM
>>>>> 2. developing an in memory implementation (i.e. without hadoop)
>>>>> 3. reduce the size of the matrix (by removing those which have no 
>>>>> words
>>>>> in common, for example)
>>>>> 4. run on real hadoop cluster with several nodes (does anyone have an
>>>>> idea of ​​the number of nodes to make it interesting)
>>>>>
>>>>> Thanks for your help,
>>>>> Yann.
>

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

If document is in first place, should i use user based recommender 
instead of item based ?

On 09/18/2012 03:21 PM, Sebastian Schelter wrote:
> Oh I overlooked that, sorry. You could give it (document,term,tfidf)
> pairs instead. If you find it awkward to use a recommender to compute
> document similarities, than maybe it would be better to think a about a
> custom in-memory implementation.
>
>
> On 18.09.2012 15:13, yamo93 wrote:
>> Thanks,
>>
>> I need some explanations :
>> GenericItemBasedRecommender needs a FileDataModel with userId, itemId,
>> score.
>> But i have some text documents and today i use seq2sparse and after
>> rowid + rowsimilarity.
>> How to call GenericItemBasedRecommender with sparse vectors ?
>>
>> Y.
>>
>> On 09/18/2012 02:57 PM, Sebastian Schelter wrote:
>>> You don't need to develop an in-memory implementation, we already have
>>> that.
>>>
>>> Simply use a GenericItemBasedRecommender and ask it for the most similar
>>> items of each item.
>>>
>>>
>>> On 18.09.2012 14:49, yamo93 wrote:
>>>> Hi,
>>>>
>>>> I have 30.000 items and the computation takes more than 2h on a
>>>> pseudo-cluster, which is too long in my case.
>>>>
>>>> I think of some ways to reduce the execution time of RowSimilarityJob
>>>> and I wonder if some of you have implemented them and how, or explored
>>>> other ways.
>>>> 1. tune the JVM
>>>> 2. developing an in memory implementation (i.e. without hadoop)
>>>> 3. reduce the size of the matrix (by removing those which have no words
>>>> in common, for example)
>>>> 4. run on real hadoop cluster with several nodes (does anyone have an
>>>> idea of ​​the number of nodes to make it interesting)
>>>>
>>>> Thanks for your help,
>>>> Yann.

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

Oh I overlooked that, sorry. You could give it (document,term,tfidf)
pairs instead. If you find it awkward to use a recommender to compute
document similarities, than maybe it would be better to think a about a
custom in-memory implementation.


On 18.09.2012 15:13, yamo93 wrote:
> Thanks,
> 
> I need some explanations :
> GenericItemBasedRecommender needs a FileDataModel with userId, itemId,
> score.
> But i have some text documents and today i use seq2sparse and after
> rowid + rowsimilarity.
> How to call GenericItemBasedRecommender with sparse vectors ?
> 
> Y.
> 
> On 09/18/2012 02:57 PM, Sebastian Schelter wrote:
>> You don't need to develop an in-memory implementation, we already have
>> that.
>>
>> Simply use a GenericItemBasedRecommender and ask it for the most similar
>> items of each item.
>>
>>
>> On 18.09.2012 14:49, yamo93 wrote:
>>> Hi,
>>>
>>> I have 30.000 items and the computation takes more than 2h on a
>>> pseudo-cluster, which is too long in my case.
>>>
>>> I think of some ways to reduce the execution time of RowSimilarityJob
>>> and I wonder if some of you have implemented them and how, or explored
>>> other ways.
>>> 1. tune the JVM
>>> 2. developing an in memory implementation (i.e. without hadoop)
>>> 3. reduce the size of the matrix (by removing those which have no words
>>> in common, for example)
>>> 4. run on real hadoop cluster with several nodes (does anyone have an
>>> idea of ​​the number of nodes to make it interesting)
>>>
>>> Thanks for your help,
>>> Yann.
> 

Re: Need to reduce execution time of RowSimilarityJob

[yamo93 <ya...@gmail.com>]

Thanks,

I need some explanations :
GenericItemBasedRecommender needs a FileDataModel with userId, itemId, 
score.
But i have some text documents and today i use seq2sparse and after 
rowid + rowsimilarity.
How to call GenericItemBasedRecommender with sparse vectors ?

Y.

On 09/18/2012 02:57 PM, Sebastian Schelter wrote:
> You don't need to develop an in-memory implementation, we already have that.
>
> Simply use a GenericItemBasedRecommender and ask it for the most similar
> items of each item.
>
>
> On 18.09.2012 14:49, yamo93 wrote:
>> Hi,
>>
>> I have 30.000 items and the computation takes more than 2h on a
>> pseudo-cluster, which is too long in my case.
>>
>> I think of some ways to reduce the execution time of RowSimilarityJob
>> and I wonder if some of you have implemented them and how, or explored
>> other ways.
>> 1. tune the JVM
>> 2. developing an in memory implementation (i.e. without hadoop)
>> 3. reduce the size of the matrix (by removing those which have no words
>> in common, for example)
>> 4. run on real hadoop cluster with several nodes (does anyone have an
>> idea of ​​the number of nodes to make it interesting)
>>
>> Thanks for your help,
>> Yann.

Re: Need to reduce execution time of RowSimilarityJob

[Sebastian Schelter <ss...@apache.org>]

You don't need to develop an in-memory implementation, we already have that.

Simply use a GenericItemBasedRecommender and ask it for the most similar
items of each item.


On 18.09.2012 14:49, yamo93 wrote:
> Hi,
> 
> I have 30.000 items and the computation takes more than 2h on a
> pseudo-cluster, which is too long in my case.
> 
> I think of some ways to reduce the execution time of RowSimilarityJob
> and I wonder if some of you have implemented them and how, or explored
> other ways.
> 1. tune the JVM
> 2. developing an in memory implementation (i.e. without hadoop)
> 3. reduce the size of the matrix (by removing those which have no words
> in common, for example)
> 4. run on real hadoop cluster with several nodes (does anyone have an
> idea of ​​the number of nodes to make it interesting)
> 
> Thanks for your help,
> Yann.