You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Grant Ingersoll <gs...@apache.org> on 2011/08/22 16:48:28 UTC
Email and Collab. Filtering
I'm working on an example (well, examples) of using Mahout with the ASF Public Data Set up on Amazon (http://aws.amazon.com/datasets/7791434387204566) and I wanted to show how to use the 3 "C's" (collab filtering, clustering, classification) with the data set. Clustering and classification are pretty straight forward, but I'm wondering about the setup around collaborative filtering.
The motivation for recommendations is pretty straightforward: provide people recs on emails that they might find useful based on what other people have interacted with. The tricky part is I am not totally sure on a valid setup of the problem. My current thinking is that I build up the rec. matrix based on whether someone has interacted with (initiated/replied) a thread or not. Thus, the columns are the thread ids and the rows are the users. Each cell contains the count of the number of times user X has interacted with thread Y. This feels to me like it is a stand in for that user's preference in that if they are replying multiple times, they have an interest in that topic. I have no idea if this will be effective or not, but it seems like it could be interesting. Does it sound reasonable? I worry that even in a really large data set as above it will simply be too sparse.
Is there a better way to think about this from a strict collaborative filtering context? In other words, I know I could do content-based recommendations but that is not what I am after here.
-Grant
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Email and Collab. Filtering
Posted by Lance Norskog <go...@gmail.com>.
With collaborative filtering: the input database is so choppy and
heterogeneous that it will be hard to get useful recs using naive
techniques. It would be a good exercise in 'know your data': instead
of finding preferences you hunt for features: does this user send lots
of mails in the evening? Do these three go round and round a lot?
You would also need good analytics software to display the results.
That's another class!
Perhaps you do some text-ish thing with text? Creating abstracts is
something that turns up a lot. You make a vector for every sentence in
a document, then do SVD on the vectors. The feature vectors are bags
of words, not correlated to sentences. You pick the first feature
vector, and find the closest sentence. That sentence is the primary
statement of the topic. Next, find a second sentence that matches the
second feature vector: this will be orthogonal to the main sentence,
and will modify it in some way. I would expect a misfire rate for
this, and would try sentence parts instead. Perhaps the vector could
be an n-gram of successive sentence clauses.
This is real machine learning that works on a very small dataset. It
could be a fun class example. In this context it could be used to
identify mail with coherent thought v.s. back&forth conversations.
On Mon, Aug 22, 2011 at 9:14 AM, Sean Owen <sr...@gmail.com> wrote:
> Here are two ideas:
>
> Recommend users to users.
> Your users and items are both e-mail senders. The strength of the
> association could be the number of e-mails from A to B (or perhaps the
> logarithm). This would find people that people like you e-mail a lot.
> Sounds interesting, if not immediately useful, because people e-mail
> others for very different reasons.
>
> Recommend threads to users.
> Users are people, items are threads. This might suggest discussions
> you should be a party to, or may be of interest since it concerns
> people you often share a thread with. I think it has slightly more
> potential to be useful, but, probably a non-starter in practice as
> it's not generally true that you'er welcome to see a thread you
> weren't copied on.
>
> Recommend users to threads.
> Kind of the "have you forgotten to include X" function from Gmail.
> Users are threads and items are people.
>
>
> All of these are sort of novelties -- I don't think CF applies so well
> -- but surely worth trying to see what you get out.
>
> In the book I tried recommending Wikipedia articles to Wikipedia
> articles -- discovering missing hyperlinks so to speak -- and while it
> was a bit novelty the results were intriguing and entertaining.
>
>
> On Mon, Aug 22, 2011 at 3:48 PM, Grant Ingersoll <gs...@apache.org> wrote:
>> I'm working on an example (well, examples) of using Mahout with the ASF Public Data Set up on Amazon (http://aws.amazon.com/datasets/7791434387204566) and I wanted to show how to use the 3 "C's" (collab filtering, clustering, classification) with the data set. Clustering and classification are pretty straight forward, but I'm wondering about the setup around collaborative filtering.
>>
>> The motivation for recommendations is pretty straightforward: provide people recs on emails that they might find useful based on what other people have interacted with. The tricky part is I am not totally sure on a valid setup of the problem. My current thinking is that I build up the rec. matrix based on whether someone has interacted with (initiated/replied) a thread or not. Thus, the columns are the thread ids and the rows are the users. Each cell contains the count of the number of times user X has interacted with thread Y. This feels to me like it is a stand in for that user's preference in that if they are replying multiple times, they have an interest in that topic. I have no idea if this will be effective or not, but it seems like it could be interesting. Does it sound reasonable? I worry that even in a really large data set as above it will simply be too sparse.
>>
>> Is there a better way to think about this from a strict collaborative filtering context? In other words, I know I could do content-based recommendations but that is not what I am after here.
>>
>> -Grant
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
Sean,
If you have a moment, I'd appreciate your thoughts on MAHOUT-802.
Thanks,
Grant
On Sep 1, 2011, at 11:37 AM, Sean Owen wrote:
> That's right. You'd just have to place the input where it expects that first
> phase's output to have been dumped, and that's a subdirectory of your --temp
> directory (userVectors I think).
>
> On Thu, Sep 1, 2011 at 4:30 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
>>
>> On Sep 1, 2011, at 10:04 AM, Sean Owen wrote:
>>
>>> Your input needs to be CSV if you want to use it all as-is. But, it
>> quickly
>>> creates vectors out of things, so really you can comment out the first
>>> mapper than creates user vectors, and just wire it to use yours instead.
>> it
>>> should do all the rest from there.
>>>
>>
>> I could use the --startPhase functionality to skip the first two phases,
>> right?
>>
>>
Re: Email and Collab. Filtering
Posted by Sean Owen <sr...@gmail.com>.
That's right. You'd just have to place the input where it expects that first
phase's output to have been dumped, and that's a subdirectory of your --temp
directory (userVectors I think).
On Thu, Sep 1, 2011 at 4:30 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Sep 1, 2011, at 10:04 AM, Sean Owen wrote:
>
> > Your input needs to be CSV if you want to use it all as-is. But, it
> quickly
> > creates vectors out of things, so really you can comment out the first
> > mapper than creates user vectors, and just wire it to use yours instead.
> it
> > should do all the rest from there.
> >
>
> I could use the --startPhase functionality to skip the first two phases,
> right?
>
>
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
On Sep 1, 2011, at 10:04 AM, Sean Owen wrote:
> Your input needs to be CSV if you want to use it all as-is. But, it quickly
> creates vectors out of things, so really you can comment out the first
> mapper than creates user vectors, and just wire it to use yours instead. it
> should do all the rest from there.
>
I could use the --startPhase functionality to skip the first two phases, right?
> On Thu, Sep 1, 2011 at 2:58 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
>> Assuming I've done my own translation (I followed Ted's piece), how do I
>> get this into the rest of the RecJob? Right now, I have a NamedVector (the
>> name is the id of the from email address) and the cells are {0,1} for each
>> message id (1 if that user has interacted with that message id). In looking
>> at the RecommenderJob, it seems like I could skip the first couple of
>> phases, but it also seems like I need a DistributedRowMatrix as input for
>> the next phase (maybePruneAndTranspose). Is my understanding correct? I
>> guess I need to convert my seq. file of NamedVectors to the
>> DistributedRowMatrix?
>>
>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Eurocon 2011: http://www.lucene-eurocon.com
Re: Email and Collab. Filtering
Posted by Sean Owen <sr...@gmail.com>.
Your input needs to be CSV if you want to use it all as-is. But, it quickly
creates vectors out of things, so really you can comment out the first
mapper than creates user vectors, and just wire it to use yours instead. it
should do all the rest from there.
On Thu, Sep 1, 2011 at 2:58 PM, Grant Ingersoll <gs...@apache.org> wrote:
> Assuming I've done my own translation (I followed Ted's piece), how do I
> get this into the rest of the RecJob? Right now, I have a NamedVector (the
> name is the id of the from email address) and the cells are {0,1} for each
> message id (1 if that user has interacted with that message id). In looking
> at the RecommenderJob, it seems like I could skip the first couple of
> phases, but it also seems like I need a DistributedRowMatrix as input for
> the next phase (maybePruneAndTranspose). Is my understanding correct? I
> guess I need to convert my seq. file of NamedVectors to the
> DistributedRowMatrix?
>
>
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
Assuming I've done my own translation (I followed Ted's piece), how do I get this into the rest of the RecJob? Right now, I have a NamedVector (the name is the id of the from email address) and the cells are {0,1} for each message id (1 if that user has interacted with that message id). In looking at the RecommenderJob, it seems like I could skip the first couple of phases, but it also seems like I need a DistributedRowMatrix as input for the next phase (maybePruneAndTranspose). Is my understanding correct? I guess I need to convert my seq. file of NamedVectors to the DistributedRowMatrix?
On Aug 31, 2011, at 11:55 AM, Sean Owen wrote:
> Yes, I'm suggesting that could at least be 80% of what you need. If you can
> generalize that bit further and refactor it, all the better.
>
> I wouldn't bother necessarily extending to support the "user: item item
> item" syntax or else we'd get into supporting a lot of stuff. That
> conversion IMHO can be left to the caller.
>
> On Wed, Aug 31, 2011 at 4:52 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
>>
>> On Aug 31, 2011, at 11:47 AM, Sean Owen wrote:
>>
>>> No it still wants "user,item[,rating]" input. But otherwise yes, it's
>>> translated and un-translated internally as needed.
>>>
>>> You could change the mapper to read that input easily though.
>>>
>>> it still wants numeric input. It's hashing longs to ints. But this could
>>> easily be changed to record a more general mapping.
>>
>> Ah, so I would still have to do the conversion, or hash on the string.
>>
>>>
>>> On Wed, Aug 31, 2011 at 4:44 PM, Grant Ingersoll <gsingers@apache.org
>>> wrote:
>>>
>>>>
>>>> On Aug 31, 2011, at 11:26 AM, Sean Owen wrote:
>>>>
>>>>> Is the problem not just a matter of "translating" from the original
>>>>> identifiers to ints, so they can be used as offsets into a vector, and
>>>> then
>>>>> back again?
>>>>
>>>> Yeah, I was wondering about that when looking at the RecommenderJob.
>>>>
>>>> If I understand you right, I could just output lines of text as:
>>>> from: msgId1, msgId3, ... msgIdn
>>>> ...
>>>>
>>>> And the RecommenderJob would automatically do the translation?
>>>>
>>>>
>>
>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Eurocon 2011: http://www.lucene-eurocon.com
Re: Email and Collab. Filtering
Posted by Sean Owen <sr...@gmail.com>.
Yes, I'm suggesting that could at least be 80% of what you need. If you can
generalize that bit further and refactor it, all the better.
I wouldn't bother necessarily extending to support the "user: item item
item" syntax or else we'd get into supporting a lot of stuff. That
conversion IMHO can be left to the caller.
On Wed, Aug 31, 2011 at 4:52 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Aug 31, 2011, at 11:47 AM, Sean Owen wrote:
>
> > No it still wants "user,item[,rating]" input. But otherwise yes, it's
> > translated and un-translated internally as needed.
> >
> > You could change the mapper to read that input easily though.
> >
> > it still wants numeric input. It's hashing longs to ints. But this could
> > easily be changed to record a more general mapping.
>
> Ah, so I would still have to do the conversion, or hash on the string.
>
> >
> > On Wed, Aug 31, 2011 at 4:44 PM, Grant Ingersoll <gsingers@apache.org
> >wrote:
> >
> >>
> >> On Aug 31, 2011, at 11:26 AM, Sean Owen wrote:
> >>
> >>> Is the problem not just a matter of "translating" from the original
> >>> identifiers to ints, so they can be used as offsets into a vector, and
> >> then
> >>> back again?
> >>
> >> Yeah, I was wondering about that when looking at the RecommenderJob.
> >>
> >> If I understand you right, I could just output lines of text as:
> >> from: msgId1, msgId3, ... msgIdn
> >> ...
> >>
> >> And the RecommenderJob would automatically do the translation?
> >>
> >>
>
>
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 31, 2011, at 11:47 AM, Sean Owen wrote:
> No it still wants "user,item[,rating]" input. But otherwise yes, it's
> translated and un-translated internally as needed.
>
> You could change the mapper to read that input easily though.
>
> it still wants numeric input. It's hashing longs to ints. But this could
> easily be changed to record a more general mapping.
Ah, so I would still have to do the conversion, or hash on the string.
>
> On Wed, Aug 31, 2011 at 4:44 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
>>
>> On Aug 31, 2011, at 11:26 AM, Sean Owen wrote:
>>
>>> Is the problem not just a matter of "translating" from the original
>>> identifiers to ints, so they can be used as offsets into a vector, and
>> then
>>> back again?
>>
>> Yeah, I was wondering about that when looking at the RecommenderJob.
>>
>> If I understand you right, I could just output lines of text as:
>> from: msgId1, msgId3, ... msgIdn
>> ...
>>
>> And the RecommenderJob would automatically do the translation?
>>
>>
Re: Email and Collab. Filtering
Posted by Sean Owen <sr...@gmail.com>.
No it still wants "user,item[,rating]" input. But otherwise yes, it's
translated and un-translated internally as needed.
You could change the mapper to read that input easily though.
it still wants numeric input. It's hashing longs to ints. But this could
easily be changed to record a more general mapping.
On Wed, Aug 31, 2011 at 4:44 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Aug 31, 2011, at 11:26 AM, Sean Owen wrote:
>
> > Is the problem not just a matter of "translating" from the original
> > identifiers to ints, so they can be used as offsets into a vector, and
> then
> > back again?
>
> Yeah, I was wondering about that when looking at the RecommenderJob.
>
> If I understand you right, I could just output lines of text as:
> from: msgId1, msgId3, ... msgIdn
> ...
>
> And the RecommenderJob would automatically do the translation?
>
>
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 31, 2011, at 11:26 AM, Sean Owen wrote:
> Is the problem not just a matter of "translating" from the original
> identifiers to ints, so they can be used as offsets into a vector, and then
> back again?
Yeah, I was wondering about that when looking at the RecommenderJob.
If I understand you right, I could just output lines of text as:
from: msgId1, msgId3, ... msgIdn
...
And the RecommenderJob would automatically do the translation?
Re: Email and Collab. Filtering
Posted by Sean Owen <sr...@gmail.com>.
Is the problem not just a matter of "translating" from the original
identifiers to ints, so they can be used as offsets into a vector, and then
back again?
The recommender stuff has a stage for this already which hashes identifiers
to an int, stores the mapping, and un-does it at the end. In between,
everything works in terms of those ints. It all works fine this way.
On Wed, Aug 31, 2011 at 4:21 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Aug 22, 2011, at 12:14 PM, Sean Owen wrote:
>
> > Here are two ideas:
> >
> > Recommend threads to users.
> > Users are people, items are threads. This might suggest discussions
> > you should be a party to, or may be of interest since it concerns
> > people you often share a thread with. I think it has slightly more
> > potential to be useful, but, probably a non-starter in practice as
> > it's not generally true that you'er welcome to see a thread you
> > weren't copied on.
>
> This is the one I am doing. But it brings up an interesting question in
> how best to convert the input to ids.
>
> To do this, I need to convert the strings (message id, from) to ids. Thus,
> I more or less modeled the code after what DictionaryVectorizer does.
> Creating the dictionaries is pretty straightforward and we likely now have
> an opportunity to make a general purpose tool that does it in an M/R way.
>
> Digging in a bit more, I am now working on doing the actual matrix
> creation. In my case, I have two dictionaries: message ids and from
> emails. In DictionaryVectorizer (used to take text to sparse vectors, which
> is comparable to what I need to do), it creates the matrix by running:
>
> for each dictionary chunk
> for each piece of text //i.e. the input sequence file, handled by
> Hadoop
> create the (partial) vector
>
> My initial thoughts for my case are to do:
>
> for each from id dictionary chunk
> for each message id dictionary chunk
> for each piece of text //i.e. the input seq. file, handled
> by Hadoop
> create the vector
>
> The output would be, for each "from" a list of message ids that the person
> interacted with (initiated or replied). It's likely that some of this moot
> b/c there will only ever be 1 or two chunks, esp. for the "froms".
>
> As you can no doubt see, that's a lot of loops and add on top of it you
> figure the hit ratio is pretty sparse. I believe the reason we do this in
> DictionaryVectorizer is so that we can use a predictable amount of memory in
> dealing with the dictionaries.
>
> Is there a better way of doing this?
>
> -Grant
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 31, 2011, at 11:43 AM, Ted Dunning wrote:
> My own recommendation here would be to run a word-count first and then
> create a dense dictionary using a sequential process. This sequential step
> should be very fast because the number of items is quite modest.
>
> I would create an additional dictionary at the same time for email
> addresses.
That's what I've done so far.
>
> Once you have the dictionary, you can distribute it to all nodes so that
> each node can extract a slice of the required matrix using a map-only
> program.
So, the mapper just loops over the dictionary chunks or are you saying that the dictionary(ies) all fit into memory? I think this is what I was doing other than I was modeling it after DictionaryVectorizer, which assumes the dictionary may become to big to fit into memory and thus it works on chunks at a time, even though in many cases there will only ever be 1 chunk.
>
> On Wed, Aug 31, 2011 at 8:21 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
>>
>> On Aug 22, 2011, at 12:14 PM, Sean Owen wrote:
>>
>>> Here are two ideas:
>>>
>>> Recommend threads to users.
>>> Users are people, items are threads. This might suggest discussions
>>> you should be a party to, or may be of interest since it concerns
>>> people you often share a thread with. I think it has slightly more
>>> potential to be useful, but, probably a non-starter in practice as
>>> it's not generally true that you'er welcome to see a thread you
>>> weren't copied on.
>>
>> This is the one I am doing. But it brings up an interesting question in
>> how best to convert the input to ids.
>>
>> To do this, I need to convert the strings (message id, from) to ids. Thus,
>> I more or less modeled the code after what DictionaryVectorizer does.
>> Creating the dictionaries is pretty straightforward and we likely now have
>> an opportunity to make a general purpose tool that does it in an M/R way.
>>
>> Digging in a bit more, I am now working on doing the actual matrix
>> creation. In my case, I have two dictionaries: message ids and from
>> emails. In DictionaryVectorizer (used to take text to sparse vectors, which
>> is comparable to what I need to do), it creates the matrix by running:
>>
>> for each dictionary chunk
>> for each piece of text //i.e. the input sequence file, handled by
>> Hadoop
>> create the (partial) vector
>>
>> My initial thoughts for my case are to do:
>>
>> for each from id dictionary chunk
>> for each message id dictionary chunk
>> for each piece of text //i.e. the input seq. file, handled
>> by Hadoop
>> create the vector
>>
>> The output would be, for each "from" a list of message ids that the person
>> interacted with (initiated or replied). It's likely that some of this moot
>> b/c there will only ever be 1 or two chunks, esp. for the "froms".
>>
>> As you can no doubt see, that's a lot of loops and add on top of it you
>> figure the hit ratio is pretty sparse. I believe the reason we do this in
>> DictionaryVectorizer is so that we can use a predictable amount of memory in
>> dealing with the dictionaries.
>>
>> Is there a better way of doing this?
>>
>> -Grant
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Eurocon 2011: http://www.lucene-eurocon.com
Re: Email and Collab. Filtering
Posted by Ted Dunning <te...@gmail.com>.
My own recommendation here would be to run a word-count first and then
create a dense dictionary using a sequential process. This sequential step
should be very fast because the number of items is quite modest.
I would create an additional dictionary at the same time for email
addresses.
Once you have the dictionary, you can distribute it to all nodes so that
each node can extract a slice of the required matrix using a map-only
program.
On Wed, Aug 31, 2011 at 8:21 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Aug 22, 2011, at 12:14 PM, Sean Owen wrote:
>
> > Here are two ideas:
> >
> > Recommend threads to users.
> > Users are people, items are threads. This might suggest discussions
> > you should be a party to, or may be of interest since it concerns
> > people you often share a thread with. I think it has slightly more
> > potential to be useful, but, probably a non-starter in practice as
> > it's not generally true that you'er welcome to see a thread you
> > weren't copied on.
>
> This is the one I am doing. But it brings up an interesting question in
> how best to convert the input to ids.
>
> To do this, I need to convert the strings (message id, from) to ids. Thus,
> I more or less modeled the code after what DictionaryVectorizer does.
> Creating the dictionaries is pretty straightforward and we likely now have
> an opportunity to make a general purpose tool that does it in an M/R way.
>
> Digging in a bit more, I am now working on doing the actual matrix
> creation. In my case, I have two dictionaries: message ids and from
> emails. In DictionaryVectorizer (used to take text to sparse vectors, which
> is comparable to what I need to do), it creates the matrix by running:
>
> for each dictionary chunk
> for each piece of text //i.e. the input sequence file, handled by
> Hadoop
> create the (partial) vector
>
> My initial thoughts for my case are to do:
>
> for each from id dictionary chunk
> for each message id dictionary chunk
> for each piece of text //i.e. the input seq. file, handled
> by Hadoop
> create the vector
>
> The output would be, for each "from" a list of message ids that the person
> interacted with (initiated or replied). It's likely that some of this moot
> b/c there will only ever be 1 or two chunks, esp. for the "froms".
>
> As you can no doubt see, that's a lot of loops and add on top of it you
> figure the hit ratio is pretty sparse. I believe the reason we do this in
> DictionaryVectorizer is so that we can use a predictable amount of memory in
> dealing with the dictionaries.
>
> Is there a better way of doing this?
>
> -Grant
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 22, 2011, at 12:14 PM, Sean Owen wrote:
> Here are two ideas:
>
> Recommend threads to users.
> Users are people, items are threads. This might suggest discussions
> you should be a party to, or may be of interest since it concerns
> people you often share a thread with. I think it has slightly more
> potential to be useful, but, probably a non-starter in practice as
> it's not generally true that you'er welcome to see a thread you
> weren't copied on.
This is the one I am doing. But it brings up an interesting question in how best to convert the input to ids.
To do this, I need to convert the strings (message id, from) to ids. Thus, I more or less modeled the code after what DictionaryVectorizer does. Creating the dictionaries is pretty straightforward and we likely now have an opportunity to make a general purpose tool that does it in an M/R way.
Digging in a bit more, I am now working on doing the actual matrix creation. In my case, I have two dictionaries: message ids and from emails. In DictionaryVectorizer (used to take text to sparse vectors, which is comparable to what I need to do), it creates the matrix by running:
for each dictionary chunk
for each piece of text //i.e. the input sequence file, handled by Hadoop
create the (partial) vector
My initial thoughts for my case are to do:
for each from id dictionary chunk
for each message id dictionary chunk
for each piece of text //i.e. the input seq. file, handled by Hadoop
create the vector
The output would be, for each "from" a list of message ids that the person interacted with (initiated or replied). It's likely that some of this moot b/c there will only ever be 1 or two chunks, esp. for the "froms".
As you can no doubt see, that's a lot of loops and add on top of it you figure the hit ratio is pretty sparse. I believe the reason we do this in DictionaryVectorizer is so that we can use a predictable amount of memory in dealing with the dictionaries.
Is there a better way of doing this?
-Grant
Re: Email and Collab. Filtering
Posted by Sean Owen <sr...@gmail.com>.
Here are two ideas:
Recommend users to users.
Your users and items are both e-mail senders. The strength of the
association could be the number of e-mails from A to B (or perhaps the
logarithm). This would find people that people like you e-mail a lot.
Sounds interesting, if not immediately useful, because people e-mail
others for very different reasons.
Recommend threads to users.
Users are people, items are threads. This might suggest discussions
you should be a party to, or may be of interest since it concerns
people you often share a thread with. I think it has slightly more
potential to be useful, but, probably a non-starter in practice as
it's not generally true that you'er welcome to see a thread you
weren't copied on.
Recommend users to threads.
Kind of the "have you forgotten to include X" function from Gmail.
Users are threads and items are people.
All of these are sort of novelties -- I don't think CF applies so well
-- but surely worth trying to see what you get out.
In the book I tried recommending Wikipedia articles to Wikipedia
articles -- discovering missing hyperlinks so to speak -- and while it
was a bit novelty the results were intriguing and entertaining.
On Mon, Aug 22, 2011 at 3:48 PM, Grant Ingersoll <gs...@apache.org> wrote:
> I'm working on an example (well, examples) of using Mahout with the ASF Public Data Set up on Amazon (http://aws.amazon.com/datasets/7791434387204566) and I wanted to show how to use the 3 "C's" (collab filtering, clustering, classification) with the data set. Clustering and classification are pretty straight forward, but I'm wondering about the setup around collaborative filtering.
>
> The motivation for recommendations is pretty straightforward: provide people recs on emails that they might find useful based on what other people have interacted with. The tricky part is I am not totally sure on a valid setup of the problem. My current thinking is that I build up the rec. matrix based on whether someone has interacted with (initiated/replied) a thread or not. Thus, the columns are the thread ids and the rows are the users. Each cell contains the count of the number of times user X has interacted with thread Y. This feels to me like it is a stand in for that user's preference in that if they are replying multiple times, they have an interest in that topic. I have no idea if this will be effective or not, but it seems like it could be interesting. Does it sound reasonable? I worry that even in a really large data set as above it will simply be too sparse.
>
> Is there a better way to think about this from a strict collaborative filtering context? In other words, I know I could do content-based recommendations but that is not what I am after here.
>
> -Grant
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
Re: Email and Collab. Filtering
Posted by Lance Norskog <go...@gmail.com>.
> I should add, of course, before lightning strikes that your mileage may
vary.
Know Your Data. I got far with KNime (visual programming for statistics) and
still use it, before breaking down and using Excel.. There's a brilliant
book called "Think Stats: Probability and statics for programmers". It is
not brilliant in execution but in concept: it plays upon the deep allergy of
programmers for Excel et. al. that it does all of its examples in Python
scripts. "Excel" and "spreadsheet" were not in the index. Skinny little book
for $30, but you can download it for free.
http://greenteapress.com/thinkstats/
The problem with ratings is very basic marketing survey principles: it is a
self-selected sample. If you want valid results in a marketing survey, you
pick the sample, he does not pick himself. But, there is a binary feature
per user: are the ratings 3.0 (default) or was the user engaged enough to
push the button? If you do some charting on this with your dataset, you may
find that because the UI is set for a default of 3.0, you can separate two
sets of users very reliably.
On Tue, Sep 6, 2011 at 2:25 PM, Ted Dunning <te...@gmail.com> wrote:
> My rationale for being such a binary bigot is that I have found that (in my
> experience) one signal always dominates pretty much completely. Other
> signals are pretty much just noise (too little engagement) are subject to
> spammy misdirection (bad titles on videos, for instance) or are too rare to
> give any significant lift (user ratings versus views/engagements).
>
> In cases where the alternative signal is more voluminous than the
> engagement
> that I am interested in, it is invariable very noisy. This is guaranteed
> since I would otherwise have used the higher volume signal. In every case
> I
> have tried, using the high volume, high noise signal degraded performance
> significantly because it made it hard to find the clean signal. The low
> volume signals have never led to any gain and often were strange enough
> that
> they hurt things badly. Besides, they typically are much less than 10% of
> the data.
>
> Aside from the general data quality and availability issues, there are the
> computational issues. Having binary data allows me to use much faster and
> cooler algorithms like LLR.
>
> The upshot is that I don't see anything but downside for including rating
> or
> synthetic rating data.
>
> I should add, of course, before lightning strikes that your mileage may
> vary.
>
> On Tue, Sep 6, 2011 at 12:56 PM, Grant Ingersoll <gsingers@apache.org
> >wrote:
>
> > Ted,
> >
> > Been meaning to follow up on this...
> >
> > On Aug 22, 2011, at 11:29 AM, Ted Dunning wrote:
> >
> > > On Mon, Aug 22, 2011 at 8:21 AM, Daniel Xiaodan Zhou <
> > danithaca@gmail.com>wrote:
> > >
> > >> I think this is reasonable. Some suggestions:
> > >>
> > >> 1. Instead of using the total number of interactions as cell value,
> map
> > the
> > >> number to a 1-5 score based on histogram
> > >>
> > >
> > > I would map to {0,1} rather than a fake rating scale.
> >
> > What's your reasoning for this, versus, something like number of replies?
> > My somewhat naive intuition thought that I would want to somehow capture
> > the fact that a particular user has interacted more frequently with an
> item
> > vs. simply a boolean preference. Or, is it just that in the big scheme
> of
> > things, it won't matter much, so why complicate it?
> >
> > Thanks,
> > Grant
> >
> >
> > --------------------------------------------
> > Grant Ingersoll
> > http://www.lucidimagination.com
> > Lucene Eurocon 2011: http://www.lucene-eurocon.com
> >
> >
>
--
Lance Norskog
goksron@gmail.com
Re: Email and Collab. Filtering
Posted by Ted Dunning <te...@gmail.com>.
My rationale for being such a binary bigot is that I have found that (in my
experience) one signal always dominates pretty much completely. Other
signals are pretty much just noise (too little engagement) are subject to
spammy misdirection (bad titles on videos, for instance) or are too rare to
give any significant lift (user ratings versus views/engagements).
In cases where the alternative signal is more voluminous than the engagement
that I am interested in, it is invariable very noisy. This is guaranteed
since I would otherwise have used the higher volume signal. In every case I
have tried, using the high volume, high noise signal degraded performance
significantly because it made it hard to find the clean signal. The low
volume signals have never led to any gain and often were strange enough that
they hurt things badly. Besides, they typically are much less than 10% of
the data.
Aside from the general data quality and availability issues, there are the
computational issues. Having binary data allows me to use much faster and
cooler algorithms like LLR.
The upshot is that I don't see anything but downside for including rating or
synthetic rating data.
I should add, of course, before lightning strikes that your mileage may
vary.
On Tue, Sep 6, 2011 at 12:56 PM, Grant Ingersoll <gs...@apache.org>wrote:
> Ted,
>
> Been meaning to follow up on this...
>
> On Aug 22, 2011, at 11:29 AM, Ted Dunning wrote:
>
> > On Mon, Aug 22, 2011 at 8:21 AM, Daniel Xiaodan Zhou <
> danithaca@gmail.com>wrote:
> >
> >> I think this is reasonable. Some suggestions:
> >>
> >> 1. Instead of using the total number of interactions as cell value, map
> the
> >> number to a 1-5 score based on histogram
> >>
> >
> > I would map to {0,1} rather than a fake rating scale.
>
> What's your reasoning for this, versus, something like number of replies?
> My somewhat naive intuition thought that I would want to somehow capture
> the fact that a particular user has interacted more frequently with an item
> vs. simply a boolean preference. Or, is it just that in the big scheme of
> things, it won't matter much, so why complicate it?
>
> Thanks,
> Grant
>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
> Lucene Eurocon 2011: http://www.lucene-eurocon.com
>
>
Re: Email and Collab. Filtering
Posted by Grant Ingersoll <gs...@apache.org>.
Ted,
Been meaning to follow up on this...
On Aug 22, 2011, at 11:29 AM, Ted Dunning wrote:
> On Mon, Aug 22, 2011 at 8:21 AM, Daniel Xiaodan Zhou <da...@gmail.com>wrote:
>
>> I think this is reasonable. Some suggestions:
>>
>> 1. Instead of using the total number of interactions as cell value, map the
>> number to a 1-5 score based on histogram
>>
>
> I would map to {0,1} rather than a fake rating scale.
What's your reasoning for this, versus, something like number of replies? My somewhat naive intuition thought that I would want to somehow capture the fact that a particular user has interacted more frequently with an item vs. simply a boolean preference. Or, is it just that in the big scheme of things, it won't matter much, so why complicate it?
Thanks,
Grant
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Lucene Eurocon 2011: http://www.lucene-eurocon.com
Re: Email and Collab. Filtering
Posted by Ted Dunning <te...@gmail.com>.
On Mon, Aug 22, 2011 at 8:21 AM, Daniel Xiaodan Zhou <da...@gmail.com>wrote:
> I think this is reasonable. Some suggestions:
>
> 1. Instead of using the total number of interactions as cell value, map the
> number to a 1-5 score based on histogram
>
I would map to {0,1} rather than a fake rating scale.
> 2. Use item-item algorithm, which is supposed to work for sparse data.
>
sort of works with sparse data.
> 3. I think the best algorithm to handle sparse data is the SVD algorithm.
>
Yes.
Re: Email and Collab. Filtering
Posted by Daniel Xiaodan Zhou <da...@gmail.com>.
I think this is reasonable. Some suggestions:
1. Instead of using the total number of interactions as cell value, map the number to a 1-5 score based on histogram
2. Use item-item algorithm, which is supposed to work for sparse data.
3. I think the best algorithm to handle sparse data is the SVD algorithm.
4. Research shows that "accuracy" is not the only way to evaluate a CF use scenario, so you might want to show "explains" (ie, why the recommendations are made), etc.
Just my 2 cents.
-daniel
--
Daniel Xiaodan Zhou
PhD student
School of Information
University of Michigan
http://michiza.com
On Aug 22, 2011, at 10:48 AM, Grant Ingersoll wrote:
> I'm working on an example (well, examples) of using Mahout with the ASF Public Data Set up on Amazon (http://aws.amazon.com/datasets/7791434387204566) and I wanted to show how to use the 3 "C's" (collab filtering, clustering, classification) with the data set. Clustering and classification are pretty straight forward, but I'm wondering about the setup around collaborative filtering.
>
> The motivation for recommendations is pretty straightforward: provide people recs on emails that they might find useful based on what other people have interacted with. The tricky part is I am not totally sure on a valid setup of the problem. My current thinking is that I build up the rec. matrix based on whether someone has interacted with (initiated/replied) a thread or not. Thus, the columns are the thread ids and the rows are the users. Each cell contains the count of the number of times user X has interacted with thread Y. This feels to me like it is a stand in for that user's preference in that if they are replying multiple times, they have an interest in that topic. I have no idea if this will be effective or not, but it seems like it could be interesting. Does it sound reasonable? I worry that even in a really large data set as above it will simply be too sparse.
>
> Is there a better way to think about this from a strict collaborative filtering context? In other words, I know I could do content-based recommendations but that is not what I am after here.
>
> -Grant
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
Re: Email and Collab. Filtering
Posted by Ted Dunning <te...@gmail.com>.
I think single interactions in a thread is an excellent indicator of
interest. This can give thread recommendations as well as related person
recommendations.
I think it will be interesting to see just how sparse it is.
On Mon, Aug 22, 2011 at 7:48 AM, Grant Ingersoll <gs...@apache.org>wrote:
> I'm working on an example (well, examples) of using Mahout with the ASF
> Public Data Set up on Amazon (
> http://aws.amazon.com/datasets/7791434387204566) and I wanted to show how
> to use the 3 "C's" (collab filtering, clustering, classification) with the
> data set. Clustering and classification are pretty straight forward, but
> I'm wondering about the setup around collaborative filtering.
>
> The motivation for recommendations is pretty straightforward: provide
> people recs on emails that they might find useful based on what other people
> have interacted with. The tricky part is I am not totally sure on a valid
> setup of the problem. My current thinking is that I build up the rec.
> matrix based on whether someone has interacted with (initiated/replied) a
> thread or not. Thus, the columns are the thread ids and the rows are the
> users. Each cell contains the count of the number of times user X has
> interacted with thread Y. This feels to me like it is a stand in for that
> user's preference in that if they are replying multiple times, they have an
> interest in that topic. I have no idea if this will be effective or not,
> but it seems like it could be interesting. Does it sound reasonable? I
> worry that even in a really large data set as above it will simply be too
> sparse.
>
> Is there a better way to think about this from a strict collaborative
> filtering context? In other words, I know I could do content-based
> recommendations but that is not what I am after here.
>
> -Grant
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>