You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Neil Ghosh <ne...@gmail.com> on 2010/09/30 16:37:49 UTC
Usage of TF-IDF weights in cbayes Mahout
Does anybody have examples/reference how to use TF-IDF weights in mahout
cbayes for particular words and phrases while doing text classification ?
--
Thanks and Regards
Neil
http://neilghosh.com
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Robin Anil <ro...@gmail.com>.
extra file or extra line, duplicated instances(to decrease the weights) or
duplicate feature in the same instance to increase the weights(classic
tf-idf)
Robin
On Thu, Sep 30, 2010 at 9:50 PM, Neil Ghosh <ne...@gmail.com> wrote:
> So All I have to do is add an extra file containing
>
> LABEL<TAB>problem<TAB>complaint<TAB>problemo
>
> Along with the usual training data in Bayes format ?
>
>
> On Thu, Sep 30, 2010 at 9:44 PM, Robin Anil <ro...@gmail.com> wrote:
>
>>
>>>> Or Do I have flexibility to give some other input specific to my problem
>>>> ? Such as if words like "Problem", "Complaint" etc are more likely to appear
>>>> in a text containing grievance.
>>>>
>>>>
>>>>
>>
>>
>>> You can provide a Weight, usually TF-IDF, that often does a good job of
>>>> factoring in the importance of words. If you have certain sentiment words
>>>> that you think influence things one way or the other, you could consider a
>>>> weighting process that adds weight to those words, I suppose, but I would
>>>> want to experiment with that a bit.
>>>>
>>>
>>> I would first get your data in the bayes format
>> <LABEL><TAB><FEATURE1><SPACE><FEATURE2>......
>>
>> Feature can be words, or pairs of word (word1_word2) or binned numerical
>> values ( 0.1, 0.2.. etc) or enums. (SEX:MALE, SEX:FEMALE)
>>
>> Give this as input to the classifier and get the output.
>>
>> If you need to add couple words hardcoded into the classifier. Add them as
>> a training instance. Since features are assumed to be independent in bayes.
>> it doesnt matter how you give them
>>
>> POS<TAB>problem<TAB>complaint<TAB>problemo
>>
>>
>>
>>
>>
>>
>>
>>>
>>>
>>> On Thu, Sep 30, 2010 at 8:55 PM, Robin Anil <ro...@gmail.com>wrote:
>>>
>>>> It does that by default for all words. What else do you have in mind?
>>>>
>>>> On Thu, Sep 30, 2010 at 8:07 PM, Neil Ghosh <ne...@gmail.com>wrote:
>>>>
>>>>> Does anybody have examples/reference how to use TF-IDF weights in
>>>>> mahout
>>>>> cbayes for particular words and phrases while doing text classification
>>>>> ?
>>>>>
>>>>> --
>>>>> Thanks and Regards
>>>>> Neil
>>>>> http://neilghosh.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks and Regards
>>> Neil
>>> http://neilghosh.com
>>>
>>>
>>>
>>>
>>
>
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>
>
>
>
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Neil Ghosh <ne...@gmail.com>.
So All I have to do is add an extra file containing
LABEL<TAB>problem<TAB>complaint<TAB>problemo
Along with the usual training data in Bayes format ?
On Thu, Sep 30, 2010 at 9:44 PM, Robin Anil <ro...@gmail.com> wrote:
>
>>> Or Do I have flexibility to give some other input specific to my problem
>>> ? Such as if words like "Problem", "Complaint" etc are more likely to appear
>>> in a text containing grievance.
>>>
>>>
>>>
>
>
>> You can provide a Weight, usually TF-IDF, that often does a good job of
>>> factoring in the importance of words. If you have certain sentiment words
>>> that you think influence things one way or the other, you could consider a
>>> weighting process that adds weight to those words, I suppose, but I would
>>> want to experiment with that a bit.
>>>
>>
>> I would first get your data in the bayes format
> <LABEL><TAB><FEATURE1><SPACE><FEATURE2>......
>
> Feature can be words, or pairs of word (word1_word2) or binned numerical
> values ( 0.1, 0.2.. etc) or enums. (SEX:MALE, SEX:FEMALE)
>
> Give this as input to the classifier and get the output.
>
> If you need to add couple words hardcoded into the classifier. Add them as
> a training instance. Since features are assumed to be independent in bayes.
> it doesnt matter how you give them
>
> POS<TAB>problem<TAB>complaint<TAB>problemo
>
>
>
>
>
>
>
>>
>>
>> On Thu, Sep 30, 2010 at 8:55 PM, Robin Anil <ro...@gmail.com> wrote:
>>
>>> It does that by default for all words. What else do you have in mind?
>>>
>>> On Thu, Sep 30, 2010 at 8:07 PM, Neil Ghosh <ne...@gmail.com>wrote:
>>>
>>>> Does anybody have examples/reference how to use TF-IDF weights in mahout
>>>> cbayes for particular words and phrases while doing text classification
>>>> ?
>>>>
>>>> --
>>>> Thanks and Regards
>>>> Neil
>>>> http://neilghosh.com
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards
>> Neil
>> http://neilghosh.com
>>
>>
>>
>>
>
--
Thanks and Regards
Neil
http://neilghosh.com
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Robin Anil <ro...@gmail.com>.
>
>
>> Or Do I have flexibility to give some other input specific to my problem ?
>> Such as if words like "Problem", "Complaint" etc are more likely to appear
>> in a text containing grievance.
>>
>>
>>
> You can provide a Weight, usually TF-IDF, that often does a good job of
>> factoring in the importance of words. If you have certain sentiment words
>> that you think influence things one way or the other, you could consider a
>> weighting process that adds weight to those words, I suppose, but I would
>> want to experiment with that a bit.
>>
>
> I would first get your data in the bayes format
<LABEL><TAB><FEATURE1><SPACE><FEATURE2>......
Feature can be words, or pairs of word (word1_word2) or binned numerical
values ( 0.1, 0.2.. etc) or enums. (SEX:MALE, SEX:FEMALE)
Give this as input to the classifier and get the output.
If you need to add couple words hardcoded into the classifier. Add them as a
training instance. Since features are assumed to be independent in bayes. it
doesnt matter how you give them
POS<TAB>problem<TAB>complaint<TAB>problemo
>
>
> On Thu, Sep 30, 2010 at 8:55 PM, Robin Anil <ro...@gmail.com> wrote:
>
>> It does that by default for all words. What else do you have in mind?
>>
>> On Thu, Sep 30, 2010 at 8:07 PM, Neil Ghosh <ne...@gmail.com> wrote:
>>
>>> Does anybody have examples/reference how to use TF-IDF weights in mahout
>>> cbayes for particular words and phrases while doing text classification ?
>>>
>>> --
>>> Thanks and Regards
>>> Neil
>>> http://neilghosh.com
>>>
>>
>>
>
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>
>
>
>
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Neil Ghosh <ne...@gmail.com>.
Thanks for replying Robin , I am quoting conversation between Grant and Me
earlier
Now I want to know how to implement the second problem ?
To be specific my problem is to classify a piece text crawled from web into
> two classes
>
> 1.It <http://1.it/> is a +ve feedback
> 2.It <http://2.it/> is -ve feed back.
>
> I can use the two news group example and create a model with some text
> (may be a large no of text ) by inputtng the trainer with these two
> labels.Should I leave everything to the trainer completely like this ?
>
>
> Yes, that should be fine. The trainer doesn't care about the name of the
> label, it just cares that the two sets are relatively independent. Keep in
> mind, you should set aside some of your data for testing as well.
>
> Or Do I have flexibility to give some other input specific to my problem ?
> Such as if words like "Problem", "Complaint" etc are more likely to appear
> in a text containing grievance.
>
>
> You can provide a Weight, usually TF-IDF, that often does a good job of
> factoring in the importance of words. If you have certain sentiment words
> that you think influence things one way or the other, you could consider a
> weighting process that adds weight to those words, I suppose, but I would
> want to experiment with that a bit.
>
On Thu, Sep 30, 2010 at 8:55 PM, Robin Anil <ro...@gmail.com> wrote:
> It does that by default for all words. What else do you have in mind?
>
> On Thu, Sep 30, 2010 at 8:07 PM, Neil Ghosh <ne...@gmail.com> wrote:
>
>> Does anybody have examples/reference how to use TF-IDF weights in mahout
>> cbayes for particular words and phrases while doing text classification ?
>>
>> --
>> Thanks and Regards
>> Neil
>> http://neilghosh.com
>>
>
>
--
Thanks and Regards
Neil
http://neilghosh.com
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Robin Anil <ro...@gmail.com>.
It does that by default for all words. What else do you have in mind?
On Thu, Sep 30, 2010 at 8:07 PM, Neil Ghosh <ne...@gmail.com> wrote:
> Does anybody have examples/reference how to use TF-IDF weights in mahout
> cbayes for particular words and phrases while doing text classification ?
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Neil Ghosh <ne...@gmail.com>.
Thanks for Clarification Ted/Robin.I got the point of TF and IDF.
Is there any other technique/weight optimization which I can use along with
Mahout especially while using cbayes ?
On Thu, Sep 30, 2010 at 11:30 PM, Ted Dunning <te...@gmail.com> wrote:
> That is exactly what it does.
>
> On Thu, Sep 30, 2010 at 8:37 AM, Neal Richter <nr...@gmail.com> wrote:
>
> > On Thu, Sep 30, 2010 at 8:37 AM, Neil Ghosh <ne...@gmail.com>
> wrote:
> > > Does anybody have examples/reference how to use TF-IDF weights in
> mahout
> > > cbayes for particular words and phrases while doing text classification
> ?
> >
> > http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf
> >
> > - Neal
> >
>
--
Thanks and Regards
Neil
http://neilghosh.com
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Ted Dunning <te...@gmail.com>.
That is exactly what it does.
On Thu, Sep 30, 2010 at 8:37 AM, Neal Richter <nr...@gmail.com> wrote:
> On Thu, Sep 30, 2010 at 8:37 AM, Neil Ghosh <ne...@gmail.com> wrote:
> > Does anybody have examples/reference how to use TF-IDF weights in mahout
> > cbayes for particular words and phrases while doing text classification ?
>
> http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf
>
> - Neal
>
Re: Usage of TF-IDF weights in cbayes Mahout
Posted by Neal Richter <nr...@gmail.com>.
On Thu, Sep 30, 2010 at 8:37 AM, Neil Ghosh <ne...@gmail.com> wrote:
> Does anybody have examples/reference how to use TF-IDF weights in mahout
> cbayes for particular words and phrases while doing text classification ?
http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf
- Neal