You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Naveenchandra <na...@gmail.com> on 2012/02/15 12:34:39 UTC
Naive-Bayes work flow
Dear all,
I am a newbie to mahout, currently working on naive bayes classification.
I ran the classification for the popular 20 newsgroups example and got quite
good result as gave in the mahout in action book.
But I tried to run the classifier on my own data set having 2 files each
containing 20k records i am getting a efficiency around 50%, please help me.
I would be grateful if you explain me how the actual process is taking place
Thanks a ton
-Naveenchandra
Re: Naive-Bayes work flow
Posted by Naveenchandra <na...@gmail.com>.
Hi Ted,
By the term efficiency i only mean the percentage of correctly classified
instances that's it.
And i am providing the same data-set for training as well as for testing, it
is in the format of ( target_variable'\t'data ) you want me to mail you the
data-set ?
I have a comma separated data-set so i just want to know if i could specify any
desired column as target-variable instead of the 1st column followed by tab ?
-Thank you
Naveenchandra
Re: Naive-Bayes work flow
Posted by Lance Norskog <go...@gmail.com>.
NGrams also has this effect in the ASF emails example.
On Wed, Feb 15, 2012 at 7:02 AM, Ted Dunning <te...@gmail.com> wrote:
> Efficiency is not normally a term used with classifiers. Can you define it?
>
> From you confusion matrix, it looks like nearly all of your documents are
> being classified into one class. That usually indicates that there is some
> fundamental formatting difference between your original training data and
> your test data.
>
> Without being able to see any of your data or the output of your training
> run, it is impossible to say more.
>
> On Wed, Feb 15, 2012 at 8:43 AM, Naveenchandra <na...@gmail.com>wrote:
>
>>
>> Hi,
>>
>> Thanks for taking your time and replying, its efficiency actually when i
>> the
>> tested the classifier with sequential as the method i got that.
>>
>> I wanted know what effect does the setting of ng makes on the result,
>> whether
>> increasing the ng increases the efficiency or it decreases ?
>>
>> Just now i trained and tested the naive bayes classifier on a input data
>> set
>> containing 2 csv files of around 30k records each, having 2 target
>> variables
>> set. I got the following confusion matrix with ng set to 3
>>
>>
>> Confusion Matrix
>> -------------------------------------------------------
>> a b <--Classified as
>> 32151 1 | 32152 a = target_a
>> 32095 57 | 32152 b = target_b
>>
>> The efficiency is 50% please help me to improve it.
>>
>>
>>
>>
>>
--
Lance Norskog
goksron@gmail.com
Re: Naive-Bayes work flow
Posted by Ted Dunning <te...@gmail.com>.
Efficiency is not normally a term used with classifiers. Can you define it?
>From you confusion matrix, it looks like nearly all of your documents are
being classified into one class. That usually indicates that there is some
fundamental formatting difference between your original training data and
your test data.
Without being able to see any of your data or the output of your training
run, it is impossible to say more.
On Wed, Feb 15, 2012 at 8:43 AM, Naveenchandra <na...@gmail.com>wrote:
>
> Hi,
>
> Thanks for taking your time and replying, its efficiency actually when i
> the
> tested the classifier with sequential as the method i got that.
>
> I wanted know what effect does the setting of ng makes on the result,
> whether
> increasing the ng increases the efficiency or it decreases ?
>
> Just now i trained and tested the naive bayes classifier on a input data
> set
> containing 2 csv files of around 30k records each, having 2 target
> variables
> set. I got the following confusion matrix with ng set to 3
>
>
> Confusion Matrix
> -------------------------------------------------------
> a b <--Classified as
> 32151 1 | 32152 a = target_a
> 32095 57 | 32152 b = target_b
>
> The efficiency is 50% please help me to improve it.
>
>
>
>
>
Re: Naive-Bayes work flow
Posted by Naveenchandra <na...@gmail.com>.
Hi,
Thanks for taking your time and replying, its efficiency actually when i the
tested the classifier with sequential as the method i got that.
I wanted know what effect does the setting of ng makes on the result, whether
increasing the ng increases the efficiency or it decreases ?
Just now i trained and tested the naive bayes classifier on a input data set
containing 2 csv files of around 30k records each, having 2 target variables
set. I got the following confusion matrix with ng set to 3
Confusion Matrix
-------------------------------------------------------
a b <--Classified as
32151 1 | 32152 a = target_a
32095 57 | 32152 b = target_b
The efficiency is 50% please help me to improve it.
Re: Naive-Bayes work flow
Posted by Naveenchandra <na...@gmail.com>.
Hi,
Thanks for your advice sreejith, but my problem is that i want to know exactly
where my csv file is read for training the classifier, do we have any java class
as csvfilereader which reads each line of the input csv file, i assume the class
org.apache.hadoop.mapred.keyvaluetextinputformat is called to perform.
But when i checked that java file i did not find any function to read the csv
file. Please help me
Re: Naive-Bayes work flow
Posted by Sreejith S <sr...@gmail.com>.
Hi Naveenchandra,
Some of your questions are already addressed in the mailing list.So please
take a look at the archives.:)
And it is not advisable in any mailing list to post questions with out have
a basic search in the past archives.
Thank You,
On Fri, Feb 17, 2012 at 9:28 AM, Naveenchandra <na...@gmail.com>wrote:
>
> Please can you explain me exactly in which format my input data set
> should be..
> I assume it is target_variable<tab>data(comma separated values)
>
> Also please explain me how naive bayes classifier works..
>
> It is calculating TfIdf, weights, ThetaNormalization.. How these are
> calculated
> and how the target variables of test data are compared with that of the
> trained
> data ?
>
> I think these are too silly questions to be asked but i want to learn
> from the
> scratch. Starting from the inputs taken from command line, how and where
> csv
> files are read, in which format they get stored, how they are processed,
> how
> does the classifier is being trained, what's the testing mechanism ?
>
> Thanks a ton
>
>
--
*Sreejith.S*
http://srijiths.wordpress.com/
* *http://sreejiths.emurse.com/
tweet2sree@twitter <http://tweet2Sree>
Re: Naive-Bayes work flow
Posted by Naveenchandra <na...@gmail.com>.
Please can you explain me exactly in which format my input data set should be..
I assume it is target_variable<tab>data(comma separated values)
Also please explain me how naive bayes classifier works..
It is calculating TfIdf, weights, ThetaNormalization.. How these are calculated
and how the target variables of test data are compared with that of the trained
data ?
I think these are too silly questions to be asked but i want to learn from the
scratch. Starting from the inputs taken from command line, how and where csv
files are read, in which format they get stored, how they are processed, how
does the classifier is being trained, what's the testing mechanism ?
Thanks a ton
Re: Naive-Bayes work flow
Posted by Lance Norskog <go...@gmail.com>.
I do not know why they were classified poorly. It would be really
helpful if the classifiers logged every item and the confidence level
for the classification choice. This would let you examine your data.
On Wed, Feb 15, 2012 at 10:52 PM, Naveenchandra
<na...@gmail.com> wrote:
>
> Hi,
>
> Lance you might have gone through the confusion matrix which i posted, can you
> rectify the problem why the target_b is so poorly classified ?
>
> My input as i said earlier are 2 csv files in which each line of record is in
> the format of "target_varible<tab>text"
>
--
Lance Norskog
goksron@gmail.com
Re: Naive-Bayes work flow
Posted by Naveenchandra <na...@gmail.com>.
Hi,
Lance you might have gone through the confusion matrix which i posted, can you
rectify the problem why the target_b is so poorly classified ?
My input as i said earlier are 2 csv files in which each line of record is in
the format of "target_varible<tab>text"
Re: Naive-Bayes work flow
Posted by Lance Norskog <go...@gmail.com>.
The file examples/bin/asf-examples.sh shows how to use the Naive Bayes
classifier. There are a few stages needed to prepare files before
they get to the classifier training and test passes.
If writing your own code, I would use the Apache Commons CSV parser:
http://commons.apache.org/sandbox/csv/
This is pulled into the Mahout integration/ sub-project, and is used
by the 'CSVVectorIterator'. If you need a Hadoop file reader for CSV,
you would create a new one from scratch.
On Wed, Feb 15, 2012 at 9:41 PM, Naveenchandra
<na...@gmail.com> wrote:
>
> Hi all,
>
> Can any one explain me how and where the input files are being read, now i am
> going through the source code of bayes classifier, i know that we are creating a
> job which calls the class to read that input file.
>
> May i know which java file is reading the records from the input csv files ?
>
>
>
>
>
--
Lance Norskog
goksron@gmail.com
Re: Naive-Bayes work flow
Posted by Naveenchandra <na...@gmail.com>.
Hi all,
Can any one explain me how and where the input files are being read, now i am
going through the source code of bayes classifier, i know that we are creating a
job which calls the class to read that input file.
May i know which java file is reading the records from the input csv files ?
Re: Naive-Bayes work flow
Posted by Ramprakash Ramamoorthy <yo...@gmail.com>.
On Wed, Feb 15, 2012 at 5:04 PM, Naveenchandra <na...@gmail.com>wrote:
>
> Dear all,
>
> I am a newbie to mahout, currently working on naive bayes classification.
> I ran the classification for the popular 20 newsgroups example and got
> quite
> good result as gave in the mahout in action book.
>
> But I tried to run the classifier on my own data set having 2 files each
> containing 20k records i am getting a efficiency around 50%, please help
> me.
>
> I would be grateful if you explain me how the actual process is taking
> place
>
> Thanks a ton
>
> -Naveenchandra
>
>
Is it efficiency or accuracy? I mean the 50% you had mentioned...
--
With Thanks and Regards,
Ramprakash Ramamoorthy,
+91 9626975420