You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Dean Jones <de...@gmail.com> on 2013/10/09 12:18:54 UTC
Naive bayes and character n-grams
Hello folks,
I see that it's possible to use mahout to train a naive bayes
classifier using n-grams as features (or I guess, strictly speaking,
mahout can be used to generate sequence files containing n-grams; I
suspect the naive bayes trainer is indifferent to the form of features
it trains on). Is there any facility to generate character n-grams
instead of word n-grams?
Thanks,
Dean.
Re: Naive bayes and character n-grams
Posted by Dean Jones <de...@gmail.com>.
Hi Suneel,
On 9 October 2013 14:27, Suneel Marthi <su...@yahoo.com> wrote:
> an example of a Naive-Bayes classifier trained on character n-grams is the LangDetect library.
> (see http://code.google.com/p/language-detection/)
>
> Agree with Ted that it should be relatively easy to build one.
>
Thanks. Yes, I need to (re-)train a language detector. We have an
existing system based on an earlier version of Mahout which I'm
looking to switch to using character n-grams instead of word tokens.
Dean.
Re: Naive bayes and character n-grams
Posted by Suneel Marthi <su...@yahoo.com>.
an example of a Naive-Bayes classifier trained on character n-grams is the LangDetect library.
(see http://code.google.com/p/language-detection/)
Agree with Ted that it should be relatively easy to build one.
On Wednesday, October 9, 2013 6:40 AM, Ted Dunning <te...@gmail.com> wrote:
Yes. Should work to use character n-grams. There are oddities in the
stats because the different n-grams are not independent, but Naive Bayes
methods are in such a state of sin that it shouldn't hurt any worse.
No... I don't think that there is a capability built in to generate the
character n-grams. Should be relatively trivial to build.
On Wed, Oct 9, 2013 at 3:18 AM, Dean Jones <de...@gmail.com> wrote:
> Hello folks,
>
> I see that it's possible to use mahout to train a naive bayes
> classifier using n-grams as features (or I guess, strictly speaking,
> mahout can be used to generate sequence files containing n-grams; I
> suspect the naive bayes trainer is indifferent to the form of features
> it trains on). Is there any facility to generate character n-grams
> instead of word n-grams?
>
> Thanks,
>
> Dean.
>
Re: Naive bayes and character n-grams
Posted by Jens Bonerz <jb...@googlemail.com>.
Hi Dean,
i might be wrong. but try googling for "shingling"... could be something to
start with.
Cheers
Jens
2013/10/9 Ted Dunning <te...@gmail.com>
> Yes. Should work to use character n-grams. There are oddities in the
> stats because the different n-grams are not independent, but Naive Bayes
> methods are in such a state of sin that it shouldn't hurt any worse.
>
> No... I don't think that there is a capability built in to generate the
> character n-grams. Should be relatively trivial to build.
>
>
>
> On Wed, Oct 9, 2013 at 3:18 AM, Dean Jones <de...@gmail.com> wrote:
>
> > Hello folks,
> >
> > I see that it's possible to use mahout to train a naive bayes
> > classifier using n-grams as features (or I guess, strictly speaking,
> > mahout can be used to generate sequence files containing n-grams; I
> > suspect the naive bayes trainer is indifferent to the form of features
> > it trains on). Is there any facility to generate character n-grams
> > instead of word n-grams?
> >
> > Thanks,
> >
> > Dean.
> >
>
<http://www.hightechmg.com>
Re: Naive bayes and character n-grams
Posted by Ted Dunning <te...@gmail.com>.
Yes. Should work to use character n-grams. There are oddities in the
stats because the different n-grams are not independent, but Naive Bayes
methods are in such a state of sin that it shouldn't hurt any worse.
No... I don't think that there is a capability built in to generate the
character n-grams. Should be relatively trivial to build.
On Wed, Oct 9, 2013 at 3:18 AM, Dean Jones <de...@gmail.com> wrote:
> Hello folks,
>
> I see that it's possible to use mahout to train a naive bayes
> classifier using n-grams as features (or I guess, strictly speaking,
> mahout can be used to generate sequence files containing n-grams; I
> suspect the naive bayes trainer is indifferent to the form of features
> it trains on). Is there any facility to generate character n-grams
> instead of word n-grams?
>
> Thanks,
>
> Dean.
>
Re: Naive bayes and character n-grams
Posted by Ted Dunning <te...@gmail.com>.
Cool. Sounds like you are ahead of the game.
Sent from my iPhone
On Oct 10, 2013, at 13:15, Dean Jones <de...@gmail.com> wrote:
> On 10 October 2013 12:46, Ted Dunning <te...@gmail.com> wrote:
>> For language detection, you are going to have a hard time doing better than
>> one of the standard packages for the purpose. See here:
>>
>> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
>
> Thanks for the pointer Ted. I'm a big fan of the Tika project, we use
> it for content extraction already. For various reasons though, we have
> rolled our own language detector (mainly, neither of these packages
> cover all of the languages we need to identify - language-detection
> doesn't do Catalan, Tika doesn't do Welsh).
>
> Dean.
Re: Naive bayes and character n-grams
Posted by Suneel Marthi <su...@yahoo.com>.
Dean,
Just a thought.
You should be able to create new language models (with LangDetect) if there's Wikipedia content for the specific language,
had to do it in the past for Pashto and Malaysian.
On Thursday, October 10, 2013 8:16 AM, Dean Jones <de...@gmail.com> wrote:
On 10 October 2013 12:46, Ted Dunning <te...@gmail.com> wrote:
> For language detection, you are going to have a hard time doing better than
> one of the standard packages for the purpose. See here:
>
> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
>
Thanks for the pointer Ted. I'm a big fan of the Tika project, we use
it for content extraction already. For various reasons though, we have
rolled our own language detector (mainly, neither of these packages
cover all of the languages we need to identify - language-detection
doesn't do Catalan, Tika doesn't do Welsh).
Dean.
Re: Naive bayes and character n-grams
Posted by Dean Jones <de...@gmail.com>.
On 10 October 2013 12:46, Ted Dunning <te...@gmail.com> wrote:
> For language detection, you are going to have a hard time doing better than
> one of the standard packages for the purpose. See here:
>
> http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
>
Thanks for the pointer Ted. I'm a big fan of the Tika project, we use
it for content extraction already. For various reasons though, we have
rolled our own language detector (mainly, neither of these packages
cover all of the languages we need to identify - language-detection
doesn't do Catalan, Tika doesn't do Welsh).
Dean.
Re: Naive bayes and character n-grams
Posted by Ted Dunning <te...@gmail.com>.
For language detection, you are going to have a hard time doing better than
one of the standard packages for the purpose. See here:
http://blog.mikemccandless.com/2011/10/accuracy-and-performance-of-googles.html
On Thu, Oct 10, 2013 at 1:01 AM, Dean Jones <de...@gmail.com> wrote:
> Hi Si,
>
> On 10 October 2013 07:59, <si...@bt.com> wrote:
> >
> > what do you mean by character n-grams? If you mean things like "&ab" or
> "ui2" then given that there are so few characters compared to words is
> there a problem that can't be solved without a look-up table for n<y (where
> y <4ish )
> >
> > Or are you looking at y >4 ish because if so then do you run into the
> issue of a sudden space explosion?
> >
>
> Yes, just tokens in a text broken up into sequences of their constituent
> characters. In my initial tests, language detection works well where n=3,
> particularly when including the head and tail bigrams. So I need something
> to generate the required sequence files from my training data.
>
Re: Naive bayes and character n-grams
Posted by Dean Jones <de...@gmail.com>.
Hi Si,
On 10 October 2013 07:59, <si...@bt.com> wrote:
>
> what do you mean by character n-grams? If you mean things like "&ab" or
"ui2" then given that there are so few characters compared to words is
there a problem that can't be solved without a look-up table for n<y (where
y <4ish )
>
> Or are you looking at y >4 ish because if so then do you run into the
issue of a sudden space explosion?
>
Yes, just tokens in a text broken up into sequences of their constituent
characters. In my initial tests, language detection works well where n=3,
particularly when including the head and tail bigrams. So I need something
to generate the required sequence files from my training data.
RE: Naive bayes and character n-grams
Posted by si...@bt.com.
Hey Dean,
what do you mean by character n-grams? If you mean things like "&ab" or "ui2" then given that there are so few characters compared to words is there a problem that can't be solved without a look-up table for n<y (where y <4ish )
Or are you looking at y >4 ish because if so then do you run into the issue of a sudden space explosion?
Best
Simon
----
Dr. Simon Thompson
________________________________________
From: Dean Jones [dean.m.jones@gmail.com]
Sent: 09 October 2013 11:18
To: user@mahout.apache.org
Subject: Naive bayes and character n-grams
Hello folks,
I see that it's possible to use mahout to train a naive bayes
classifier using n-grams as features (or I guess, strictly speaking,
mahout can be used to generate sequence files containing n-grams; I
suspect the naive bayes trainer is indifferent to the form of features
it trains on). Is there any facility to generate character n-grams
instead of word n-grams?
Thanks,
Dean.