You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Alex Kudlick <ad...@gmail.com> on 2012/04/23 19:41:12 UTC
Document Classification
Hi,
I've just started using open nlp for a project to classify scientific
articles in to subjects. I have a few questions:
1. How do I configure logging for the model? I'm using sf4j-log4j for the
rest of my application, but the training output from the model just goes to
stdout.
2. Is there any support for classifying documents with multiple classes?
For instance, a given article may be classified as Computational Biology,
Cell Biology, and Molecular Biology.
Thanks,
Alex Kudlick
Re: Document Classification
Posted by Jörn Kottmann <ko...@gmail.com>.
On 04/26/2012 03:37 AM, Lance Norskog wrote:
> Cool! Yeah, Tika has one also.
>
> Now for the annoying use case: older web sites and pre-web text in
> Southeast Asia and India/Pakistan are written in phonetic USASCII.
> (They only had that technology available. Does anybody do
> classification on that kind of text?
>
I never did. Its only doing bag-of-word feature generation,
to make that work you need to tokenize your input text.
We have a learn-able tokenizer (must be trained), character-class
and whitespace tokenizer.
Jörn
Re: Document Classification
Posted by Lance Norskog <go...@gmail.com>.
Cool! Yeah, Tika has one also.
Now for the annoying use case: older web sites and pre-web text in
Southeast Asia and India/Pakistan are written in phonetic USASCII.
(They only had that technology available. Does anybody do
classification on that kind of text?
On Tue, Apr 24, 2012 at 7:17 AM, Jason Baldridge
<ja...@gmail.com> wrote:
> Naive Bayes, perceptron variants (incl passive agressive), faster training
> for maxent, and a better overall architecture. These are things my students
> and I are working on independently, and I will bring in to OpenNLP when
> time frees up to do so.
>
> On Tue, Apr 24, 2012 at 2:26 AM, Jörn Kottmann <ko...@gmail.com> wrote:
>
>> What are you planning to add?
>>
>> Jörn
>>
>>
>> On 04/24/2012 03:53 AM, Jason Baldridge wrote:
>>
>>> FWIW, there will be more classification capabilities coming in the next
>>> several months.
>>>
>>> -Jason
>>>
>>> On Mon, Apr 23, 2012 at 5:12 PM, Jörn Kottmann<ko...@gmail.com>
>>> wrote:
>>>
>>> OpenNLP is using either a Maxent or Perceptron classifier
>>>> to classify a piece of text. This can give you back the provabilities
>>>> for the various categories, but its not designed to tell you how
>>>> much each topic is represented in your input document.
>>>>
>>>> You could take a document and assume each paragraph has one topic
>>>> and then classify it paragraph by paragraph.
>>>> We sadly don't have support for topic models, such as LDA.
>>>>
>>>> All the training logs are still written to the console, we have plans
>>>> to properly capture them and report training process back via an
>>>> API. This output should then be logged and maybe just stored in inside
>>>> the model for later debugging.
>>>>
>>>> Jörn
>>>>
>>>>
>>>> On 04/23/2012 07:41 PM, Alex Kudlick wrote:
>>>>
>>>> Hi,
>>>>>
>>>>> I've just started using open nlp for a project to classify scientific
>>>>> articles in to subjects. I have a few questions:
>>>>>
>>>>> 1. How do I configure logging for the model? I'm using sf4j-log4j for
>>>>> the
>>>>> rest of my application, but the training output from the model just goes
>>>>> to
>>>>> stdout.
>>>>>
>>>>> 2. Is there any support for classifying documents with multiple classes?
>>>>> For instance, a given article may be classified as Computational
>>>>> Biology,
>>>>> Cell Biology, and Molecular Biology.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Alex Kudlick
>>>>>
>>>>>
>>>>>
>>>
>>
>
>
> --
> Jason Baldridge
> Associate Professor, Department of Linguistics
> The University of Texas at Austin
> http://www.jasonbaldridge.com
> http://twitter.com/jasonbaldridge
--
Lance Norskog
goksron@gmail.com
Re: Document Classification
Posted by Jason Baldridge <ja...@gmail.com>.
Naive Bayes, perceptron variants (incl passive agressive), faster training
for maxent, and a better overall architecture. These are things my students
and I are working on independently, and I will bring in to OpenNLP when
time frees up to do so.
On Tue, Apr 24, 2012 at 2:26 AM, Jörn Kottmann <ko...@gmail.com> wrote:
> What are you planning to add?
>
> Jörn
>
>
> On 04/24/2012 03:53 AM, Jason Baldridge wrote:
>
>> FWIW, there will be more classification capabilities coming in the next
>> several months.
>>
>> -Jason
>>
>> On Mon, Apr 23, 2012 at 5:12 PM, Jörn Kottmann<ko...@gmail.com>
>> wrote:
>>
>> OpenNLP is using either a Maxent or Perceptron classifier
>>> to classify a piece of text. This can give you back the provabilities
>>> for the various categories, but its not designed to tell you how
>>> much each topic is represented in your input document.
>>>
>>> You could take a document and assume each paragraph has one topic
>>> and then classify it paragraph by paragraph.
>>> We sadly don't have support for topic models, such as LDA.
>>>
>>> All the training logs are still written to the console, we have plans
>>> to properly capture them and report training process back via an
>>> API. This output should then be logged and maybe just stored in inside
>>> the model for later debugging.
>>>
>>> Jörn
>>>
>>>
>>> On 04/23/2012 07:41 PM, Alex Kudlick wrote:
>>>
>>> Hi,
>>>>
>>>> I've just started using open nlp for a project to classify scientific
>>>> articles in to subjects. I have a few questions:
>>>>
>>>> 1. How do I configure logging for the model? I'm using sf4j-log4j for
>>>> the
>>>> rest of my application, but the training output from the model just goes
>>>> to
>>>> stdout.
>>>>
>>>> 2. Is there any support for classifying documents with multiple classes?
>>>> For instance, a given article may be classified as Computational
>>>> Biology,
>>>> Cell Biology, and Molecular Biology.
>>>>
>>>> Thanks,
>>>>
>>>> Alex Kudlick
>>>>
>>>>
>>>>
>>
>
--
Jason Baldridge
Associate Professor, Department of Linguistics
The University of Texas at Austin
http://www.jasonbaldridge.com
http://twitter.com/jasonbaldridge
Re: Document Classification
Posted by Jörn Kottmann <ko...@gmail.com>.
What are you planning to add?
Jörn
On 04/24/2012 03:53 AM, Jason Baldridge wrote:
> FWIW, there will be more classification capabilities coming in the next
> several months.
>
> -Jason
>
> On Mon, Apr 23, 2012 at 5:12 PM, Jörn Kottmann<ko...@gmail.com> wrote:
>
>> OpenNLP is using either a Maxent or Perceptron classifier
>> to classify a piece of text. This can give you back the provabilities
>> for the various categories, but its not designed to tell you how
>> much each topic is represented in your input document.
>>
>> You could take a document and assume each paragraph has one topic
>> and then classify it paragraph by paragraph.
>> We sadly don't have support for topic models, such as LDA.
>>
>> All the training logs are still written to the console, we have plans
>> to properly capture them and report training process back via an
>> API. This output should then be logged and maybe just stored in inside
>> the model for later debugging.
>>
>> Jörn
>>
>>
>> On 04/23/2012 07:41 PM, Alex Kudlick wrote:
>>
>>> Hi,
>>>
>>> I've just started using open nlp for a project to classify scientific
>>> articles in to subjects. I have a few questions:
>>>
>>> 1. How do I configure logging for the model? I'm using sf4j-log4j for the
>>> rest of my application, but the training output from the model just goes
>>> to
>>> stdout.
>>>
>>> 2. Is there any support for classifying documents with multiple classes?
>>> For instance, a given article may be classified as Computational Biology,
>>> Cell Biology, and Molecular Biology.
>>>
>>> Thanks,
>>>
>>> Alex Kudlick
>>>
>>>
>
Re: Document Classification
Posted by Jason Baldridge <ja...@gmail.com>.
FWIW, there will be more classification capabilities coming in the next
several months.
-Jason
On Mon, Apr 23, 2012 at 5:12 PM, Jörn Kottmann <ko...@gmail.com> wrote:
> OpenNLP is using either a Maxent or Perceptron classifier
> to classify a piece of text. This can give you back the provabilities
> for the various categories, but its not designed to tell you how
> much each topic is represented in your input document.
>
> You could take a document and assume each paragraph has one topic
> and then classify it paragraph by paragraph.
> We sadly don't have support for topic models, such as LDA.
>
> All the training logs are still written to the console, we have plans
> to properly capture them and report training process back via an
> API. This output should then be logged and maybe just stored in inside
> the model for later debugging.
>
> Jörn
>
>
> On 04/23/2012 07:41 PM, Alex Kudlick wrote:
>
>> Hi,
>>
>> I've just started using open nlp for a project to classify scientific
>> articles in to subjects. I have a few questions:
>>
>> 1. How do I configure logging for the model? I'm using sf4j-log4j for the
>> rest of my application, but the training output from the model just goes
>> to
>> stdout.
>>
>> 2. Is there any support for classifying documents with multiple classes?
>> For instance, a given article may be classified as Computational Biology,
>> Cell Biology, and Molecular Biology.
>>
>> Thanks,
>>
>> Alex Kudlick
>>
>>
>
--
Jason Baldridge
Associate Professor, Department of Linguistics
The University of Texas at Austin
http://www.jasonbaldridge.com
http://twitter.com/jasonbaldridge
Re: Document Classification
Posted by Jörn Kottmann <ko...@gmail.com>.
OpenNLP is using either a Maxent or Perceptron classifier
to classify a piece of text. This can give you back the provabilities
for the various categories, but its not designed to tell you how
much each topic is represented in your input document.
You could take a document and assume each paragraph has one topic
and then classify it paragraph by paragraph.
We sadly don't have support for topic models, such as LDA.
All the training logs are still written to the console, we have plans
to properly capture them and report training process back via an
API. This output should then be logged and maybe just stored in inside
the model for later debugging.
Jörn
On 04/23/2012 07:41 PM, Alex Kudlick wrote:
> Hi,
>
> I've just started using open nlp for a project to classify scientific
> articles in to subjects. I have a few questions:
>
> 1. How do I configure logging for the model? I'm using sf4j-log4j for the
> rest of my application, but the training output from the model just goes to
> stdout.
>
> 2. Is there any support for classifying documents with multiple classes?
> For instance, a given article may be classified as Computational Biology,
> Cell Biology, and Molecular Biology.
>
> Thanks,
>
> Alex Kudlick
>
Re: Re: Document Classification
Posted by ad...@gmail.com.
Sure, but OpenNLP comes out of the box with a document classifier. Is that
no longer being supported?
On , Lance Norskog <go...@gmail.com> wrote:
> Classification is more of a machine learning problem, not a
> linguistics problem. There are a few large, sophisticated (complex)
> implementations for text analysis: Weka, R, and Mahout.
> On Mon, Apr 23, 2012 at 10:41 AM, Alex Kudlick adkudlick@gmail.com> wrote:
> > Hi,
> >
> > I've just started using open nlp for a project to classify scientific
> > articles in to subjects. I have a few questions:
> >
> > 1. How do I configure logging for the model? I'm using sf4j-log4j for
> the
> > rest of my application, but the training output from the model just
> goes to
> > stdout.
> >
> > 2. Is there any support for classifying documents with multiple classes?
> > For instance, a given article may be classified as Computational
> Biology,
> > Cell Biology, and Molecular Biology.
> >
> > Thanks,
> >
> > Alex Kudlick
> --
> Lance Norskog
> goksron@gmail.com
Re: Document Classification
Posted by Lance Norskog <go...@gmail.com>.
Classification is more of a machine learning problem, not a
linguistics problem. There are a few large, sophisticated (complex)
implementations for text analysis: Weka, R, and Mahout.
On Mon, Apr 23, 2012 at 10:41 AM, Alex Kudlick <ad...@gmail.com> wrote:
> Hi,
>
> I've just started using open nlp for a project to classify scientific
> articles in to subjects. I have a few questions:
>
> 1. How do I configure logging for the model? I'm using sf4j-log4j for the
> rest of my application, but the training output from the model just goes to
> stdout.
>
> 2. Is there any support for classifying documents with multiple classes?
> For instance, a given article may be classified as Computational Biology,
> Cell Biology, and Molecular Biology.
>
> Thanks,
>
> Alex Kudlick
--
Lance Norskog
goksron@gmail.com