You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@opennlp.apache.org by Richard Eckart de Castilho <ri...@gmail.com> on 2016/08/09 13:58:49 UTC
Alternative way of training OpenNLP models
Hi all,
every once in a while somebody here asks about OpenNLP models.
The typical answer then is that there are the models on Sourceforge
but that people should rather train their own. Then often somebody
mentions that support for this-or-that corpus format in OpenNLP
would be cool in order to train on this-or-that dataset. And that
is where it normally ends.
So I thought, why not add the ability to train OpenNLP models to DKPro Core?
DKPro Core is an open-source collection of components for Apache UIMA
integrating many NLP tools including OpenNLP into a uniform toolkit.
DKPro Core already offers readers for many corpus formats.
We have also started adding a dataset API to conveniently access
different standard corpora/datasets that are publicly/freely
available on the net.
And finally, we added support for the OpenNLP training tools for
tokenizer, sentence splitter, POS tagger, chunker, and name finder.
This makes it really easy to train new models for OpenNLP for many
datasets in just a few lines of code, e.g. [1]
DatasetLoader loader = new DatasetLoader(new File("cache"));
Dataset ds = loader.loadEnglishGUMCorpus();
CollectionReaderDescription trainReader = createReaderDescription(
Conll2006Reader.class,
Conll2006Reader.PARAM_PATTERNS, ds.getTrainingFiles(),
Conll2006Reader.PARAM_LANGUAGE, ds.getLanguage());
AnalysisEngineDescription trainer = createEngineDescription(
OpenNlpPosTaggerTrainer.class,
OpenNlpPosTaggerTrainer.PARAM_TARGET_LOCATION, new File(targetFolder, "model.bin"),
OpenNlpPosTaggerTrainer.PARAM_LANGUAGE, ds.getLanguage());
SimplePipeline.runPipeline(trainReader, trainer);
Large parts of DKPro Core are ASL-licensed, but it is not an Apache project.
I hope this will be useful to people.
Happy for any feedback! :)
Cheers,
-- Richard
[1] https://github.com/dkpro/dkpro-core/blob/master/dkpro-core-opennlp-asl/src/test/java/de/tudarmstadt/ukp/dkpro/core/opennlp/OpenNlpPosTaggerTrainerTest.java
Re: Alternative way of training OpenNLP models
Posted by Richard Eckart de Castilho <ri...@gmail.com>.
On 17.08.2016, at 11:24, Joern Kottmann <ko...@gmail.com> wrote:
>
> Hello,
>
> thanks for sharing this, it is very nice to have the ability to train on
> more data sets.
>
> Do you have an overview of the data sets you support in DKPro?
More relevant than the datasets may be the formats supported. A list
of formats supported by the 1.8.0 release can be found here:
https://dkpro.github.io/dkpro-core/releases/1.8.0/docs/format-reference.html
For some of the formats, there are also pointers to datasets.
We also have such a format reference for the 1.9.0-SNAPSHOT version
linked here [1].
There is no "nice" overview of the datasets supported by the loader so far,
but that will eventually be added. You can check out the methods of the
DatasetLoader class [2]. For your convenience, here is a snapshot of the methods
available so far:
loadAncientGreekAndLatingDependencyTreebank()
loadCatalanConll2009()
loadEnglishBrownCorpus()
loadEnglishConll2000()
loadEnglishGUMCorpus()
loadFrenchDeepSequoiaCorpus()
loadGermanConll2009()
loadGermanHamburgDependencyTreebank()
loadGermEval2014NER()
loadJapaneseConll2009()
loadNEMGP()
loadSpanishConll2009()
loadUniversalDependencyTreebankV1_3()
The API of the dataset loader and the information exposed about the
datasets is evolving. Feedback welcome.
Best,
-- Richard
[1] https://dkpro.github.io/dkpro-core/documentation/
[2] https://github.com/dkpro/dkpro-core/blob/master/dkpro-core-datasets-asl/src/main/java/de/tudarmstadt/ukp/dkpro/core/datasets/DatasetLoader.java
Re: Alternative way of training OpenNLP models
Posted by Joern Kottmann <ko...@gmail.com>.
Hello,
thanks for sharing this, it is very nice to have the ability to train on
more data sets.
Do you have an overview of the data sets you support in DKPro?
Jörn
On Tue, Aug 9, 2016 at 3:58 PM, Richard Eckart de Castilho <
richard.eckart@gmail.com> wrote:
> Hi all,
>
> every once in a while somebody here asks about OpenNLP models.
> The typical answer then is that there are the models on Sourceforge
> but that people should rather train their own. Then often somebody
> mentions that support for this-or-that corpus format in OpenNLP
> would be cool in order to train on this-or-that dataset. And that
> is where it normally ends.
>
>
> So I thought, why not add the ability to train OpenNLP models to DKPro
> Core?
>
>
> DKPro Core is an open-source collection of components for Apache UIMA
> integrating many NLP tools including OpenNLP into a uniform toolkit.
>
> DKPro Core already offers readers for many corpus formats.
>
> We have also started adding a dataset API to conveniently access
> different standard corpora/datasets that are publicly/freely
> available on the net.
>
> And finally, we added support for the OpenNLP training tools for
> tokenizer, sentence splitter, POS tagger, chunker, and name finder.
>
> This makes it really easy to train new models for OpenNLP for many
> datasets in just a few lines of code, e.g. [1]
>
>
> DatasetLoader loader = new DatasetLoader(new File("cache"));
> Dataset ds = loader.loadEnglishGUMCorpus();
>
> CollectionReaderDescription trainReader = createReaderDescription(
> Conll2006Reader.class,
> Conll2006Reader.PARAM_PATTERNS, ds.getTrainingFiles(),
> Conll2006Reader.PARAM_LANGUAGE, ds.getLanguage());
>
> AnalysisEngineDescription trainer = createEngineDescription(
> OpenNlpPosTaggerTrainer.class,
> OpenNlpPosTaggerTrainer.PARAM_TARGET_LOCATION, new File(targetFolder,
> "model.bin"),
> OpenNlpPosTaggerTrainer.PARAM_LANGUAGE, ds.getLanguage());
>
> SimplePipeline.runPipeline(trainReader, trainer);
>
>
> Large parts of DKPro Core are ASL-licensed, but it is not an Apache
> project.
>
> I hope this will be useful to people.
>
> Happy for any feedback! :)
>
> Cheers,
>
> -- Richard
>
> [1] https://github.com/dkpro/dkpro-core/blob/master/dkpro-
> core-opennlp-asl/src/test/java/de/tudarmstadt/ukp/dkpro/core/opennlp/
> OpenNlpPosTaggerTrainerTest.java