You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by jo...@apache.org on 2011/11/30 11:17:07 UTC
svn commit: r1208363 -
/incubator/opennlp/trunk/opennlp-docs/src/docbkx/chunker.xml
Author: joern
Date: Wed Nov 30 10:17:07 2011
New Revision: 1208363
URL: http://svn.apache.org/viewvc?rev=1208363&view=rev
Log:
OPENNLP-218 Added API section. Fixed a couple of typos. Replaced programmlisting with screen when for the cases where it does not contain an actual programm listing. Some updated for OPENNLP-402 for screen listings. Thanks to Aliaksandr Autayeu for providing a patch.
Modified:
incubator/opennlp/trunk/opennlp-docs/src/docbkx/chunker.xml
Modified: incubator/opennlp/trunk/opennlp-docs/src/docbkx/chunker.xml
URL: http://svn.apache.org/viewvc/incubator/opennlp/trunk/opennlp-docs/src/docbkx/chunker.xml?rev=1208363&r1=1208362&r2=1208363&view=diff
==============================================================================
--- incubator/opennlp/trunk/opennlp-docs/src/docbkx/chunker.xml (original)
+++ incubator/opennlp/trunk/opennlp-docs/src/docbkx/chunker.xml Wed Nov 30 10:17:07 2011
@@ -42,22 +42,22 @@ under the License.
Download the english maxent chunker model from the website and start the Chunker Tool with this command:
</para>
<para>
- <programlisting>
+ <screen>
<![CDATA[
-bin/opennlp ChunkerME en-chunker.bin]]>
- </programlisting>
+$ opennlp ChunkerME en-chunker.bin]]>
+ </screen>
The Chunker now reads a pos tagged sentence per line from stdin.
Copy these two sentences to the console:
- <programlisting>
+ <screen>
<![CDATA[
Rockwell_NNP International_NNP Corp._NNP 's_POS Tulsa_NNP unit_NN said_VBD it_PRP signed_VBD
a_DT tentative_JJ agreement_NN extending_VBG its_PRP$ contract_NN with_IN Boeing_NNP Co._NNP
to_TO provide_VB structural_JJ parts_NNS for_IN Boeing_NNP 's_POS 747_CD jetliners_NNS ._.
Rockwell_NNP said_VBD the_DT agreement_NN calls_VBZ for_IN it_PRP to_TO supply_VB 200_CD
additional_JJ so-called_JJ shipsets_NNS for_IN the_DT planes_NNS ._.]]>
- </programlisting>
- the Chunker will now echo the sentences grouped tokens to the console:
- <programlisting>
+ </screen>
+ The Chunker will now echo the sentences grouped tokens to the console:
+ <screen>
<![CDATA[
[NP Rockwell_NNP International_NNP Corp._NNP ] [NP 's_POS Tulsa_NNP unit_NN ] [VP said_VBD ]
[NP it_PRP ] [VP signed_VBD ] [NP a_DT tentative_JJ agreement_NN ] [VP extending_VBG ]
@@ -66,9 +66,8 @@ Rockwell_NNP said_VBD the_DT agreement_N
[NP Rockwell_NNP ] [VP said_VBD ] [NP the_DT agreement_NN ] [VP calls_VBZ ] [SBAR for_IN ]
[NP it_PRP ] [VP to_TO supply_VB ] [NP 200_CD additional_JJ so-called_JJ shipsets_NNS ]
[PP for_IN ] [NP the_DT planes_NNS ] ._.]]>
- </programlisting>
- The tag set used by the english pos model is the Penn Treebank tag set.
- See the link below for a description of the tags.
+ </screen>
+ The tag set used by the english pos model is the <ulink url="http://www.cis.upenn.edu/~treebank/">Penn Treebank tag set</ulink>.
</para>
</section>
<section id="tools.parser.chunking.api">
@@ -161,8 +160,9 @@ Sequence topSequences[] = chunk.topKSequ
corpus or on a corpus which is extended by private training data taken from the data which should be analyzed.
</para>
<para>
- The training data must be converted to the OpenNLP chunker training format,
- that is based on <ulink url="http://www.cnts.ua.ac.be/conll2000/chunking">CoNLL2000</ulink>:
+ The training data can be converted to the OpenNLP chunker training format,
+ that is based on <ulink url="http://www.cnts.ua.ac.be/conll2000/chunking">CoNLL2000</ulink>.
+ Other formats may also be available.
The train data consist of three columns separated by spaces. Each word has been put on a
separate line and there is an empty line after each sentence. The first column contains
the current word, the second its part-of-speech tag and the third its chunk tag.
@@ -173,7 +173,7 @@ Sequence topSequences[] = chunk.topKSequ
</para>
<para>
Sample sentence of the training data:
- <programlisting>
+ <screen>
<![CDATA[
He PRP B-NP
reckons VBZ B-VP
@@ -191,7 +191,7 @@ billion CD I-NP
in IN B-PP
September NNP B-NP
. . O]]>
- </programlisting>
+ </screen>
</para>
<section id="tools.chunker.training.tool">
<title>Training Tool</title>
@@ -200,34 +200,72 @@ September NNP B-NP
model download page on various corpora.
</para>
<para>
- Usage of the tool:
- <programlisting>
+ Usage of the tool:
+ <screen>
<![CDATA[
-$ bin/opennlp ChunkerTrainerME
-Usage: opennlp ChunkerTrainerME-lang language -encoding charset [-iterations num] \
-[-cutoff num] -data trainingData -model model
--lang language specifies the language which is being processed.
--encoding charset specifies the encoding which should be used for reading and writing text.
--iterations num specified the number of training iterations
--cutoff num specifies the min number of times a feature must be seen]]>
- </programlisting>
+$ opennlp ChunkerTrainerME
+Usage: opennlp ChunkerTrainerME[.ad] [-params paramsFile] [-iterations num] [-cutoff num] \
+ -model modelFile -lang language -data sampleData [-encoding charsetName]
+
+Arguments description:
+ -params paramsFile
+ training parameters file.
+ -iterations num
+ number of training iterations, ignored if -params is used.
+ -cutoff num
+ minimal number of times a feature must be seen, ignored if -params is used.
+ -model modelFile
+ output model file.
+ -lang language
+ language which is being processed.
+ -data sampleData
+ data to be used, usually a file name.
+ -encoding charsetName
+ encoding for reading and writing text, if absent the system default is used.]]>
+ </screen>
Its now assumed that the english chunker model should be trained from a file called
en-chunker.train which is encoded as UTF-8. The following command will train the
name finder and write the model to en-chunker.bin:
- <programlisting>
+ <screen>
<![CDATA[
-bin/opennlp ChunkerTrainerME -encoding UTF-8 -lang en -data en-chunker.train -model en-chunker.bin]]>
- </programlisting>
+$ opennlp ChunkerTrainerME -model en-chunker.bin -lang en -data en-chunker.train -encoding UTF-8]]>
+ </screen>
Additionally its possible to specify the number of iterations, the cutoff and to overwrite
all types in the training data with a single type.
</para>
</section>
- <section id="tools.chunker.training.api">
+ <section id="tools.chunker.training.api">
<title>Training API</title>
- <para>TODO: Write documentation about the chunker training api. Any contributions
- are very welcome. If you want to contribute please contact us on the mailing list
- or comment on the jira issue <ulink url="https://issues.apache.org/jira/browse/OPENNLP-218">OPENNLP-218</ulink>.
- </para>
+ <para>
+ The Chunker offers an API to train a new chunker model. The following sample code
+ illustrates how to do it:
+ <programlisting language="java">
+ <![CDATA[
+Charset charset = Charset.forName("UTF-8");
+ObjectStream<String> lineStream =
+ new PlainTextByLineStream(new FileInputStream("en-chunker.train"),charset);
+ObjectStream<ChunkSample> sampleStream = new ChunkSampleStream(lineStream);
+
+ChunkerModel model;
+
+try {
+ model = ChunkerME.train("en", sampleStream,
+ new DefaultChunkerContextGenerator(), TrainingParameters.defaultParams());
+}
+finally {
+ sampleStream.close();
+}
+
+OutputStream modelOut = null;
+try {
+ modelOut = new BufferedOutputStream(new FileOutputStream(modelFile));
+ model.serialize(modelOut);
+} finally {
+ if (modelOut != null)
+ modelOut.close();
+}]]>
+ </programlisting>
+ </para>
</section>
</section>
@@ -240,42 +278,60 @@ bin/opennlp ChunkerTrainerME -encoding U
<section id="tools.chunker.evaluation.tool">
<title>Chunker Evaluation Tool</title>
<para>
- The following command shows how the tool can be run:
- <programlisting>
+ The following command shows how the tool can be run:
+ <screen>
<![CDATA[
-bin/opennlp ChunkerEvaluator
-Usage: opennlp ChunkerEvaluator [-encoding charsetName] -data data -model model]]>
- </programlisting>
+$ opennlp ChunkerEvaluator
+Usage: opennlp ChunkerEvaluator[.ad] -model model [-misclassified true|false] \
+ [-detailedF true|false] -lang language -data sampleData [-encoding charsetName]]]>
+ </screen>
A sample of the command considering you have a data sample named en-chunker.eval
- and you trainned a model called en-chunker.bin:
- <programlisting>
+ and you trained a model called en-chunker.bin:
+ <screen>
<![CDATA[
-bin/opennlp ChunkerEvaluator -lang en -encoding UTF-8 -data en-chunker.eval -model en-chunker.bin]]>
- </programlisting>
+$ opennlp ChunkerEvaluator -model en-chunker.bin -lang en -data en-chunker.eval -encoding UTF-8]]>
+ </screen>
and here is a sample output:
- <programlisting>
+ <screen>
<![CDATA[
Precision: 0.9255923572240226
Recall: 0.9220610430991112
F-Measure: 0.9238233255623465]]>
- </programlisting>
+ </screen>
You can also use the tool to perform 10-fold cross validation of the Chunker.
he following command shows how the tool can be run:
- <programlisting>
+ <screen>
<![CDATA[
-bin/opennlp ChunkerCrossValidator
-Usage: opennlp ChunkerCrossValidator -lang language -encoding charset [-iterations num] [-cutoff num]
--lang language specifies the language which is being processed.
--encoding charset specifies the encoding which should be used for reading and writing text.
--iterations num specified the number of training iterations
--cutoff num specifies the min number of times a feature must be seen
--data trainingData training data used for cross validation]]>
- </programlisting>
+$ opennlp ChunkerCrossValidator
+Usage: opennlp ChunkerCrossValidator[.ad] [-params paramsFile] [-iterations num] [-cutoff num] \
+ [-misclassified true|false] [-folds num] [-detailedF true|false] \
+ -lang language -data sampleData [-encoding charsetName]
+
+Arguments description:
+ -params paramsFile
+ training parameters file.
+ -iterations num
+ number of training iterations, ignored if -params is used.
+ -cutoff num
+ minimal number of times a feature must be seen, ignored if -params is used.
+ -misclassified true|false
+ if true will print false negatives and false positives.
+ -folds num
+ number of folds, default is 10.
+ -detailedF true|false
+ if true will print detailed FMeasure results.
+ -lang language
+ language which is being processed.
+ -data sampleData
+ data to be used, usually a file name.
+ -encoding charsetName
+ encoding for reading and writing text, if absent the system default is used.]]>
+ </screen>
It is not necessary to pass a model. The tool will automatically split the data to train and evaluate:
- <programlisting>
- <![CDATA[
-bin/opennlp ChunkerCrossValidator -lang pt -encoding UTF-8 -data en-chunker.cross]]>
- </programlisting>
+ <screen>
+ <![CDATA[
+$ opennlp ChunkerCrossValidator -lang pt -data en-chunker.cross -encoding UTF-8]]>
+ </screen>
</para>
</section>
</section>