svn commit: r1050126 [1/2] - /incubator/opennlp/tags/asf_migration/opennlp-docs/

[jo...@apache.org]

Author: joern
Date: Thu Dec 16 19:50:51 2010
New Revision: 1050126

URL: http://svn.apache.org/viewvc?rev=1050126&view=rev
Log:
OPENNLP-3 Imported wiki content from SourceForge

Added:
    incubator/opennlp/tags/asf_migration/opennlp-docs/Arvores_Deitadas.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Conll03.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Conll06.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Detokenizer.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Main_Page.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Name_Finder.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/OpenNLP_Installation.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/POS_Tagger.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Parser.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Postagger.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Sentence_Detector.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Tokenization.wiki
    incubator/opennlp/tags/asf_migration/opennlp-docs/Tokenizer.wiki

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Arvores_Deitadas.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Arvores_Deitadas.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Arvores_Deitadas.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Arvores_Deitadas.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,42 @@
+This page describes how to convert the Arvores Deitadas corpus
+into training data for the [[Name Finder]].
+
+= The Corpus =
+Floresta Sintá(c)tica (syntactic forest) is a publicly available treebank for Portuguese, created as a collaboration project between the [http://visl.sdu.dk VISL project], and [http://www.linguateca.pt Linguateca] (formerly the Computational Processing of Portuguese project). (from the project [http://www.linguateca.pt/floresta/info_floresta_English.html website])
+
+The Floresta Sintá(c)tica provides corpora in a variety of formats. For now, the OpenNLP only supports the Árvores Deitadas (AD) (lying trees). The documentation of AD format is available [http://www.linguateca.pt/documentos/Afonso2006ArvoresDeitadas.pdf here] (Portuguese only).
+
+To generate the NER models, we are using the Amazônia (Amazon) corpus, which contains 4.6 million words (about 275,000 sentences) from the Web collaborative Overmundo, a virtual collective that aims to express the Brazilian cultural production. Because it is collaborative, the site includes a large number of authors, from different parts of Brazil, which is also reflected in different writing styles. The text where extracted from "Overblog" and all non-fiction texts of the "Banco de Cultura" (culture bank) available on September 30, 2008, a total of 4070 texts (1303 and authors). Unlike other corpora of the Forest, the Amazon is not a balanced corpus from the Portuguese of Brazil and Portugal: all texts are Brazilians. (from the project [http://www.linguateca.pt/floresta/corpus.html website])
+
+= Getting the data =
+The Corpus can be downloaded from here:
+http://www.linguateca.pt/floresta/corpus.html
+
+The direct link to the corpus file:
+http://www.linguateca.pt/floresta/ficheiros/gz/amazonia.ad.gz
+
+= Converting the data =
+For now only the Token Name Finder is available:
+<pre>
+bin/opennlp TokenNameFinderConverter ad -encoding ISO-8859-1 -data ../corpus/amazonia.ad -lang pt -types per > corpus.txt
+</pre>
+
+= Evaluation = 
+
+To perform the evaluation the corpus was split into a training and a test part.
+
+<pre>
+sed '1,55172d' corpus.txt > corpus_train.txt
+sed '55172,100000000d' corpus.txt > corpus_test.txt
+</pre>
+
+<pre>
+
+bin/opennlp TokenNameFinderTrainer -lang PT -encoding UTF-8 -data corpus_train.txt -model pt-ner.bin -cutoff 20
+..
+bin/opennlp TokenNameFinderEvaluator -encoding UTF-8 -model ../model/pt-ner.bin -data corpus_test.txt
+
+Precision: 0.8005071889818507
+Recall: 0.7450581122145297
+F-Measure: 0.7717879983140168
+</pre>

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Conll03.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Conll03.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Conll03.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Conll03.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,88 @@
+This page describes how to create training data out of the Conll03 data.
+
+= CoNLL =
+== What is CoNLL? ==
+CoNLL stands for the Confernece on Computational Natural Language Learning and is not a single project but a consordium of developers attempting to broaden the computing environment.
+More information about the entire conference series can be obtained here for [http://www.cnts.ua.ac.be/conll2003/ CoNLL].
+
+== Data Collection ==
+The CoNLL group has a set of data available that provide the required tools and scripts to generate the training data for the project from the original sources.  This is required; because the sources for the training sets are copyrighted by the respected copyright holders.
+To better understand the data format and the required news articles at [http://www.cnts.ua.ac.be/conll2003/ner/ NER] (Name Entity Recognision) for details.
+<p>English corpus uses the Reuters Corpus released 2000-11-03 Volume I containing articles from 1996-08-20 to 1997-08-19 in the English language.</p>
+<p>German corpus uses (add additional information on this corpus)... </p>
+
+== Creating the Data ==
+Please follow the steps in the [http://www.cnts.ua.ac.be/conll2003/ner/000README README] for the CoNLL-03 data.  You may omit English or German if you do not have the original data for the set.  ''NOTE: German is not yet supported by OpenNLP converter [verification needed].'' 
+<p>Your task is not yet done.  This will generate data for the format for CoNLL-03; however, OpenNLP cannot directly use this data to train or validate the models.
+
+= Using The Collected Data =
+== OpenNLP Name Finder Converter ==
+To convert the information to the OpenNLP format:
+<pre>
+bin/opennlp TokenNameFinderConverter conll03 -data eng.train -lang en -types per > corpus_train.txt
+</pre>
+Optionally, you can convert the training test samples as well.
+<pre>
+bin/opennlp TokenNameFinderConverter conll03 -data eng.testa -lang en -types per > corpus_testa.txt
+bin/opennlp TokenNameFinderConverter conll03 -data eng.testb -lang en -types per > corpus_testb.txt
+</pre>
+
+== Name Finder Training ==
+To train the model for the name finder,
+<pre>
+bin/opennlp TokenNameFinderTrainer -lang en -encoding utf8 -iterations 500 -data corpus_train.txt -model en_ner_person.bin
+</pre>
+<pre>
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 203621 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 203621 events to 179409.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 179409
+	    Number of Outcomes: 3
+	  Number of Predicates: 58814
+...done.
+Computing model parameters...
+Performing 500 iterations.
+  1:  .. loglikelihood=-223700.5328318588	0.9453494482396216
+  2:  .. loglikelihood=-40525.939777363084	0.9467933071736215
+  3:  .. loglikelihood=-24893.98837874921	0.9598518816821447
+  4:  .. loglikelihood=-18420.3379471033	0.9712996203731442
+... cut lots of iterations ...
+498:  .. loglikelihood=-952.8501399442295	0.9988950059178572
+499:  .. loglikelihood=-952.0600155746948	0.9988950059178572
+500:  .. loglikelihood=-951.2722802086295	0.9988950059178572
+Writing name finder model ... done (1.638s)
+
+Wrote name finder model to
+path: .\en_ner_person.bin
+</pre>
+
+== Name Finder Evaluation ==
+Since we created the test A and B files above, we can use them to evaluate the model.
+<pre>
+bin/opennlp TokenNameFinderEvaluator -lang en -encoding utf8 -model en_ner_person.bin -data corpus_testa.txt
+</pre>
+<pre>
+Loading Token Name Finder model ... done (0.359s)
+current: 190.2 sent/s avg: 190.2 sent/s total: 199 sent
+current: 648.3 sent/s avg: 415.9 sent/s total: 850 sent
+current: 530.1 sent/s avg: 453.6 sent/s total: 1380 sent
+current: 793.8 sent/s avg: 539.0 sent/s total: 2178 sent
+current: 705.4 sent/s avg: 571.9 sent/s total: 2882 sent
+
+
+Average: 569.4 sent/s
+Total: 3251 sent
+Runtime: 5.71s
+
+Precision: 0.9366247297154147
+Recall: 0.739956568946797
+F-Measure: 0.8267557582133971
+</pre>
+
+= Important Notes =
+1) The scripts for CoNLL DO NOT work correctly in CYGWIN and will error out when converting the HTML/XML code into plain text.

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Conll06.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Conll06.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Conll06.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Conll06.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,519 @@
+This page describes how to create training data out of the Conll06 data.
+
+= Overview =
+
+== Goal of tutorial ==
+
+The goal of this tutorial is to explain how to create models using OpenNLP for sentence detection, tokenization, and POS (part of speech) tagging for the following languages:
+* Danish
+* Dutch
+* Portuguese
+* Swedish
+
+For demonstration purposes, this tutorial will use Portuguese as the example language.
+
+== Creating models ==
+
+The basic steps for creating each OpenNLP model will be as follows:
+# Download and extract the data for the languages.
+# Convert the data to the OpenNLP format, which will provide us with a training file and testing file for each language.
+# Train a model using the training file.
+# Evaluate the model using the testing file.
+# Tag raw text with the model.
+
+= Getting OpenNLP =
+
+This tutorial was created for the OpenNLP Toolkit v1.5.0. See the instructions on the [[OpenNLP_Installation|installation]] page for help getting OpenNLP installed.
+
+= Language Data =
+
+== Download data ==
+
+The training and testing data for these languages can be downloaded from the following website: [http://nextens.uvt.nl/~conll/free_data.html]
+
+The following files will need to be downloaded and extracted to some location:
+* conll06_data_danish_ddt_train_v1.1.tar.bz2
+* conll06_data_dutch_alpino_train_v1.4.tar.bz2
+* conll06_data_portuguese_bosque_train_v1.2.tar.bz2
+* conll06_data_swedish_talbanken05_train_v1.1.tar.bz2
+* conll06_data_free_test.tar.bz2
+
+== Extract data ==
+
+Under Linux, the following commands can be used for each file (e.g. for the file "conll06_data_danish_ddt_train_v1.1.tar.bz2"):
+<pre>
+$ bunzip2 conll06_data_danish_ddt_train_v1.1.tar.bz2
+$ tar -xf conll06_data_danish_ddt_train_v1.1.tar
+</pre>
+
+Alternatively, you can use the following command to extract all the files at once:
+<pre>
+$ bunzip2 *.bz2 && for i in *.tar; do tar -xf $i; done
+</pre>
+
+This will create the following directories and files, which will be used as the data for training and testing new models:
+<pre>
+./data/danish/ddt/test/danish_ddt_test.conll
+./data/danish/ddt/train/danish_ddt_train.conll
+./data/dutch/alpino/test/dutch_alpino_test.conll
+./data/dutch/alpino/train/dutch_alpino_train.conll
+./data/portuguese/bosque/test/portuguese_bosque_test.conll
+./data/portuguese/bosque/treebank/portuguese_bosque_train.conll
+./data/swedish/talbanken05/test/swedish_talbanken05_test.conll
+./data/swedish/talbanken05/train/swedish_talbanken05_train.conll
+</pre>
+
+== Data format ==
+
+The provided data is annotated but needs to be reformatted to the OpenNLP format for each model we want to make. Here is an example of the provided Portuguese data:
+<pre>
+1	Um	um	art	art	<arti>|M|S	2	>N	_	_
+2	revivalismo	revivalismo	n	n	M|S	0	UTT	_	_
+3	refrescante	refrescante	adj	adj	M|S	2	N<	_	_
+
+1	O	o	art	art	<artd>|M|S	2	>N	_	_
+2	7_e_Meio	7_e_Meio	prop	prop	M|S	3	SUBJ	_	_
+3	é	ser	v	v-fin	PR|3S|IND	0	STA	_	_
+4	um	um	art	art	<arti>|M|S	5	>N	_	_
+5	ex-libris	ex-libris	n	n	M|P	3	SC	_	_
+6	de	de	prp	prp	<sam->	5	N<	_	_
+7	a	o	art	art	<-sam>|<artd>|S	8	>N	_	_
+8	noite	noite	n	n	F|S	6	P<	_	_
+9	algarvia	algarvio	adj	adj	F|S	8	N<	_	_
+10	.	.	punc	punc	_	3	PUNC	_	_
+
+</pre>
+
+The basic structure of the data is as follows:
+* There is one token per line
+* The token and all annotations are separated by tabs
+* Each sentence is separated by a blank line
+* There is a sentence token number before each token
+
+
+= Sentence Detector =
+
+== Format data ==
+
+=== Required format ===
+
+For sentence detection, OpenNLP requires the following format:
+<pre>
+Um revivalismo refrescante
+O 7_e_Meio é um ex-libris de a noite algarvia.
+É uma de as mais antigas discotecas de o Algarve, situada em Albufeira, que continua a manter os traços decorativos e as clientelas de sempre.
+</pre>
+
+To reformat the data, we need to remove the annotations, display one sentence per line, and remove the unnecessary spaces between tokens.
+
+=== Remove annotations ===
+
+Create a file called "formatSentence.sh" and make it executable using the command:
+<pre>
+$ chmod a+x formatSentence.sh
+</pre>
+
+Copy the following code into the file and save:
+<pre>
+#!/bin/bash
+
+SEP="\t";
+TAG="[^${SEP}]*";
+SENTENCESEP="<SENTENCE123456789SEP>";
+
+exec cat $1 | perl -pe "s/^${TAG}${SEP}(${TAG}).*$/\1/g" | perl -pe "s/^\s*$/\n/g" | perl -pe "s/^$/${SENTENCESEP}/g" | perl -pe "s/\n/ /g" | perl -pe "s/ ${SENTENCESEP} /\n/g" | grep "[[:punct:]]$"
+</pre>
+
+Run the script on the train and test files to create OpenNLP-formatted files:
+<pre>
+$ ./formatSentence.sh portuguese_bosque_train.conll > pt.sentdetect.train.tmp
+$ ./formatSentence.sh portuguese_bosque_test.conll > pt.sentdetect.test.tmp
+</pre>
+
+=== Reverse tokenization ===
+
+Remove the unnecessary spaces between tokens using the detokenization script, which can be found here: [[Detokenizing_script]]
+<pre>
+$ ./createTokTrainingData.pl pt.sentdetect.train.tmp > pt.sentdetect.train
+$ ./createTokTrainingData.pl pt.sentdetect.test.tmp > pt.sentdetect.test
+</pre>
+(You can delete the *.tmp files after this is done.)
+
+== Train a sentence detector model ==
+
+Train a sentence detector model with the following command:
+<pre>
+$ opennlp SentenceDetectorTrainer -lang pt -encoding UTF-8 -data pt.sentdetect.train -model pt.sentdetect.model
+</pre>
+
+This will display output like the following:
+<pre>
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 8448 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 8448 events to 1885.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 1885
+	    Number of Outcomes: 2
+	  Number of Predicates: 376
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-5855.707381370515	0.9269649621212122
+  2:  .. loglikelihood=-1192.584282377142	0.9473248106060606
+  3:  .. loglikelihood=-848.2674244670022	0.9636600378787878
+  4:  .. loglikelihood=-705.6746366897534	0.9717092803030303
+  5:  .. loglikelihood=-623.4294031335301	0.9739583333333334
+
+ ...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-201.27908305083162	0.9921875
+ 96:  .. loglikelihood=-200.82000607403657	0.9921875
+ 97:  .. loglikelihood=-200.36869004527148	0.9923058712121212
+ 98:  .. loglikelihood=-199.9249099062947	0.9923058712121212
+ 99:  .. loglikelihood=-199.48844948519036	0.9923058712121212
+100:  .. loglikelihood=-199.059101060208	0.9923058712121212
+Wrote sentence detector model.
+Path: <YOUR_PATH>/pt.sentdetect.model
+</pre>
+
+== Evaluate the sentence detector model ==
+
+Evaluate the sentence detector model with the following command:
+<pre>
+$ opennlp SentenceDetectorEvaluator -encoding UTF-8 -model pt.sentdetect.model -data pt.sentdetect.test
+</pre>
+
+This will display the resulting scores, e.g.:
+<pre>
+Loading model ... done
+Evaluating ... done
+
+Precision: 0.9
+Recall: 0.8666666666666667
+F-Measure: 0.8830188679245283
+</pre>
+
+== Detect sentence boundaries on raw text using the sentence detector model ==
+
+=== Create raw text ===
+
+To create raw text from the provided data, you can run the following command:
+<pre>
+$ tr '\n' ' ' < pt.sentdetect.test | perl -pe "s/$/\n\n/" > pt.sentdetect.raw
+</pre>
+
+=== Process raw text ===
+
+You can run the sentence detector on raw text with the following command:
+<pre>
+$ opennlp SentenceDetector pt.sentdetect.model < pt.sentdetect.raw > pt.sentdetect.raw.processed
+</pre>
+
+= Tokenizer =
+
+== Format data ==
+
+=== Required format ===
+
+For tokenization, OpenNLP requires the following format:
+<pre>
+Um revivalismo refrescante
+O 7_e_Meio é um ex-libris de a noite algarvia<SPLIT>.
+É uma de as mais antigas discotecas de o Algarve<SPLIT>, situada em Albufeira<SPLIT>, que continua a manter os traços decorativos e as clientelas de sempre<SPLIT>.
+</pre>
+
+To reformat the data, we need to remove the annotations, display one sentence per line, and replace the unnecessary spaces between tokens with "<SPLIT>" (e.g. spaces between some words and punctuation marks).
+
+=== Remove annotations ===
+
+Create a file called "formatToken.sh" and make it executable using the command:
+<pre>
+$ chmod a+x formatToken.sh
+</pre>
+
+Copy the following code into the file and save:
+<pre>
+#!/bin/bash
+
+SEP="\t";
+TAG="[^${SEP}]*";
+SENTENCESEP="<SENTENCE123456789SEP>";
+
+exec cat $1 | perl -pe "s/^${TAG}${SEP}(${TAG}).*$/\1/g" | perl -pe "s/^\s*$/\n/g" | perl -pe "s/^$/${SENTENCESEP}/g" | perl -pe "s/\n/ /g" | perl -pe "s/ ${SENTENCESEP} /\n/g"
+</pre>
+
+Run the script on the train and test files to create OpenNLP-formatted files:
+<pre>
+$ ./formatToken.sh portuguese_bosque_train.conll > pt.tokenizer.train.tmp
+$ ./formatToken.sh portuguese_bosque_test.conll > pt.tokenizer.test.tmp
+</pre>
+
+=== Reverse tokenization ===
+
+Replace the unnecessary spaces between tokens using the detokenization script, which can be found here: [[Detokenizing_script]]
+<pre>
+$ ./createTokTrainingData.pl pt.tokenizer.train.tmp -s > pt.tokenizer.train
+$ ./createTokTrainingData.pl pt.tokenizer.test.tmp -s > pt.tokenizer.test
+</pre>
+(You can delete the *.tmp files after this is done.)
+
+== Train a tokenizer model ==
+
+Train a tokenizer model with the following command:
+<pre>
+$ opennlp TokenizerTrainer -lang pt -encoding UTF-8 -data pt.tokenizer.train -model pt.tokenizer.model
+</pre>
+
+This will display output like the following:
+<pre>
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 746798 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 746798 events to 169238.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 169238
+	    Number of Outcomes: 2
+	  Number of Predicates: 34986
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-517640.9281477159	0.970032056861427
+  2:  .. loglikelihood=-61083.857670207995	0.9813497090243948
+  3:  .. loglikelihood=-25584.140914434272	0.9946772755149318
+  4:  .. loglikelihood=-14884.343035303249	0.9978133310480211
+  5:  .. loglikelihood=-10044.208596832164	0.9983543073227299
+
+ ...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-470.61068080912185	0.9999076055372403
+ 96:  .. loglikelihood=-467.8478678819583	0.9999076055372403
+ 97:  .. loglikelihood=-465.1374468311724	0.9999076055372403
+ 98:  .. loglikelihood=-462.47783490907756	0.9999076055372403
+ 99:  .. loglikelihood=-459.86751461783246	0.9999076055372403
+100:  .. loglikelihood=-457.30503034185386	0.9999076055372403
+Wrote tokenizer model.
+Path: <YOUR_PATH>/pt.tokenizer.model
+</pre>
+
+== Evaluate the tokenizer model ==
+
+Evaluate the tokenizer model with the following command:
+<pre>
+$ opennlp TokenizerMEEvaluator -encoding UTF-8 -model pt.tokenizer.model -data pt.tokenizer.test
+</pre>
+
+This will display the resulting scores, e.g.:
+<pre>
+Evaluating ... done
+
+Precision: 0.9978796931469459
+Recall: 0.9981251065280382
+F-Measure: 0.9980023847504222
+</pre>
+
+== Tokenize raw text with the tokenizer model ==
+
+=== Create raw text ===
+
+To create raw text from the provided data, you can run the following command:
+<pre>
+$ perl -pe "s/<SPLIT>//g" < pt.tokenizer.test > pt.tokenizer.raw
+</pre>
+
+=== Process raw text ===
+
+You can run the tokenizer on raw text with the following command:
+<pre>
+$ opennlp TokenizerME pt.tokenizer.model < pt.tokenizer.raw > pt.tokenizer.raw.processed
+</pre>
+
+= POS (Part of Speech) Tagger =
+
+== Format data ==
+
+=== Required format ===
+
+For POS tagging, OpenNLP requires the following format:
+<pre>
+Um_art revivalismo_n refrescante_adj
+O_art 7_e_Meio_prop é_v-fin um_art ex-libris_n de_prp a_art noite_n algarvia_adj ._punc
+</pre>
+
+To reformat the data, we need to extract <TOKEN>_<POS_TAG> pairs and display one sentence per line.
+
+=== Reformat data ===
+
+Create a file called "formatPOS.sh" and make it executable using the command:
+<pre>
+$ chmod a+x formatPOS.sh
+</pre>
+
+Copy the following code into the file and save:
+<pre>
+#!/bin/bash
+
+SEP="\t";
+TAG="[^${SEP}]*";
+SENTENCESEP="<SENTENCE123456789SEP>";
+
+exec cat $1 | perl -pe "s/^${TAG}${SEP}(${TAG})${SEP}${TAG}${SEP}${TAG}${SEP}(${TAG}).*$/\1_\2/g" | perl -pe "s/^\s*$/\n/g" | perl -pe "s/^$/${SENTENCESEP}/g" | perl -pe "s/\n/ /g" | perl -pe "s/ ${SENTENCESEP} /\n/g"
+</pre>
+
+Run the script on the train and test files to create OpenNLP-formatted files:
+<pre>
+$ ./formatPOS.sh portuguese_bosque_train.conll > pt.postagger.train
+$ ./formatPOS.sh portuguese_bosque_test.conll > pt.postagger.test
+</pre>
+
+== Train a POS tagger model ==
+
+Train a POS tagger model with the following command:
+<pre>
+$ opennlp POSTaggerTrainer -lang pt -encoding utf-8 -data pt.postagger.train -model pt.postagger.model
+</pre>
+
+This will display output like the following:
+<pre>
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 206678 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 206678 events to 193001.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 193001
+	    Number of Outcomes: 22
+	  Number of Predicates: 29155
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-638850.47217427	0.13807468622688432
+  2:  .. loglikelihood=-290753.190567394	0.8510291371118358
+  3:  .. loglikelihood=-185870.4166969049	0.912041920281791
+  4:  .. loglikelihood=-138215.2811448813	0.9380921046265205
+  5:  .. loglikelihood=-111406.56720261769	0.9499172626017283
+
+ ...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-14247.47118606518	0.9899166819884071
+ 96:  .. loglikelihood=-14160.555866756104	0.9899892586535577
+ 97:  .. loglikelihood=-14075.142503355162	0.9900328046526481
+ 98:  .. loglikelihood=-13991.189602799803	0.9900811890960818
+ 99:  .. loglikelihood=-13908.657233947128	0.9901150582064855
+100:  .. loglikelihood=-13827.50695352086	0.9901537657612325
+Wrote pos tagger model.
+Path: <YOUR_PATH>/pt.postagger.model
+</pre>
+
+== Evaluate the POS tagger model ==
+
+Evaluate the POS tagger model with the following command:
+<pre>
+$ opennlp POSTaggerEvaluator -encoding utf-8 -model pt.postagger.model -data pt.postagger.test
+</pre>
+
+This will display the resulting scores, e.g.:
+<pre>
+Loading model ... done
+Evaluating ... done
+
+Accuracy: 0.9659110277825124
+</pre>
+
+== Tag raw text with the POS tagger model ==
+
+=== Create raw text ===
+
+To create raw text from the provided data, you can run the following command:
+<pre>
+$ cat pt.postagger.test | perl -pe "s/([^ ]+)\_[^ _\n]+/\1/g" > pt.postagger.raw
+</pre>
+
+Note that this will preserve the existing sentence detection and tokenization, e.g. there will be one sentence per line and tokens will be separated by spaces. If you want to tag completely unformatted raw text, you will need to run sentence detection and tokenization on the text first. Assuming an unprocessed raw text file called "pt.raw", this can be done with the following command:
+<pre>
+$ opennlp SentenceDetector pt.sentdetect.model < pt.raw | opennlp TokenizerME pt.tokenizer.model > pt.postagger.raw
+</pre>
+
+''(As a note, you can use the file "pt.sentdetect.raw" created earlier in the sentence detection tutorial as the "pt.raw" file in this command.)''
+
+=== Process raw text ===
+
+You can run the POS tagger on raw text that has been through tokenization and sentence detection with the following command:
+<pre>
+$ opennlp POSTagger pt.postagger.model < pt.postagger.raw  > pt.postagger.raw.processed
+</pre>
+
+= Scored evaluations of models =
+
+These are the scored evaluation results of the models created for Danish, Dutch, Portuguese, and Swedish:
+
+== Danish ==
+
+=== Sentence Detector ===
+* Precision: 0.862876254180602
+* Recall: 0.8543046357615894
+* F-Measure: 0.8585690515806988
+
+=== Tokenizer ===
+* Precision: 0.9946374862960629
+* Recall: 0.9947026657552973
+* F-Measure: 0.9946700749578983
+
+=== POS Tagger ===
+* Accuracy: 0.951298701298701
+
+
+== Dutch ==
+
+=== Sentence Detector ===
+* Precision: 0.9767441860465116
+* Recall: 0.9368029739776952
+* F-Measure: 0.9563567362428843
+
+=== Tokenizer ===
+* Precision: 0.9998071758143379
+* Recall: 0.9996418979409132
+* F-Measure: 0.9997245300465499
+
+=== POS Tagger ===
+* Accuracy: 0.9328558639212176
+
+
+== Portuguese ==
+
+=== Sentence Detector ===
+* Precision: 0.9
+* Recall: 0.8666666666666667
+* F-Measure: 0.8830188679245283
+
+=== Tokenizer ===
+* Precision: 0.9978796931469459
+* Recall: 0.9981251065280382
+* F-Measure: 0.9980023847504222
+
+=== POS Tagger ===
+* Accuracy: 0.9659110277825124
+
+
+== Swedish ==
+
+=== Sentence Detector ===
+* Precision: 0.9695121951219511
+* Recall: 0.9607250755287009
+* F-Measure: 0.9650986342943854
+
+=== Tokenizer ===
+* Precision: 0.9985453697962308
+* Recall: 0.9977015558698727
+* F-Measure: 0.9981232844929043
+
+=== POS Tagger ===
+* Accuracy: 0.9276874115983027

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Detokenizer.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Detokenizer.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Detokenizer.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Detokenizer.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,13 @@
+The page describes how to use the OpenNLP Detokenizer.
+
+= Detokenization =
+The Detokenizer undos the tokenization of a sentence.
+It assigns to every token an operation which describes how it should be attached to
+one long character sequence which represents the original sentence.  
+
+== Detokenizer Tool ==
+TBD
+== Detokenizer API ==
+TBD
+== Rule Dictionary ==
+TBD

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Main_Page.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Main_Page.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Main_Page.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Main_Page.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,26 @@
+<big>'''OpenNLP Wiki.'''</big> [[Image:MediaWikiSidebarLogo.png]]
+
+Wiki to get started with OpenNLP documentation and OpenNLP tutorials.
+== OpenNLP Documentation ==
+* [[Sentence Detector|Sentence Detector]]
+* [[Tokenizer|Tokenizer]]
+* [[Detokenizer|Detokenizer (WIP)]]
+* [[Name Finder|Name Finder]]
+* [[POS Tagger| POS Tagger (WIP)]]
+* [[Chunker| Chunker (WIP)]]
+* [[Parser| Parser (WIP)]]
+
+== OpenNLP Development ==
+* [[OpenNLP_Installation| Getting the source and building]]
+
+== OpenNLP Tutorials ==
+* [[tokenization|Sentence splitters and tokenizers]]
+* [[postagger|Part-of-speech taggers]]
+
+== Supported Corpora ==
+The following pages explain how to convert the various corpora into the OpenNLP training
+formats: 
+* [[Conll02| Conll 2002 (WIP)]]
+* [[Conll03| Conll 2003 (WIP, needs current 1.5.1-SNAPSHOT))]]
+* [[Conll06| Conll 2006 (WIP)]]
+* [[Arvores Deitadas|Floresta Sita(c)tica Arvores Deitadas (WIP, needs current 1.5.1-SNAPSHOT)]]

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Name_Finder.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Name_Finder.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Name_Finder.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Name_Finder.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,305 @@
+The page describes how to use the OpenNLP Name Finder.
+
+= Detecting Names =
+
+The Name Finder can detect named entities and numbers in text. 
+
+To be able to detect entities the Name Finder needs a model.
+The model is dependent on the language and entity type it was trained
+for. The OpenNLP projects offers a number of pre-trained name finder models
+which are trained on various freely available corpora. They can be downloaded
+at our  [http://opennlp.sourceforge.net/models-1.5/en-ner-person.bin model download page].
+
+To find names in raw text the text must be segmented into tokens
+and sentences. A detailed description is given in the sentence detector
+and tokenizer tutorial. Its important that the tokenization for the training
+data and the input text is identical.
+
+== Name Finder Tool ==
+The easiest way to try out the Name Finder is the command line tool. The tool
+is only intended for demonstration and testing.
+
+Download the [http://opennlp.sourceforge.net/models-1.5/en-ner-person.bin english person model] and start 
+the Name Finder Tool with this command:
+<pre>
+bin/opennlp TokenNameFinder en-ner-person.bin
+</pre>
+
+The name finder now reads a tokenized sentence per line from stdin,
+an empty line indicates a document boundary and resets the adaptive
+feature generators.
+
+Just copy this text to the terminal:
+<pre>
+Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
+Mr . Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
+Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC , was named a director of this British industrial conglomerate .
+
+</pre>
+
+the name finder will now output the text with markup for
+person names:
+<pre>
+<START:person> Pierre Vinken <END> , 61 years old , will join the board as a nonexecutive director Nov. 29 .
+Mr . <START:person> Vinken <END> is chairman of Elsevier N.V. , the Dutch publishing group .
+<START:person> Rudolph Agnew <END> , 55 years old and former chairman of Consolidated Gold Fields PLC , was named a director of this British industrial conglomerate .
+
+</pre>
+
+== Name Finder API ==
+To use the Name Finder in a production system its strongly recommended
+to embed it directly into the application instead of using the command line interface.
+
+First the name finder model must be loaded into memory from disk or an other source.
+In the sample below its loaded from disk.
+<pre>
+InputStream modelIn = new FileInputStream("en-ner-person.bin");
+
+try {
+  TokenNameFinder model = new TokenNameFinderModel(modelIn);
+}
+catch (IOException e) {
+  e.printStackTrace();
+}
+finally {
+  if (modelIn != null) {
+    try {
+      modelIn.close();
+    }
+    catch (IOException e) {
+    }
+  }
+}
+</pre>
+
+There is a number of reasons why the model loading can fail:
+* Issues with the underlying I/O
+* The version of the model is not compatible with the OpenNLP version
+* The model is loaded into the wrong component, for example a tokenizer model is loaded with TokenNameFinderModel class.
+* The model content is not valid for some other reason
+
+After the model is loaded the NameFinderME can be instantiated.
+
+<pre>
+NameFinderME nameFinder = new NameFinderME(model);
+</pre>
+
+The initialization is now finished and the Name Finder can be used.
+The NameFinderME class is not thread safe, it must only be called from one thread. To use multiple
+threads multiple NameFinderME instances sharing the same model instance can be created.
+
+The input text should be segmented into documents, sentences
+and tokens. 
+
+To perform entity detection an application calls the find method for every sentence in the document. 
+After every document clearAdaptiveData must be called to clear the adaptive data in the feature generators.
+Not calling clearAdaptiveData can lead to a sharp drop in the detection rate after a few documents.
+
+The following code illustrates that:
+<pre>
+for (String document[][] : documents) {
+
+  for (String[] sentence : document) {
+    Span nameSpans[] = find(sentence);
+    // do something with the names
+  }
+
+  nameFinder.clearAdaptiveData()
+}
+</pre>
+
+the following snippet shows a call to find
+<pre>
+String sentence = new String[]{
+    "Pierre",
+    "Vinken",
+    "is",
+    "61",
+    "years"
+    "old",
+    "."
+    };
+
+Span nameSpans[] = nameFinder.find(sentence);
+
+</pre>
+ 
+The nameSpans arrays contains now exactly one Span which marks the name Pierre Vinken. The elements
+between the begin and end offsets are the name tokens. In this case the begin offset is 0 and the end offset is 2.
+The Span object also knows the type of the entity. In this case its person (defined by the model). It can be retrieved
+with a call to Span.getType().
+
+Additionally to the statistical Name Finder, OpenNLP also offers a dictionary and a regular expression name finder implementation.
+
+TODO: Explain how to retrieve probs from the name finder for names and for non recognized names
+
+= Training =
+The pre-trained models might not be available for a desired language, can not
+detect important entities or the performance is not good enough outside the news domain.
+
+These are the typical reason to do custom training of the name finder on a new corpus
+or on a corpus which is extended by private training data taken from the data which
+should be analyzed.
+
+== Training Tool ==
+
+OpenNLP has a command line tool which is used to train the models available from
+the model download page on various corpora.
+
+The data must be converted to the OpenNLP name finder training format.
+Which is one sentence per line. The sentence must be tokenized and contain spans which mark the
+entities. Documents are separated by empty lines which trigger the reset of the adaptive feature
+generators. A training file can contain multiple types. If the training file contains multiple types
+the created model will also be able to detect these multiple types.
+For now its recommended to only train single type models, since multi type support is
+stil experimental.
+
+Sample sentence of the data:
+<pre>
+<START:person> Pierre Vinken <END> , 61 years old , will join the board as a nonexecutive director Nov. 29 .
+Mr . <START:person> Vinken <END> is chairman of Elsevier N.V. , the Dutch publishing group .
+</pre>
+
+The training data should contain at least 15000 sentences to create a model which performs well.
+
+Usage of the tool:
+<pre>
+$ bin/opennlp TokenNameFinderTrainer
+Usage: opennlp TokenNameFinderTrainer -lang language -encoding charset [-iterations num] [-cutoff num] [-type type] -data trainingData -model model
+-lang language     specifies the language which is being processed.
+-encoding charset  specifies the encoding which should be used for reading and writing text.
+-iterations num    specified the number of training iterations
+-cutoff num        specifies the min number of times a feature must be seen
+-type The type of the token name finder model
+</pre>
+
+Its now assumed that the english person name finder model should be trained from a file
+called en-ner-person.train which is encoded as UTF-8.
+The following command will train the name finder and write the model to en-ner-person.bin:
+<pre>
+bin/opennlp TokenNameFinderTrainer -encoding UTF-8 -lang en -data en-ner-person.train -model en-ner-person.bin
+</pre>
+
+Additionally its possible to specify the number of iterations, the cutoff and to overwrite all types in the training data with a single type.
+
+== Training API ==
+To train the name finder from within an application its recommended to use the training API instead
+of the command line tool.
+
+Basically three steps are necessary to train it:
+* The application must open a sample data stream
+* Call the NameFinderME.train method
+* Save the TokenNameFinderModel to a file or database
+
+The three steps are illustrated by the following sample code:
+<pre>
+ObjectStream<String> lineStream = new PlainTextByLineStream(new FileInputStream("en-ner-person.train"), "UTF-8");
+ObjectStream<NameSample> sampleStream = new NameSampleDataStream(lineStream);
+
+TokenNameFinderModel model = NameFinderME.train("en", "person", sampleStream, Collections.<String, Object>emptyMap(), 100, 5);
+
+try {
+  modelOut = new BufferedOutputStream(new FileOutputStream(modelFile));
+  model.serialize(modelOut);
+} finally {
+  if (modelOut != null) 
+     modelOut.close();      
+}
+</pre>
+
+== Custom Feature Generation ==
+OpenNLP defines a default feature generation which is used when no custom feature generation is specified.
+Users which want to experiment with the feature generation can provide a custom feature generator.
+The custom generator must be used for training and for detecting the names. If the feature generation during
+training time and detection time is different the name finder might not be able to detect names.
+
+The following lines show how to construct a custom feature generator
+<pre>
+AdaptiveFeatureGenerator featureGenerator = new CachedFeatureGenerator(
+         new AdaptiveFeatureGenerator[]{
+           new WindowFeatureGenerator(new TokenFeatureGenerator(), 2, 2),
+           new WindowFeatureGenerator(new TokenClassFeatureGenerator(true), 2, 2),
+           new OutcomePriorFeatureGenerator(),
+           new PreviousMapFeatureGenerator(),
+           new BigramNameFeatureGenerator(),
+           new SentenceFeatureGenerator(true, false)
+           });
+</pre>
+which is similar to the default feature generator.
+
+The javadoc of the feature generator classes explain what the individual feature generators do.
+To write a custom feature generator please implement the AdaptiveFeatureGenerator interface
+or if it must not be adaptive extend the FeatureGeneratorAdapter.
+
+The train method which should be used is defined as
+<pre>
+public static TokenNameFinderModel train(String languageCode, String type, ObjectStream<NameSample> samples, 
+       AdaptiveFeatureGenerator generator, final Map<String, Object> resources, 
+       int iterations, int cutoff) throws IOException
+</pre>
+and can take feature generator as an argument.
+
+To detect names the model which was returned from the train method
+and the feature generator must be passed to the NameFinderME constructor.
+
+<pre>
+new NameFinderME(model, featureGenerator, NameFinderME.DEFAULT_BEAM_SIZE);
+</pre>
+
+== Named Entity Annotation Guidelines ==
+Annotation guidelines define what should be labeled as an entity. To build a private corpus
+its important to know these guidelines and maybe write a custom one. 
+
+Here is a list of publicly available annotation guidelines:
+* [http://cs.nyu.edu/cs/faculty/grishman/NEtask20.book_1.html MUC6]
+* [http://acl.ldc.upenn.edu/muc7/ne_task.html MUC7]
+* [http://projects.ldc.upenn.edu/ace/docs/English-Entities-Guidelines_v5.6.1.pdf ACE]
+* [http://www.cnts.ua.ac.be/conll2003/ner/annotation.txt CONLL-2003]
+
+= Evaluation =
+
+The built in evaluation can measure the named entity recognition performance of the name finder.
+The performance is either measured on a test dataset or via cross validation.
+
+== Evaluator Tool ==
+
+The following command shows how the tool can be run:
+<pre>
+bin/opennlp TokenNameFinderEvaluator -encoding UTF-8 -model en-ner-person.bin -data en-ner-person.test
+</pre>
+and here is a sample output:
+<pre>
+Precision: 0.8005071889818507
+Recall: 0.7450581122145297
+F-Measure: 0.7717879983140168
+</pre>
+
+Note: The command line interface does not support cross evaluation in the current version.
+
+== Evaluation API ==
+The evaluation can be performed on a pre-trained model and a test dataset or via cross validation.
+
+In the first case the model must be loaded and a NameSample ObjectStream must be created (see code samples above),
+assuming these two objects exist the following code shows how to perform the evaluation:
+
+<pre>
+TokenNameFinderEvaluator evaluator = new TokenNameFinderEvaluator(new NameFinderME(model));
+evaluator.evaluate(sampleStream);
+
+FMeasure result = evaluator.getFMeasure();
+
+System.out.println(result.toString());
+</pre>
+
+In the cross validation case all the training arguments must be provided (see the Training API section above).
+To perform cross validation the ObjectStream must be resettable.
+<pre>
+FileInputStream sampleDataIn = new FileInputStream("en-ner-person.train");
+ObjectStream<NameSample> sampleStream = new PlainTextByLineStream(sampleDataIn.getChannel(), "UTF-8"); 
+TokenNameFinderCrossValidator evaluator = new TokenNameFinderCrossValidator("en", 100, 5);
+evaluator.evaluate(sampleStream, 10);
+
+FMeasure result = evaluator.getFMeasure();
+
+System.out.println(result.toString());
+</pre>

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/OpenNLP_Installation.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/OpenNLP_Installation.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/OpenNLP_Installation.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/OpenNLP_Installation.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,100 @@
+= Setting up OpenNLP =
+
+== Obtaining OpenNLP ==
+
+=== Download a released version ===
+
+Go to [https://sourceforge.net/projects/opennlp/files/ the download page] and choose the appropriate file to download. To get an already compiled system, choose opennlp-tools-version.bin.zip for Windows and opennlp-tools-version.tar.gz for other operating systems. 
+
+=== Get OpenNLP from CVS ===
+
+If you need to be on the bleeding edge, you can get the latest code from the CVS repository. (Yes, CVS is ancient---we will be moving to git or mercurial soon!) 
+
+Instructions to get it are available on [https://sourceforge.net/scm/?type=cvs&group_id=3368 the Sourceforge page for OpenNLP's CVS]. You want the 'OpenNLP' module:
+<pre>
+$ cvs -z3 -d:pserver:anonymous@opennlp.cvs.sourceforge.net:/cvsroot/opennlp co -P opennlp
+</pre>
+
+This will check-out the source code to the folder <code><YOUR_PATH>/opennlp</code>. After you have done that, you can follow the instructions in the README. 
+
+== Build OpenNLP ==
+
+In the "opennlp" directory, the following command will build the project:
+<pre>
+$ mvn install
+</pre>
+
+If the build is successful, you should see output similar to the following:
+<pre>
+[INFO] Scanning for projects...
+[INFO] artifact org.apache.maven.wagon:wagon-ssh: checking for updates from OpenNLPRepository
+[INFO] artifact org.apache.maven.wagon:wagon-ssh: checking for updates from central
+Downloading: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh/1.0-beta-6/wagon-ssh-1.0-beta-6.pom
+      
+Downloading: http://opennlp.sourceforge.net/maven2/org/apache/maven/wagon/wagon-providers/1.0-beta-6/wagon-providers-1.0-beta-6.pom
+[INFO] Unable to find resource 'org.apache.maven.wagon:wagon-providers:pom:1.0-beta-6' in repository OpenNLPRepository (http://opennlp.sourceforge.net/maven2)
+Downloading: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-providers/1.0-beta-6/wagon-providers-1.0-beta-6.pom
+      
+...<skipping a bunch of output>...
+
+[INFO] [jar:jar {execution: default-jar}]
+[INFO] Building jar: <YOUR_PATH>/opennlp/target/opennlp-tools-1.5.0-SNAPSHOT.jar
+[INFO] [install:install {execution: default-install}]
+[INFO] Installing <YOUR_PATH>/opennlp/target/opennlp-tools-1.5.0-SNAPSHOT.jar to <YOUR_HOME_DIRECTORY>/.m2/repository/opennlp/tools/1.5.0-SNAPSHOT/tools-1.5.0-SNAPSHOT.jar
+[INFO] ------------------------------------------------------------------------
+[INFO] BUILD SUCCESSFUL
+[INFO] ------------------------------------------------------------------------
+[INFO] Total time: 2 minutes 50 seconds
+[INFO] Finished at: Tue Aug 03 14:10:43 CDT 2010
+[INFO] Final Memory: 36M/386M
+[INFO] ------------------------------------------------------------------------
+</pre>
+
+This will create a new directory, <code><YOUR_PATH>/opennlp/target</code>
+
+== Get OpenNLP's Maximum Entropy Package from CVS ==
+
+You may also need to check-out the most recent MaxEnt project from CVS. Instructions to get it are available on [https://sourceforge.net/scm/?type=cvs&group_id=5961 the Sourceforge page for OpenNLP's Maximum Entropy Package CVS]. You want the 'maxent' module:
+<pre>
+$ cvs -z3 -d:pserver:anonymous@maxent.cvs.sourceforge.net:/cvsroot/maxent co -P maxent
+</pre>
+
+This will check-out the source code to the folder <code><YOUR_PATH>/maxent</code>.
+
+== Build OpenNLP's Maximum Entropy Package (Optional step) ==
+
+In the "maxent" directory, the following command will build the project:
+<pre>
+$ mvn install
+</pre>
+
+Again, you should see the "BUILD SUCCESSFUL" message in the output. This will produce the file <code><YOUR_PATH>/maxent/target/maxent-3.0-SNAPSHOT.jar</code>. Since creating the models in this tutorial require the latest MaxEnt package, you will need to copy this file to <code><YOUR_PATH>/opennlp/target/maxent-3.0-SNAPSHOT.jar</code>:
+<pre>
+$ cp <YOUR_PATH>/maxent/target/maxent-3.0.0.jar <YOUR_PATH>/opennlp/target/
+</pre>
+
+== Running OpenNLP ==
+
+=== opennlp script ===
+
+The OpenNLP package should include the executable file <code><YOUR_PATH>/opennlp/bin/opennlp</code>. The example commands in this tutorial call the "opennlp" script to run OpenNLP. You should add <code><YOUR_PATH>/opennlp/bin</code> to your PATH environment variable.
+
+If the opennlp script file is not working correctly, you may need to fix it by replacing it with the following code:
+<pre>
+#!/bin/sh
+
+if [ -z "$JAVACMD" ] ; then
+  if [ -n "$JAVA_HOME"  ] ; then
+    JAVACMD="$JAVA_HOME/bin/java"
+  else
+    JAVACMD="`which java`"
+  fi
+fi
+
+# sets OPENNLP environment variables
+JARS=`echo $OPENNLP_DIR/lib/*.jar $OPENNLP_DIR/target/*.jar | tr ' ' ':'`
+CP=$OPENNLP_DIR/target/classes:${JARS}
+JAVACMD="$JAVACMD -Xmx1024m -classpath $CP -Dopennlp.dir=$OPENNLP_DIR"
+
+$JAVACMD opennlp.tools.cmdline.CLI $@
+</pre>

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/POS_Tagger.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/POS_Tagger.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/POS_Tagger.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/POS_Tagger.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,65 @@
+The page describes how to use the OpenNLP Part of Speech Tagger.
+
+= Part of Speech Tagging =
+The Part of Speech Tagger marks tokens with their corresponding word type based on the token itself and the context of the token. A token can have multiple pos tags depending on the token and the context. The OpenNLP POS Tagger uses a probability model to guess the correct pos tag out of the tag set. 
+To limit the possible tags for a token a tag dictionary can be used which increases the tagging and runtime performance of the tagger.
+
+== POS Tagger Tool ==
+The easiest way to try out the POS Tagger is the command line tool. The tool is only intended for demonstration and testing.
+
+Download the [http://opennlp.sourceforge.net/models-1.5/en-pos-maxent.bin english maxent pos model] 
+and start the POS Tagger Tool with this command:
+
+<pre>
+bin/opennlp POSTagger en-pos-maxent.bin
+</pre>
+
+The POS Tagger now reads a tokenized sentence per line from stdin.
+
+Copy these two sentences to the console:
+<pre>
+Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
+Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
+</pre>
+
+the POS Tagger will now echo the sentences with pos tags to the console:
+<pre>
+Pierre_NNP Vinken_NNP ,_, 61_CD years_NNS old_JJ ,_, will_MD join_VB the_DT board_NN as_IN a_DT nonexecutive_JJ director_NN Nov._NNP 29_CD ._.
+Mr._NNP Vinken_NNP is_VBZ chairman_NN of_IN Elsevier_NNP N.V._NNP ,_, the_DT Dutch_NNP publishing_VBG group_NN
+</pre>
+
+The tag set used by the english pos model is the Penn Treebank tag set. See the link below for a description of the tags.
+
+== POS Tagger API ==
+
+== Public Tag Sets ==
+* [http://en.wikipedia.org/wiki/Brown_Corpus#Part-of-speech_tags_used Brown Corpus Tags]
+* [http://www.ims.uni-stuttgart.de/projekte/CorpusWorkbench/CQP-HTMLDemo/PennTreebankTS.html Penn Treebank Tags]
+
+TODO: Add more tag sets, also for non-english languages
+
+= Training =
+== Training Tool ==
+== Training API ==
+
+== Custom Feature Generation ==
+= Evaluation =
+
+The tagging performance is measured as an accuracy. The accuracy score is the ratio of the number of  correctly tagged tokens out
+of the totally tagged tokens. 
+
+<math>\mbox{Accurarcy} = \frac{\mbox{correctly tagged tokens}}{\mbox{total tokens}}</math>
+== Evaluator Tool ==
+This is also very easy, given that you have more annotated material available.
+
+<pre>
+$ opennlp POSTaggerEvaluator -encoding utf-8 -model english.postagger.model -data dev.pos 
+Loading model ... done
+Evaluating ... done
+
+Accuracy: 0.9657527774403002
+</pre>
+
+Which is pretty standard performance for this sort of tagger.
+
+== Evaluator API ==

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Parser.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Parser.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Parser.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Parser.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,139 @@
+The page describes how to use the OpenNLP Parser.
+
+= Parsing =
+TODO: Write an introduction for the parser.
+
+== Parser Tool ==
+
+The easiest way to try out the Parser is the command line tool. The tool is only intended for demonstration and testing.
+
+Download the  [http://opennlp.sourceforge.net/models-1.5/en-parser-chunking.bin english chunking parser model] from the our website and start
+the Parser Tool with the following command. 
+<pre>
+$ bin/opennlp Parser en-parser.bin en-parser-chunking.bin
+</pre>
+Loading the big parser model can take several seconds, be patient.
+
+Copy this sample sentence to the console.
+<pre>
+The quick brown fox jumps over the lazy dog .
+</pre>
+
+The parser should now print the following to the console.
+<pre>
+(TOP (NP (NP (DT The) (JJ quick) (JJ brown) (NN fox) (NNS jumps)) (PP (IN over) (NP (DT the) (JJ lazy) (NN dog))) (. .)))
+</pre>
+
+With the following command the input can be read from a file and be written
+to an output file.
+<pre>
+$ bin/opennlp Parser en-parser.bin en-parser-chunking.bin < article-tokenized.txt > article-parsed.txt.
+</pre>
+The article-tokenized.txt file must contain one sentence per line which is tokenized with the english
+tokenizer model from our website. See the Tokenizer documentation for further details.
+
+== Parser API ==
+The Parser can be easily integrated into an application via its API.
+
+To instantiate a Parser the parser model must be loaded first.
+<pre>
+InputStream modelIn = new FileInputStream("en-parser-chunking.bin");
+try {
+  ParserModel model = new ParserModel(modelIn);
+}
+catch (IOException e) {
+  e.printStackTrace();
+}
+finally {
+  if (modelIn != null) {
+    try {
+      modelIn.close();
+    }
+    catch (IOException e) {
+    }
+  }
+}
+</pre>
+
+Unlike the other components to instantiate the Parser a factory method
+should be used instead of creating the Parser via the new operator.
+The parser model is either trained for the chunking parser or the tree insert parser
+the parser implementation must be chosen correctly. The factory method
+will read a type parameter from the model and create an instance of
+the corresponding parser implementation.
+<pre>
+Parser parser = ParserFactory.create(model);
+</pre>
+Right now the tree insert parser is still experimental and there is no pre-trained model for it.
+
+The parser expect a whitespace tokenized sentence.
+A utility method from the command line tool can parse the sentence String.
+The following code shows how the parser can be called.
+<pre>
+String sentence = "The quick brown fox jumps over the lazy dog .";
+Parse topParses[] = ParserTool.parseLine(sentence, parser, 1);
+</pre>
+The topParses array only contains one parse because the number of parses is set to 1.
+The Parse object contains the parse tree.
+
+To display the parse tree call the show method. It either prints the parse to the console
+or into a provided StringBuffer. Similar to Exception.printStackTrace.
+
+TODO: Extend this section with more information about the Parse object
+
+= Training =
+The OpenNLP offers two different parser implementations, the chunking parser and the treeinsert parser. 
+The later one is still experimental and not recommended for production use.
+
+(TODO: Add a section which explains the two different approches)
+
+The training can either be done with the command line tool or the training API.
+In the first case the training data must be available in the OpenNLP format. 
+Which is the Penn Treebank format, but with the limitation of a sentence per line.
+
+<pre>
+(TOP (S (NP-SBJ (DT Some) )(VP (VBP say) (NP (NNP November) ))(. .) ))
+(TOP (S (NP-SBJ (PRP I) )(VP (VBP say) (NP (CD 1992) ))(. .) ('' '') ))
+</pre>
+
+(TODO: Insert link which explains the penn treebank format.)
+
+A parser model also contains a pos tagger model, depending on the amount of available
+training data it is recommend to switch the tagger model against a tagger model which 
+was trained on a larger corpus. The pre-trained parser model provided on the website is doing this to achieve
+a better performance. (TODO: On which data is the model on the website trained, and say on which data the tagger
+model is trained)
+
+== Training Tool ==
+
+OpenNLP has a command line tool which is used to train the models available from the model download page on various corpora.
+The data must be converted to the OpenNLP parser training format, which is shortly explained above.
+
+To train the parser a head rules file is also needed. (TODO: Add documentation about the head rules file)
+
+Usage of the tool:
+<pre>
+$ bin/opennlp ParserTrainer
+Usage: opennlp ParserTrainer-lang language -encoding charset [-iterations num] [-cutoff num] -head-rules head_rules -data trainingData -model model
+-lang language     specifies the language which is being processed.
+-encoding charset  specifies the encoding which should be used for reading and writing text.
+-iterations num    specified the number of training iterations
+-cutoff num        specifies the min number of times a feature must be seen
+</pre>
+
+The model on the website was trained with the following command:
+<pre>
+bin/opennlp ParserTrainer -encoding ISO-8859-1 -lang en -parserType CHUNKING -head-rules head_rules -data train.all -model en-parser-chunking.bin
+</pre>
+Its also possible to specify the cutoff and the number of iterations, these parameters are used for all trained models. <br>
+The -parserType parameter is an optional parameter, to use the tree insertion parser, specify TREEINSERT as type.
+
+The TaggerModelReplacer tool replaces the tagger model inside the parser model with a new one. <br>
+Note: The original parser model will be overwritten with the new parser model which contains the replaced
+tagger model.
+
+<pre>
+bin/opennlp TaggerModelReplacer  models/en-parser-chunking.bin models/en-pos-maxent.bin
+</pre>
+
+Additionally there are tools to just retrain the build or the check model.

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Postagger.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Postagger.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Postagger.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Postagger.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,157 @@
+The page describes how to train and use a POS tagger with OpenNLP.
+
+= Get OpenNLP from CVS =
+
+
+This tutorial assumes you are working with the code in the CVS (a new release is in the works). Instructions to get it are available on [https://sourceforge.net/scm/?type=cvs&group_id=3368 the Sourceforge page for OpenNLP's CVS]. You want the 'OpenNLP' module.
+
+After you have done that, make sure to follow the instructions in the README.
+
+= Get data =
+
+You need a corpus of words that have each been labeled with a part-of-speech. In computational linguistics, the standard dataset for this is the Penn Treebank. Many people will not have access to this dataset due to the cost, but this tutorial will work an example with it that should apply to other datasets.
+
+You should have at least a training set and a test set, both of which contain words and their annotated tags. The training set will be used to teach the POS tagger, and the test set will be used to evaluate its accuracy on data it didn't get to learn on. Ideally, you'd also have a development set so you can try out different parameters and such, but that may not be possible in all cases and doesn't matter as much if you aren't publishing papers.
+
+The form of the annotated data is ''word1_tag1 word2_tag2 ... wordN_tagN'', with one sentence per line, e.g.: (Note: the word/tag separator will probably be made a configurable option soon.)
+
+<pre>
+Pierre_NNP Vinken_NNP ,_, 61_CD years_NNS old_JJ ,_, will_MD join_VB the_DT board_NN as_IN a_DT nonexecutive_JJ director_NN Nov._NNP 29_CD ._.
+Mr._NNP Vinken_NNP is_VBZ chairman_NN of_IN Elsevier_NNP N.V._NNP ,_, the_DT Dutch_NNP publishing_VBG group_NN ._.
+Rudolph_NNP Agnew_NNP ,_, 55_CD years_NNS old_JJ and_CC former_JJ chairman_NN of_IN Consolidated_NNP Gold_NNP Fields_NNP PLC_NNP ,_, was_VBD named_VBN a_DT nonexecutive_JJ director_NN of_IN this_DT British_JJ industrial_JJ conglomerate_NN ._.
+</pre>
+
+In what follows, we'll refer to the annotated files as <code>train.pos</code>, <code>dev.pos</code>, and <code>test.pos</code>.  The data being used below is the Wall Street Journal portion of the Penn Treebank, with:
+
+* train.pos: sections 00-18
+* dev.pos: sections 19-21
+* test.pos: sections 22-24
+
+These are standardly used sections in much published part-of-speech tagging research. 
+
+'''Tip''': Here's how to convert from the PTB *.pos format to this one, including creating the train, dev and test sets.
+
+Save the following as <code>/tmp/make_tagger_format.sh</code>:
+<pre>
+cat $1 | tr -s '[]\n' ' ' | perl -pe "s/==+/\n/g" | tr -s ' ' ' ' | perl -pe "s/_/UNDERSCORE/g" | perl -pe "s/(.\/\.)[^\n](\w)/\1\n\2/g" | perl -pe "s/([^\\\\])\//\1_/g" | perl -pe "s/^ //g" | perl -pe "s/\s+\n$/\n/g" | perl -pe "s/^\s+//g"
+</pre>
+
+Make it executable:
+<pre>
+$ cd /tmp
+$ chmod a+x make_tagger_format.sh
+</pre>
+
+Now go to your copy of the Penn Treebank and do the following commands.
+
+<pre>
+$ cd <PATH_TO_TREEBANK>/penn-treebank-rel3/tagged/pos/wsj/
+$ cat 0[0-9]/* > /tmp/wsj.train.pos
+$ cat 1[0-8]/* >> /tmp/wsj.train.pos
+$ cat 19/* > /tmp/wsj.dev.pos
+$ cat 2[01]/* >> /tmp/wsj.dev.pos
+$ cat 2[2-4]/* > /tmp/wsj.test.pos
+$ cd /tmp
+$ cat wsj.train.pos | ./make_tagger_format.sh > train.pos
+$ cat wsj.dev.pos | ./make_tagger_format.sh > dev.pos
+$ cat wsj.test.pos | ./make_tagger_format.sh > test.pos
+</pre>
+
+Some free datasets with POS tag annotations are available, see the links from the [http://alias-i.com/lingpipe/demos/tutorial/posTags/read-me.html LingPipe tutorial for part-of-speech tagging]. (This tutorial will probably be updated to use one or more of them.) 
+
+Datasets with part-of-speech tags for several other languages are available from [http://nextens.uvt.nl/~conll/free_data.html the CoNLL-X Shared Task site].
+
+The [http://code.google.com/p/copenhagen-dependency-treebank/wiki/CDT Copenhagen Dependency Treebank] promises to deliver even more freely available annotations, for English, Italian, Spanish and German -- expected delivery of Fall 2010.
+
+= Train a tagging model =
+
+Training a model is very simple. Assuming the <code>train.pos</code> annotated file is in your current working directory, the following should work:
+
+<pre>
+$ opennlp POSTaggerTrainer -lang en -encoding utf-8 -data train.pos -model english.postagger.model
+</pre>
+
+
+This trains a maxent model that makes the classification decisions in the tagger. You should see output like the following:
+
+<pre>
+$ opennlp POSTaggerTrainer -lang en -encoding utf-8 -data train.pos -model english.postagger.model 
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 969905 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 969905 events to 887620.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 887620
+	    Number of Outcomes: 77
+	  Number of Predicates: 78768
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-4213078.597772371	0.09687134306968208
+  2:  .. loglikelihood=-1755066.9973144585	0.9087023986885313
+  3:  .. loglikelihood=-956515.2443204354	0.9387342059273847
+  4:  .. loglikelihood=-642486.0984935976	0.9513024471468855
+  5:  .. loglikelihood=-485562.16818665236	0.9582453951675679
+
+...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-59048.760230481486	0.9884617565637872
+ 96:  .. loglikelihood=-58728.33420953592	0.9885040287450833
+ 97:  .. loglikelihood=-58413.35118933447	0.9885566112145003
+ 98:  .. loglikelihood=-58103.66235651382	0.9886071316262933
+ 99:  .. loglikelihood=-57799.12447062581	0.9886545589516499
+100:  .. loglikelihood=-57499.599600066285	0.9886947690753218
+Wrote pos tagger model.
+Path: /tmp/english.postagger.model
+</pre>
+
+The log-likelihood is the log probability that the model at the current iteration assigns to the training data (bigger is better, corresponding to higher probability). The second value gives the classification accuracy on the training set on that iteration.
+
+One can also try a perceptron with the option <code>-model perceptron</code>. For a full list of options, do:
+
+<pre>
+$ opennlp POSTaggerTrainer help
+Usage: opennlp POSTaggerTrainer-lang language -encoding charset [-iterations num] [-cutoff num] [-dict tagdict] [-model maxent|perceptron|perceptron_sequence] -data trainingData -model model 
+</pre>
+
+= Evaluating the model =
+
+This is also very easy, given that you have more annotated material available.
+
+<pre>
+$ opennlp POSTaggerEvaluator -encoding utf-8 -model english.postagger.model -data dev.pos 
+Loading model ... done
+Evaluating ... done
+
+Accuracy: 0.9657527774403002
+</pre>
+
+Which is pretty standard performance for this sort of tagger.
+
+= Tagging raw text with the model =
+
+Of course, the ultimate reason for having a POS tagger is to use it to tag raw texts. If you are sitting there with only annotated texts with ''word_pos'' format as discussed above, here's an easy way to strip the ''_pos'' part of each ''word_pos'' pair (in Unix):
+
+<pre>
+$ cat dev.pos | perl -pe "s/([^ _]+)\_[^ \n]+/\1/g" > dev.txt
+</pre>
+
+Or, you can just pull in some raw text and tag it. Of course, you need to make sure that your raw text has been tokenized according to the same conventions as the text you trained your tagger on.((Doing sentence detection and tokenization will be another tutorial.))
+
+Once you have raw text, you can tag it:
+
+<pre>
+$ opennlp POSTagger english.postagger.model < dev.txt  > dev.tagged.pos
+Loading model ... done
+$ more dev.tagged.pos 
+The_DT Arizona_NNP Corporations_NNP Commission_NNP authorized_VBD an_DT 11.5_CD %_NN rate_NN increase_NN at_IN Tucson_NNP Electric_NNP Power_NNP Co._NNP ,_, substantially_RB lower_JJR than_IN recommended_VBN last_JJ month_NN by_IN a_DT commission_NN hearing_NN officer_NN and_CC barely_RB half_NN the_DT rise_NN sought_VBN by_IN the_DT utility_NN ._.
+The_DT ruling_NN follows_VBZ a_DT host_NN of_IN problems_NNS at_IN Tucson_NNP Electric_NNP ,_, including_VBG major_JJ write-downs_NNS ,_, a_DT 60_CD %_NN slash_NN in_IN the_DT common_JJ stock_NN dividend_NN and_CC the_DT departure_NN of_IN former_JJ Chairman_NNP Einar_NNP Greve_NNP during_IN a_DT company_NN investigation_NN of_IN his_PRP$ stock_NN sales_NNS ._.
+The_DT Arizona_NNP regulatory_JJ ruling_NN calls_VBZ for_IN $_$ 42_CD million_CD in_IN added_JJ revenue_NN yearly_NN ,_, compared_VBN with_IN a_DT $_$ 57_CD million_CD boost_NN proposed_VBN by_IN the_DT commission_NN hearing_NN officer_NN ._.
+The_DT company_NN had_VBD sought_VBN increases_NNS totaling_VBG $_$ 80.3_CD million_CD ,_, or_CC 22_CD %_NN ._.
+The_DT decision_NN was_VBD announced_VBN after_IN trading_NN ended_VBD ._.
+Tucson_NNP Electric_NNP closed_VBD at_IN $_$ 20.875_CD a_DT share_NN ,_, down_RB 25_CD cents_NNS ,_, in_IN New_NNP York_NNP Stock_NNP Exchange_NNP composite_JJ trading_NN ._.
+...
+</pre>

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Sentence_Detector.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Sentence_Detector.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Sentence_Detector.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Sentence_Detector.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,189 @@
+The page describes how to use the OpenNLP Sentence Detector.
+
+= Detecting Sentence Boundaries =
+The OpenNLP Sentence Detector can detect that a punctuation character marks the end of a sentence or not. In this sense
+a sentence is defined as the longest white space trimmed character sequence between two punctuation marks. The first and last
+sentence make an exception to this rule. The first non whitespace character is assumed to be the begin
+of a sentence, and the last non whitespace character is assumed to be a sentence end.  
+
+The sample text below should be segmented into its sentences.
+<blockquote>
+''Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.  Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.  Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields PLC, was named a director of this British industrial conglomerate. ''
+</blockquote>
+
+After detecting the sentence boundaries each sentence is written in its own line.
+<blockquote>
+Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.<br>
+Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.<br>
+Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields PLC, was named a director of this British industrial conglomerate.<br>
+</blockquote>
+
+Usually Sentence Detection is done before the text is tokenized and thats the way the pre-trained models on the web site are trained,
+but it is also possible to perform tokenization first and let the Sentence Detector process the already tokenized text.
+
+The OpenNLP Sentence Detector cannot identify sentence boundaries based on the contents of the sentence. A prominent example is
+the first sentence in an article where the title is mistakenly identified to be the first part of the first sentence.
+
+Most components in OpenNLP expect input which is segmented into sentences.
+== Sentence Detector Tool ==
+The easiest way to try out the Sentence Detector is the command line tool. The tool
+is only intended for demonstration and testing.
+
+Download the [http://opennlp.sourceforge.net/models-1.5/en-sent.bin english sentence detector model] and start 
+the Sentence Detector Tool with this command:
+<pre>
+bin/opennlp SentenceDetector en-sent.bin
+</pre>
+Just copy the sample text from above to the console. The Sentence Detector will read it and
+echo one sentence per line to the console.
+
+Usually the input is read from a file and the output is redirected to another file.
+This can be achieved with the following command.
+<pre>
+bin/opennlp SentenceDetector en-sent.bin < input.txt > output.txt
+</pre>
+For the english sentence model from the website the input text should not be tokenized.
+
+== Sentence Detector API ==
+The Sentence Detector can be easily integrated into an application via its API.
+
+To instantiate the Sentence Detector the sentence model must be loaded
+first.
+<pre>
+InputStream modelIn = new FileInputStream("en-sent.bin");
+
+try {
+  SentenceModel model = new SentenceModel(modelIn);
+}
+catch (IOException e) {
+  e.printStackTrace();
+}
+finally {
+  if (modelIn != null) {
+    try {
+      modelIn.close();
+    }
+    catch (IOException e) {
+    }
+  }
+}
+</pre>
+
+After the model is loaded the SentenceDetectorME can be instantiated.
+<pre>
+SentenceDetectorME sentenceDetector = new SentenceDetectorME(model);
+</pre>
+
+The Sentence Detector can output an array of Strings, where each String is one 
+sentence.
+<pre>
+String sentences[] = sentenceDetector.sentDetect("  First sentence. Second sentence. ");
+</pre>
+The result array now contains two entires. The first String is "First sentence." and the second
+String is "Second sentence." The whitespace before, between and after the input String is removed.
+
+The API also offers a method which simply returns the span of the sentence in the input
+string.
+<pre>
+Span sentences[] = sentenceDetector.sentDetect("  First sentence. Second sentence. ");
+</pre>
+The result array again contains two entires. The first span beings at index 2 and ends at 17.
+The second span begins at 18 and ends at 34. The utility method Span.getCoveredText
+can be used to create a substring which only covers the chars in the span.
+
+= Training =
+== Training Tool ==
+OpenNLP has a command line tool which is used to train the models available from the model download page on various corpora.
+The data must be converted to the OpenNLP Sentence Detector training format. Which is one sentence per line. An empty line
+indicates a document boundary. In case the document boundary is unknown, its recommended to have an empty line every
+few ten sentences.
+Exactly like the output in the sample above.
+
+Usage of the tool:
+<pre>
+bin/opennlp SentenceDetectorTrainer
+Usage: opennlp SentenceDetectorTrainer -lang language -encoding charset [-iterations num] [-cutoff num] -data trainingData -model model
+-lang language     specifies the language which is being processed.
+-encoding charset  specifies the encoding which should be used for reading and writing text.
+-iterations num    specified the number of training iterations
+-cutoff num        specifies the min number of times a feature must be seen
+</pre>
+
+To train an english sentence detector use the following command:
+<pre>
+bin/opennlp SentenceDetectorTrainer -encoding UTF-8 -lang en -data en-sent.train -model en-sent.bin
+
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 4883 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 4883 events to 2945.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 2945
+	    Number of Outcomes: 2
+	  Number of Predicates: 467
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-3384.6376826743144	0.38951464263772273
+  2:  .. loglikelihood=-2191.9266688597672	0.9397911120212984
+  3:  .. loglikelihood=-1645.8640771555981	0.9643661683391358
+  4:  .. loglikelihood=-1340.386303774519	0.9739913987302887
+  5:  .. loglikelihood=-1148.4141548519624	0.9748105672742167
+
+ ...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-288.25556805874436	0.9834118369854598
+ 96:  .. loglikelihood=-287.2283680343481	0.9834118369854598
+ 97:  .. loglikelihood=-286.2174830344526	0.9834118369854598
+ 98:  .. loglikelihood=-285.222486981048	0.9834118369854598
+ 99:  .. loglikelihood=-284.24296917223916	0.9834118369854598
+100:  .. loglikelihood=-283.2785335773966	0.9834118369854598
+Wrote sentence detector model.
+Path: en-sent.bin
+</pre>
+
+== Training API ==
+
+The Sentence Detector also offers an API to train a new sentence detection model.
+
+Basically three steps are necessary to train it:
+* The application must open a sample data stream
+* Call the SentenceDetectorME.train method
+* Save the SentenceModel to a file or directly use it
+
+The following sample code illustrates these steps:
+<pre>
+ObjectStream<String> lineStream = new PlainTextByLineStream(new FileInputStream("en-sent.train"), "UTF-8");
+ObjectStream<SentenceSample> sampleStream = new SentenceSampleStream(lineStream);
+
+SentenceModel model = SentenceDetectorME.train("en",sampleStream, true, null, 5, 100);
+
+try {
+  modelOut = new BufferedOutputStream(new FileOutputStream(modelFile));
+  model.serialize(modelOut);
+} finally {
+  if (modelOut != null) 
+     modelOut.close();      
+}
+</pre>
+
+== Custom Feature Generation ==
+TBD
+
+= Evaluation =
+==  Evaluator Tool ==
+The command shows how the evaluator tool can be run:
+<pre>
+$ bin/opennlp SentenceDetectorEvaluator -encoding UTF-8 -model en-sent.bin -data en-sent.eval  
+
+Loading model ... done
+Evaluating ... done
+
+Precision: 0.9465737514518002
+Recall: 0.9095982142857143
+F-Measure: 0.9277177006260672
+</pre>
+The en-sent.eval file has the same format as the training data.

Added: incubator/opennlp/tags/asf_migration/opennlp-docs/Tokenization.wiki
URL: http://svn.apache.org/viewvc/incubator/opennlp/tags/asf_migration/opennlp-docs/Tokenization.wiki?rev=1050126&view=auto
==============================================================================
--- incubator/opennlp/tags/asf_migration/opennlp-docs/Tokenization.wiki (added)
+++ incubator/opennlp/tags/asf_migration/opennlp-docs/Tokenization.wiki Thu Dec 16 19:50:51 2010
@@ -0,0 +1,260 @@
+This page describes how to train and use OpenNLP models for sentence detection and tokenization. The goal is to take text that looks like this:
+
+<blockquote>
+''Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.  Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.  Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields PLC, was named a nonexecutive director of this British industrial conglomerate.  A form of asbestos once used to make Kent cigarette filters has caused a high percentage of cancer deaths among a group of workers exposed to it more than 30 years ago, researchers reported.''
+</blockquote>
+
+And turn it into text that looks like this:
+
+<blockquote>
+Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .<br>
+Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .<br>
+Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC , was named a nonexecutive director of this British industrial conglomerate .<br>
+A form of asbestos once used to make Kent cigarette filters has caused a high percentage of cancer deaths among a group of workers exposed to it more than 30 years ago , researchers reported .<br>
+</blockquote>
+
+Most part-of-speech taggers, parsers and so on, work with text tokenized in this manner. It is important to ensure that your tokenizer produces tokens of the type expected by your later text processing components.
+
+With OpenNLP (as with many systems), tokenization is a two-stage process: first, sentence boundaries are identified, then tokens within each sentence are identified. As such, this tutorial will deal with both.
+
+= Data Preparation =
+
+We'll use the portion of the Penn Treebank that is freely distributed with the NLTK datasets as an example. 
+Download the [http://www.nltk.org/data NLTK data] and go to the treebank directory. The content of the directory should look like this:
+
+<pre>
+$ ls
+chunked  parsed  raw  README
+</pre>
+
+We only need the ''raw'' file here. Here's what it looks like:
+
+<pre>
+$ more raw
+Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
+
+Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
+
+Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC , was na
+med a nonexecutive director of this British industrial conglomerate .
+
+A form of asbestos once used to make Kent cigarette filters has caused a high percentage 
+of cancer deaths among a group of workers exposed to it more than 30 years ago , research
+ers reported .
+</pre>
+
+Of course, this is actually tokenized, not raw. This is an artifact of the process of creating the treebank that seems to be pretty common with other treebank releases. This means we'll need to "reverse" the tokenization -- more on that later.
+
+== Create training and development sets ==
+
+First, we need to split the raw file into a train and dev set so that you can follow along and make sure your trained model gets the same result. We strip out the empty lines and use files with one sentence per line (spl). Then we use 3000 sentence for training and the remaining 896 for development to evaluate the model's performance.
+
+<pre>
+$ grep -v "^$" raw > raw_spl.txt
+$ wc raw_spl.txt 
+  3896  93427 513928 raw_spl.txt
+$ head -3000 raw_spl.txt > train_spl.txt
+$ tail -896 raw_spl.txt > dev_spl.txt
+</pre>
+
+== Create training and development files formatted for model training ==
+
+First, get the [[Detokenizing script]] set-up. After that, the contents of your ''treebank'' directory should look like this:
+
+<pre>
+$ ls
+chunked                   dev_spl.txt  raw          README
+createTokTrainingData.pl  parsed       raw_spl.txt  train_spl.txt
+</pre>
+
+Next, creating training material for sentence detection:
+
+<pre>
+$ ./createTokTrainingData.pl train_spl.txt > train_sentdetect.txt
+$ ./createTokTrainingData.pl dev_spl.txt > dev_sentdetect.txt 
+</pre>
+
+Finally, create training material for tokenization:
+
+<pre>
+$ ./createTokTrainingData.pl train_spl.txt -s > train_tokenize.txt 
+$ ./createTokTrainingData.pl dev_spl.txt -s > dev_tokenize.txt 
+</pre>
+
+Your directory should now look like this:
+
+<pre>
+$ ls
+chunked                   dev_tokenize.txt  raw_spl.txt           train_tokenize.txt
+createTokTrainingData.pl  foo               README
+dev_sentdetect.txt        parsed            train_sentdetect.txt
+dev_spl.txt               raw               train_spl.txt
+</pre>
+
+= Sentence detection =
+
+Train a model and save it as ''sentdetect.model''.
+
+<pre>
+$ opennlp SentenceDetectorTrainer -lang en -encoding UTF-8 -data train_sentdetect.txt -model sentdetect.model
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 4883 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 4883 events to 2945.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 2945
+	    Number of Outcomes: 2
+	  Number of Predicates: 467
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-3384.6376826743144	0.38951464263772273
+  2:  .. loglikelihood=-2191.9266688597672	0.9397911120212984
+  3:  .. loglikelihood=-1645.8640771555981	0.9643661683391358
+  4:  .. loglikelihood=-1340.386303774519	0.9739913987302887
+  5:  .. loglikelihood=-1148.4141548519624	0.9748105672742167
+
+ ...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-288.25556805874436	0.9834118369854598
+ 96:  .. loglikelihood=-287.2283680343481	0.9834118369854598
+ 97:  .. loglikelihood=-286.2174830344526	0.9834118369854598
+ 98:  .. loglikelihood=-285.222486981048	0.9834118369854598
+ 99:  .. loglikelihood=-284.24296917223916	0.9834118369854598
+100:  .. loglikelihood=-283.2785335773966	0.9834118369854598
+Wrote sentence detector model.
+Path: <PATH-TO-TREEBANK>/sentdetect.model
+</pre>
+
+Finally, evaluate it on the development set:
+
+<pre>
+$ opennlp SentenceDetectorEvaluator -encoding UTF-8 -model sentdetect.model -data dev_sentdetect.txt 
+Loading model ... done
+Evaluating ... done
+
+Precision: 0.9465737514518002
+Recall: 0.9095982142857143
+F-Measure: 0.9277177006260672
+</pre>
+
+Of course, you'll likely want to use it on raw text to find sentence boundaries. If you have such text in a file called <code>my_raw_text_file.txt</code>, you need only do:
+
+<pre>
+$ opennlp SentenceDetector sentdetect.model < my_raw_text_file.txt | more
+</pre>
+
+To try it out right away, you can take away all of the breaks in the development file <code>dev_sentdetect.txt</code>:
+
+<pre>
+$ tr '\n' ' ' < dev_sentdetect.txt | perl -pe "s/$/\n\n/" > dev_no_breaks.txt
+</pre>
+
+(The perl call is to pad two newlines to the end of the file, since the SentenceDetector code doesn't seem to want to process something which has just a single line -- probably should be fixed.)
+
+Then, run the sentence detector model on it:
+
+<pre>
+$ opennlp SentenceDetector sentdetect.model < dev_no_breaks.txt | more
+Loading model ... done
+Showa Shell gained 20 to 1,570 and Mitsubishi Oil rose 50 to 1,500.
+Sumitomo Metal Mining fell five yen to 692 and Nippon Mining added 15 to 960.
+Among other winners Wednesday was Nippon Shokubai, which was up 80 at 2,410.
+Marubeni advanced 11 to 890.
+London share prices were bolstered largely by continued gains on Wall Street and technical factors affecting demand for London's blue-chip stocks.
+...etc...
+</pre>
+
+= Tokenization =
+
+Train a model and save it as ''tokenize.model'':
+
+<pre>
+$ opennlp TokenizerTrainer -lang en -encoding UTF-8 -data train_tokenize.txt -model tokenize.model
+Indexing events using cutoff of 5
+
+	Computing event counts...  done. 262271 events
+	Indexing...  done.
+Sorting and merging events... done. Reduced 262271 events to 59060.
+Done indexing.
+Incorporating indexed data for training...  
+done.
+	Number of Event Tokens: 59060
+	    Number of Outcomes: 2
+	  Number of Predicates: 15695
+...done.
+Computing model parameters...
+Performing 100 iterations.
+  1:  .. loglikelihood=-181792.40419263614	0.9614292087192255
+  2:  .. loglikelihood=-34208.094253153664	0.9629238459456059
+  3:  .. loglikelihood=-18784.123872910015	0.9729211388220581
+  4:  .. loglikelihood=-13246.88162585859	0.9856103038460219
+  5:  .. loglikelihood=-10209.262670265718	0.9894422181636552
+
+ ...<skipping a bunch of iterations>...
+
+ 95:  .. loglikelihood=-769.2107474529454	0.999511955191386
+ 96:  .. loglikelihood=-763.8891914534009	0.999511955191386
+ 97:  .. loglikelihood=-758.6685383254891	0.9995157680414533
+ 98:  .. loglikelihood=-753.5458314695236	0.9995157680414533
+ 99:  .. loglikelihood=-748.5182305519613	0.9995157680414533
+100:  .. loglikelihood=-743.5830058068038	0.9995157680414533
+Wrote tokenizer model.
+Path: <PATH-TO-TREEBANK>/tokenize.model
+</pre>
+
+
+Evaluate it on the development set:
+
+<pre>
+$ opennlp TokenizerMEEvaluator -encoding UTF-8 -model tokenize.model -data dev_tokenize.txt 
+Loading model ... done
+Evaluating ... done
+
+Precision: 0.9961179896496408
+Recall: 0.9953222453222453
+F-Measure: 0.9957199585032571
+</pre>
+
+The model is now ready to deal with texts that have been split into sentences. Here's an easy way to get some text to play with:
+
+<pre>
+$ perl -pe "s/<SPLIT>//g" < dev_tokenize.txt > dev_no_splits.txt
+</pre>
+
+Now run the model on it:
+
+<pre>
+$ opennlp TokenizerME tokenize.model < dev_no_splits.txt | more
+Loading model ... done
+Showa Shell gained 20 to 1,570 and Mitsubishi Oil rose 50 to 1,500.
+Sumitomo Metal Mining fell five yen to 692 and Nippon Mining added 15 to 960 .
+Among other winners Wednesday was Nippon Shokubai , which was up 80 at 2,410 .
+Marubeni advanced 11 to 890 .
+London share prices were bolstered largely by continued gains on Wall Street and technica
+l factors affecting demand for London 's blue-chip stocks .
+...etc...
+</pre>
+
+= Sentence splitting and tokenization together =
+
+Since most text comes truly raw and doesn't have sentence boundaries and such, you can pipe the two models just created above together to go from raw texts to ones which have one-sentence per line and are tokenized. 
+
+Recall that we created such a raw file above, called <code>dev_no_breaks.txt</code>. The following will produce 
+
+<pre>
+$ opennlp SentenceDetector sentdetect.model < dev_no_breaks.txt | opennlp TokenizerME tokenize.model | more
+Loading model ... Loading model ... done
+done
+Showa Shell gained 20 to 1,570 and Mitsubishi Oil rose 50 to 1,500.
+Sumitomo Metal Mining fell five yen to 692 and Nippon Mining added 15 to 960 .
+Among other winners Wednesday was Nippon Shokubai , which was up 80 at 2,410 .
+Marubeni advanced 11 to 890 .
+London share prices were bolstered largely by continued gains on Wall Street and technical factors affecting demand for London 's blue-chip stocks .
+...etc...
+</pre>
+
+Of course this is all on the command line. Many people use the models directly in their Java code by creating SentenceDetector and Tokenizer objects and calling their methods as appropriate.