You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by jo...@apache.org on 2011/01/26 12:46:48 UTC
svn commit: r1063682 - in /incubator/opennlp/trunk/opennlp-docs/src/docbkx:
parser.xml sentdetect.xml tokenizer.xml
Author: joern
Date: Wed Jan 26 11:46:48 2011
New Revision: 1063682
URL: http://svn.apache.org/viewvc?rev=1063682&view=rev
Log:
OPENNLP-95 Add java listings should be declared as such
Modified:
incubator/opennlp/trunk/opennlp-docs/src/docbkx/parser.xml
incubator/opennlp/trunk/opennlp-docs/src/docbkx/sentdetect.xml
incubator/opennlp/trunk/opennlp-docs/src/docbkx/tokenizer.xml
Modified: incubator/opennlp/trunk/opennlp-docs/src/docbkx/parser.xml
URL: http://svn.apache.org/viewvc/incubator/opennlp/trunk/opennlp-docs/src/docbkx/parser.xml?rev=1063682&r1=1063681&r2=1063682&view=diff
==============================================================================
--- incubator/opennlp/trunk/opennlp-docs/src/docbkx/parser.xml (original)
+++ incubator/opennlp/trunk/opennlp-docs/src/docbkx/parser.xml Wed Jan 26 11:46:48 2011
@@ -64,7 +64,7 @@ $ bin/opennlp Parser en-parser.bin en-pa
<para>
The Parser can be easily integrated into an application via its API.
To instantiate a Parser the parser model must be loaded first.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
InputStream modelIn = new FileInputStream("en-parser-chunking.bin");
try {
@@ -89,13 +89,13 @@ finally {
insert parser the parser implementation must be chosen correctly.
The factory method will read a type parameter from the model and create
an instance of the corresponding parser implementation.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
Parser parser = ParserFactory.create(model);]]>
</programlisting>
Right now the tree insert parser is still experimental and there is no pre-trained model for it.
The parser expect a whitespace tokenized sentence. A utility method from the command line tool can parse the sentence String. The following code shows how the parser can be called.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
String sentence = "The quick brown fox jumps over the lazy dog .";
Parse topParses[] = ParserTool.parseLine(sentence, parser, 1);]]>
Modified: incubator/opennlp/trunk/opennlp-docs/src/docbkx/sentdetect.xml
URL: http://svn.apache.org/viewvc/incubator/opennlp/trunk/opennlp-docs/src/docbkx/sentdetect.xml?rev=1063682&r1=1063681&r2=1063682&view=diff
==============================================================================
--- incubator/opennlp/trunk/opennlp-docs/src/docbkx/sentdetect.xml (original)
+++ incubator/opennlp/trunk/opennlp-docs/src/docbkx/sentdetect.xml Wed Jan 26 11:46:48 2011
@@ -76,8 +76,8 @@ $bin/opennlp SentenceDetector en-sent.bi
<title>Sentence Detection API</title>
<para>
The Sentence Detector can be easily integrated into an application via its API.
-To instantiate the Sentence Detector the sentence model must be loaded first.
- <programlisting>
+ To instantiate the Sentence Detector the sentence model must be loaded first.
+ <programlisting language="java">
<![CDATA[
InputStream modelIn = new FileInputStream("en-sent.bin");
@@ -98,18 +98,18 @@ finally {
}]]>
</programlisting>
After the model is loaded the SentenceDetectorME can be instantiated.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
SentenceDetectorME sentenceDetector = new SentenceDetectorME(model);]]>
</programlisting>
The Sentence Detector can output an array of Strings, where each String is one sentence.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
String sentences[] = sentenceDetector.sentDetect(" First sentence. Second sentence. ");]]>
</programlisting>
The result array now contains two entires. The first String is "First sentence." and the second String is "Second sentence." The whitespace before, between and after the input String is removed.
The API also offers a method which simply returns the span of the sentence in the input string.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
Span sentences[] = sentenceDetector.sentPosDetect(" First sentence. Second sentence. ");]]>
</programlisting>
@@ -194,7 +194,7 @@ Path: en-sent.bin
</listitem>
</itemizedlist>
The following sample code illustrates these steps:
- <programlisting>
+ <programlisting language="java">
<![CDATA[
ObjectStream<String> lineStream = new PlainTextByLineStream(new FileInputStream("en-sent.train"), "UTF-8");
ObjectStream<SentenceSample> sampleStream = new SentenceSampleStream(lineStream);
@@ -220,7 +220,7 @@ try {
<title>Evaluation Tool</title>
<para>
The command shows how the evaluator tool can be run:
- <programlisting>
+ <programlisting>
<![CDATA[
$bin/opennlp SentenceDetectorEvaluator -encoding UTF-8 -model en-sent.bin -data en-sent.eval
@@ -230,7 +230,7 @@ Evaluating ... done
Precision: 0.9465737514518002
Recall: 0.9095982142857143
F-Measure: 0.9277177006260672]]>
- </programlisting>
+ </programlisting>
The en-sent.eval file has the same format as the training data.
</para>
</section>
Modified: incubator/opennlp/trunk/opennlp-docs/src/docbkx/tokenizer.xml
URL: http://svn.apache.org/viewvc/incubator/opennlp/trunk/opennlp-docs/src/docbkx/tokenizer.xml?rev=1063682&r1=1063681&r2=1063682&view=diff
==============================================================================
--- incubator/opennlp/trunk/opennlp-docs/src/docbkx/tokenizer.xml (original)
+++ incubator/opennlp/trunk/opennlp-docs/src/docbkx/tokenizer.xml Wed Jan 26 11:46:48 2011
@@ -150,7 +150,7 @@ London share prices were bolstered large
To instantiate the TokenizerME (the learnable tokenizer) a Token Model
must be created first. The following code sample shows how a model
can be loaded.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
InputStream modelIn = new FileInputStream("en-token.bin");
@@ -172,7 +172,7 @@ finally {
]]>
</programlisting>
After the model is loaded the TokenizerME can be instantiated.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
Tokenizer tokenizer = new TokenizerME(model);]]>
</programlisting>
@@ -181,7 +181,7 @@ Tokenizer tokenizer = new TokenizerME(mo
should be a sentence, but depending on the training of the learnable
tokenizer this is not required. The first returns an array of
Strings, where each String is one token.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
String tokens[] = tokenizer.tokenize("An input sample sentence.");]]>
</programlisting>
@@ -193,7 +193,7 @@ String tokens[] = tokenizer.tokenize("An
The second method, tokenizePos returns an array of Spans, each Span
contain the begin and end character offsets of the token in the input
String.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
Span tokenSpans[] = tokenizer.tokenizePos("An input sample sentence.");]]>
</programlisting>
@@ -203,7 +203,7 @@ Span tokenSpans[] = tokenizer.tokenizePo
The TokenizerME is able to output the probabilities for the detected
tokens. The getTokenProbabilities method must be called directly
after one of the tokenize methods was called.
- <programlisting>
+ <programlisting language="java">
<![CDATA[
TokenizerME tokenizer = ...