You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucy.apache.org by nw...@apache.org on 2013/06/30 19:42:56 UTC
[lucy-commits] svn commit: r1498162 [1/5] - in /lucy/site/trunk/content/docs/test: ./ Lucy/
Lucy/Analysis/ Lucy/Docs/ Lucy/Docs/Cookbook/ Lucy/Docs/Tutorial/
Lucy/Document/ Lucy/Highlight/ Lucy/Index/ Lucy/Object/ Lucy/Plan/
Lucy/Search/ Lucy/Search/Collector/ Lucy/S...
Author: nwellnhof
Date: Sun Jun 30 17:42:54 2013
New Revision: 1498162
URL: http://svn.apache.org/r1498162
Log:
Another documentation test
Added:
lucy/site/trunk/content/docs/test/Lucy/
lucy/site/trunk/content/docs/test/Lucy/Analysis/
lucy/site/trunk/content/docs/test/Lucy/Analysis/Analyzer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/CaseFolder.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/EasyAnalyzer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/Normalizer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/PolyAnalyzer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/RegexTokenizer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStemmer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStopFilter.mdtext
lucy/site/trunk/content/docs/test/Lucy/Analysis/StandardTokenizer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/
lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/
lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQueryParser.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/DevGuide.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/DocIDs.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/FileFormat.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/FileLocking.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/IRTheory.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/Analysis.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/BeyondSimple.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/FieldType.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/Highlighter.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/QueryObjects.mdtext
lucy/site/trunk/content/docs/test/Lucy/Docs/Tutorial/Simple.mdtext
lucy/site/trunk/content/docs/test/Lucy/Document/
lucy/site/trunk/content/docs/test/Lucy/Document/Doc.mdtext
lucy/site/trunk/content/docs/test/Lucy/Document/HitDoc.mdtext
lucy/site/trunk/content/docs/test/Lucy/Highlight/
lucy/site/trunk/content/docs/test/Lucy/Highlight/Highlighter.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/
lucy/site/trunk/content/docs/test/Lucy/Index/BackgroundMerger.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/DataReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/DataWriter.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/DeletionsWriter.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/DocReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/IndexManager.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/IndexReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/Indexer.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/Lexicon.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/LexiconReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/PolyReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/PostingList.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/PostingListReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/SegReader.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/SegWriter.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/Segment.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/Similarity.mdtext
lucy/site/trunk/content/docs/test/Lucy/Index/Snapshot.mdtext
lucy/site/trunk/content/docs/test/Lucy/Object/
lucy/site/trunk/content/docs/test/Lucy/Object/BitVector.mdtext
lucy/site/trunk/content/docs/test/Lucy/Object/Obj.mdtext
lucy/site/trunk/content/docs/test/Lucy/Plan/
lucy/site/trunk/content/docs/test/Lucy/Plan/Architecture.mdtext
lucy/site/trunk/content/docs/test/Lucy/Plan/BlobType.mdtext
lucy/site/trunk/content/docs/test/Lucy/Plan/FieldType.mdtext
lucy/site/trunk/content/docs/test/Lucy/Plan/FullTextType.mdtext
lucy/site/trunk/content/docs/test/Lucy/Plan/Schema.mdtext
lucy/site/trunk/content/docs/test/Lucy/Plan/StringType.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/
lucy/site/trunk/content/docs/test/Lucy/Search/ANDQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Collector/
lucy/site/trunk/content/docs/test/Lucy/Search/Collector.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Collector/BitCollector.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Compiler.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Hits.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/IndexSearcher.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/LeafQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/MatchAllQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Matcher.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/NOTQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/NoMatchQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/ORQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/PhraseQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/PolyQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/PolySearcher.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Query.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/QueryParser.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/RangeQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/RequiredOptionalQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Searcher.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/SortRule.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/SortSpec.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/Span.mdtext
lucy/site/trunk/content/docs/test/Lucy/Search/TermQuery.mdtext
lucy/site/trunk/content/docs/test/Lucy/Simple.mdtext
lucy/site/trunk/content/docs/test/Lucy/Store/
lucy/site/trunk/content/docs/test/Lucy/Store/FSFolder.mdtext
lucy/site/trunk/content/docs/test/Lucy/Store/Folder.mdtext
lucy/site/trunk/content/docs/test/Lucy/Store/Lock.mdtext
lucy/site/trunk/content/docs/test/Lucy/Store/LockErr.mdtext
lucy/site/trunk/content/docs/test/Lucy/Store/LockFactory.mdtext
lucy/site/trunk/content/docs/test/Lucy/Store/RAMFolder.mdtext
lucy/site/trunk/content/docs/test/LucyX/
lucy/site/trunk/content/docs/test/LucyX/Index/
lucy/site/trunk/content/docs/test/LucyX/Index/ByteBufDocReader.mdtext
lucy/site/trunk/content/docs/test/LucyX/Index/ByteBufDocWriter.mdtext
lucy/site/trunk/content/docs/test/LucyX/Index/LongFieldSim.mdtext
lucy/site/trunk/content/docs/test/LucyX/Index/ZlibDocReader.mdtext
lucy/site/trunk/content/docs/test/LucyX/Index/ZlibDocWriter.mdtext
lucy/site/trunk/content/docs/test/LucyX/Remote/
lucy/site/trunk/content/docs/test/LucyX/Remote/ClusterSearcher.mdtext
lucy/site/trunk/content/docs/test/LucyX/Remote/SearchClient.mdtext
lucy/site/trunk/content/docs/test/LucyX/Remote/SearchServer.mdtext
lucy/site/trunk/content/docs/test/LucyX/Search/
lucy/site/trunk/content/docs/test/LucyX/Search/Filter.mdtext
lucy/site/trunk/content/docs/test/LucyX/Search/MockMatcher.mdtext
lucy/site/trunk/content/docs/test/LucyX/Search/ProximityQuery.mdtext
Modified:
lucy/site/trunk/content/docs/test/Lucy.mdtext
Modified: lucy/site/trunk/content/docs/test/Lucy.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy.mdtext?rev=1498162&r1=1498161&r2=1498162&view=diff
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy.mdtext (original)
+++ lucy/site/trunk/content/docs/test/Lucy.mdtext Sun Jun 30 17:42:54 2013
@@ -1,4 +1,4 @@
-Title: Apache Lucy Documentation - Lucy
+Title: Lucy - Apache Lucy Documentation
<div>
<a name='___top' class='dummyTopAnchor' ></a>
@@ -110,28 +110,28 @@ name="Features"
name="Getting_Started"
>Getting Started</a></h2>
-<p><a href="http://search.cpan.org/perldoc?Lucy%3A%3ASimple" class="podlinkpod"
+<p><a href="./Lucy/Simple.html" class="podlinkpod"
>Lucy::Simple</a> provides a stripped down API which may suffice for many tasks.</p>
-<p><a href="http://search.cpan.org/perldoc?Lucy%3A%3ADocs%3A%3ATutorial" class="podlinkpod"
+<p><a href="./Lucy/Docs/Tutorial.html" class="podlinkpod"
>Lucy::Docs::Tutorial</a> demonstrates how to build a basic CGI search application.</p>
<p>The tutorial spends most of its time on these five classes:</p>
<ul>
-<li><a href="http://search.cpan.org/perldoc?Lucy%3A%3APlan%3A%3ASchema" class="podlinkpod"
+<li><a href="./Lucy/Plan/Schema.html" class="podlinkpod"
>Lucy::Plan::Schema</a> - Plan out your index.</li>
-<li><a href="http://search.cpan.org/perldoc?Lucy%3A%3APlan%3A%3AFieldType" class="podlinkpod"
+<li><a href="./Lucy/Plan/FieldType.html" class="podlinkpod"
>Lucy::Plan::FieldType</a> - Define index fields.</li>
-<li><a href="http://search.cpan.org/perldoc?Lucy%3A%3AIndex%3A%3AIndexer" class="podlinkpod"
+<li><a href="./Lucy/Index/Indexer.html" class="podlinkpod"
>Lucy::Index::Indexer</a> - Manipulate index content.</li>
-<li><a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3AIndexSearcher" class="podlinkpod"
+<li><a href="./Lucy/Search/IndexSearcher.html" class="podlinkpod"
>Lucy::Search::IndexSearcher</a> - Search an index.</li>
-<li><a href="http://search.cpan.org/perldoc?Lucy%3A%3AAnalysis%3A%3AEasyAnalyzer" class="podlinkpod"
+<li><a href="./Lucy/Analysis/EasyAnalyzer.html" class="podlinkpod"
>Lucy::Analysis::EasyAnalyzer</a> - A one-size-fits-all parser/tokenizer.</li>
</ul>
@@ -139,24 +139,24 @@ name="Getting_Started"
name="Delving_Deeper"
>Delving Deeper</a></h2>
-<p><a href="http://search.cpan.org/perldoc?Lucy%3A%3ADocs%3A%3ACookbook" class="podlinkpod"
+<p><a href="./Lucy/Docs/Cookbook.html" class="podlinkpod"
>Lucy::Docs::Cookbook</a> augments the tutorial with more advanced recipes.</p>
-<p>For creating complex queries, see <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3AQuery" class="podlinkpod"
->Lucy::Search::Query</a> and its subclasses <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3ATermQuery" class="podlinkpod"
->TermQuery</a>, <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3APhraseQuery" class="podlinkpod"
->PhraseQuery</a>, <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3AANDQuery" class="podlinkpod"
->ANDQuery</a>, <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3AORQuery" class="podlinkpod"
->ORQuery</a>, <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3ANOTQuery" class="podlinkpod"
->NOTQuery</a>, <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3ARequiredOptionalQuery" class="podlinkpod"
->RequiredOptionalQuery</a>, <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3AMatchAllQuery" class="podlinkpod"
->MatchAllQuery</a>, and <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3ANoMatchQuery" class="podlinkpod"
->NoMatchQuery</a>, plus <a href="http://search.cpan.org/perldoc?Lucy%3A%3ASearch%3A%3AQueryParser" class="podlinkpod"
+<p>For creating complex queries, see <a href="./Lucy/Search/Query.html" class="podlinkpod"
+>Lucy::Search::Query</a> and its subclasses <a href="./Lucy/Search/TermQuery.html" class="podlinkpod"
+>TermQuery</a>, <a href="./Lucy/Search/PhraseQuery.html" class="podlinkpod"
+>PhraseQuery</a>, <a href="./Lucy/Search/ANDQuery.html" class="podlinkpod"
+>ANDQuery</a>, <a href="./Lucy/Search/ORQuery.html" class="podlinkpod"
+>ORQuery</a>, <a href="./Lucy/Search/NOTQuery.html" class="podlinkpod"
+>NOTQuery</a>, <a href="./Lucy/Search/RequiredOptionalQuery.html" class="podlinkpod"
+>RequiredOptionalQuery</a>, <a href="./Lucy/Search/MatchAllQuery.html" class="podlinkpod"
+>MatchAllQuery</a>, and <a href="./Lucy/Search/NoMatchQuery.html" class="podlinkpod"
+>NoMatchQuery</a>, plus <a href="./Lucy/Search/QueryParser.html" class="podlinkpod"
>Lucy::Search::QueryParser</a>.</p>
-<p>For distributed searching, see <a href="http://search.cpan.org/perldoc?LucyX%3A%3ARemote%3A%3ASearchServer" class="podlinkpod"
->LucyX::Remote::SearchServer</a>, <a href="http://search.cpan.org/perldoc?LucyX%3A%3ARemote%3A%3ASearchClient" class="podlinkpod"
->LucyX::Remote::SearchClient</a>, and <a href="http://search.cpan.org/perldoc?LucyX%3A%3ARemote%3A%3AClusterSearcher" class="podlinkpod"
+<p>For distributed searching, see <a href="./LucyX/Remote/SearchServer.html" class="podlinkpod"
+>LucyX::Remote::SearchServer</a>, <a href="./LucyX/Remote/SearchClient.html" class="podlinkpod"
+>LucyX::Remote::SearchClient</a>, and <a href="./LucyX/Remote/ClusterSearcher.html" class="podlinkpod"
>LucyX::Remote::ClusterSearcher</a>.</p>
<h2><a class='u' href='#___top' title='click to go to top of document'
@@ -179,7 +179,7 @@ name="error"
<pre> my $instream = $folder->open_in( file => 'foo' ) or die Clownfish->error;</pre>
-<p>Access a shared variable which is set by some routines on failure. It will always be either a <a href="http://search.cpan.org/perldoc?Clownfish%3A%3AErr" class="podlinkpod"
+<p>Access a shared variable which is set by some routines on failure. It will always be either a <a href="./Clownfish/Err.html" class="podlinkpod"
>Clownfish::Err</a> object or undef.</p>
<h1><a class='u' href='#___top' title='click to go to top of document'
@@ -205,4 +205,5 @@ name="COPYRIGHT"
>COPYRIGHT</a></h1>
<p>Apache Lucy is distributed under the Apache License, Version 2.0, as described in the file <code>LICENSE</code> included with the distribution.</p>
+
</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/Analyzer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/Analyzer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/Analyzer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/Analyzer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,32 @@
+Title: Lucy::Analysis::Analyzer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::Analyzer - Tokenize/modify/filter text.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> # Abstract base class.</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>An Analyzer is a filter which processes text, transforming it from one form into another. For instance, an analyzer might break up a long text into smaller pieces (<a href="../../Lucy/Analysis/RegexTokenizer.html" class="podlinkpod"
+>RegexTokenizer</a>), or it might perform case folding to facilitate case-insensitive search (<a href="../../Lucy/Analysis/Normalizer.html" class="podlinkpod"
+>Normalizer</a>).</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::Analyzer isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/CaseFolder.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/CaseFolder.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/CaseFolder.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/CaseFolder.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,51 @@
+Title: Lucy::Analysis::CaseFolder - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::CaseFolder - Normalize case,
+facilitating case-insensitive search.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $case_folder = Lucy::Analysis::CaseFolder->new;
+
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $case_folder, $stemmer ],
+ );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>CaseFolder is DEPRECATED. Use <a href="../../Lucy/Analysis/Normalizer.html" class="podlinkpod"
+>Normalizer</a> instead.</p>
+
+<p>CaseFolder normalizes text according to Unicode case-folding rules, so that searches will be case-insensitive.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new()"
+>new()</a></h2>
+
+<pre> my $case_folder = Lucy::Analysis::CaseFolder->new;</pre>
+
+<p>Constructor. Takes no arguments.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::CaseFolder isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/EasyAnalyzer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/EasyAnalyzer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/EasyAnalyzer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/EasyAnalyzer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,76 @@
+Title: Lucy::Analysis::EasyAnalyzer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::EasyAnalyzer - A simple analyzer chain.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $schema = Lucy::Plan::Schema->new;
+ my $analyzer = Lucy::Analysis::EasyAnalyzer->new(
+ language => 'en',
+ );
+ my $type = Lucy::Plan::FullTextType->new(
+ analyzer => $analyzer,
+ );
+ $schema->spec_field( name => 'title', type => $type );
+ $schema->spec_field( name => 'content', type => $type );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>EasyAnalyzer is an analyzer chain consisting of a <a href="../../Lucy/Analysis/StandardTokenizer.html" class="podlinkpod"
+>StandardTokenizer</a>, a <a href="../../Lucy/Analysis/Normalizer.html" class="podlinkpod"
+>Normalizer</a>, and a <a href="../../Lucy/Analysis/SnowballStemmer.html" class="podlinkpod"
+>SnowballStemmer</a>.</p>
+
+<p>Supported languages:</p>
+
+<pre> en => English,
+ da => Danish,
+ de => German,
+ es => Spanish,
+ fi => Finnish,
+ fr => French,
+ hu => Hungarian,
+ it => Italian,
+ nl => Dutch,
+ no => Norwegian,
+ pt => Portuguese,
+ ro => Romanian,
+ ru => Russian,
+ sv => Swedish,
+ tr => Turkish,</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new(_[labeled_params]_)"
+>new( <i>[labeled params]</i> )</a></h2>
+
+<pre> my $analyzer = Lucy::Analysis::EasyAnalyzer->new(
+ language => 'es',
+ );</pre>
+
+<ul>
+<li><b>language</b> - An ISO code from the list of supported languages.</li>
+</ul>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::EasyAnalyzer isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/Normalizer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/Normalizer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/Normalizer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/Normalizer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,61 @@
+Title: Lucy::Analysis::Normalizer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::Normalizer - Unicode normalization,
+case folding and accent stripping.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $normalizer = Lucy::Analysis::Normalizer->new;
+
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stemmer ],
+ );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>Normalizer is an <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Analyzer</a> which normalizes tokens to one of the Unicode normalization forms. Optionally, it performs Unicode case folding and converts accented characters to their base character.</p>
+
+<p>If you use highlighting, Normalizer should be run after tokenization because it might add or remove characters.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new(_[labeled_params]_)"
+>new( <i>[labeled params]</i> )</a></h2>
+
+<pre> my $normalizer = Lucy::Analysis::Normalizer->new(
+ normalization_form => 'NFKC',
+ case_fold => 1,
+ strip_accents => 0,
+ );</pre>
+
+<ul>
+<li><b>normalization_form</b> - Unicode normalization form, can be one of 'NFC', 'NFKC', 'NFD', 'NFKD'. Defaults to 'NFKC'.</li>
+
+<li><b>case_fold</b> - Perform case folding, default is true.</li>
+
+<li><b>strip_accents</b> - Strip accents, default is false.</li>
+</ul>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::Normalizer isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/PolyAnalyzer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/PolyAnalyzer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/PolyAnalyzer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/PolyAnalyzer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,95 @@
+Title: Lucy::Analysis::PolyAnalyzer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::PolyAnalyzer - Multiple Analyzers in series.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $schema = Lucy::Plan::Schema->new;
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => \@analyzers,
+ );
+ my $type = Lucy::Plan::FullTextType->new(
+ analyzer => $polyanalyzer,
+ );
+ $schema->spec_field( name => 'title', type => $type );
+ $schema->spec_field( name => 'content', type => $type );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>A PolyAnalyzer is a series of <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Analyzers</a>, each of which will be called upon to "analyze" text in turn. You can either provide the Analyzers yourself, or you can specify a supported language, in which case a PolyAnalyzer consisting of a <a href="../../Lucy/Analysis/CaseFolder.html" class="podlinkpod"
+>CaseFolder</a>, a <a href="../../Lucy/Analysis/RegexTokenizer.html" class="podlinkpod"
+>RegexTokenizer</a>, and a <a href="../../Lucy/Analysis/SnowballStemmer.html" class="podlinkpod"
+>SnowballStemmer</a> will be generated for you.</p>
+
+<p>The language parameter is DEPRECATED. Use <a href="../../Lucy/Analysis/EasyAnalyzer.html" class="podlinkpod"
+>EasyAnalyzer</a> instead.</p>
+
+<p>Supported languages:</p>
+
+<pre> en => English,
+ da => Danish,
+ de => German,
+ es => Spanish,
+ fi => Finnish,
+ fr => French,
+ hu => Hungarian,
+ it => Italian,
+ nl => Dutch,
+ no => Norwegian,
+ pt => Portuguese,
+ ro => Romanian,
+ ru => Russian,
+ sv => Swedish,
+ tr => Turkish,</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new(_[labeled_params]_)"
+>new( <i>[labeled params]</i> )</a></h2>
+
+<pre> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
+ my $normalizer = Lucy::Analysis::Normalizer->new;
+ my $stemmer = Lucy::Analysis::SnowballStemmer->new( language => 'en' );
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stemmer, ], );</pre>
+
+<ul>
+<li><b>language</b> - An ISO code from the list of supported languages. DEPRECATED, use <a href="../../Lucy/Analysis/EasyAnalyzer.html" class="podlinkpod"
+>EasyAnalyzer</a> instead.</li>
+
+<li><b>analyzers</b> - An array of Analyzers. The order of the analyzers matters. Don't put a SnowballStemmer before a RegexTokenizer (can't stem whole documents or paragraphs -- just individual words), or a SnowballStopFilter after a SnowballStemmer (stemmed words, e.g. "themselv", will not appear in a stoplist). In general, the sequence should be: tokenize, normalize, stopalize, stem.</li>
+</ul>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="METHODS"
+>METHODS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="get_analyzers()"
+>get_analyzers()</a></h2>
+
+<p>Getter for "analyzers" member.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::PolyAnalyzer isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/RegexTokenizer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/RegexTokenizer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/RegexTokenizer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/RegexTokenizer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,79 @@
+Title: Lucy::Analysis::RegexTokenizer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::RegexTokenizer - Split a string into tokens.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $whitespace_tokenizer
+ = Lucy::Analysis::RegexTokenizer->new( pattern => '\S+' );
+
+ # or...
+ my $word_char_tokenizer
+ = Lucy::Analysis::RegexTokenizer->new( pattern => '\w+' );
+
+ # or...
+ my $apostrophising_tokenizer = Lucy::Analysis::RegexTokenizer->new;
+
+ # Then... once you have a tokenizer, put it into a PolyAnalyzer:
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $word_char_tokenizer, $normalizer, $stemmer ], );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>Generically, "tokenizing" is a process of breaking up a string into an array of "tokens". For instance, the string "three blind mice" might be tokenized into "three", "blind", "mice".</p>
+
+<p>Lucy::Analysis::RegexTokenizer decides where it should break up the text based on a regular expression compiled from a supplied <code>pattern</code> matching one token. If our source string is...</p>
+
+<pre> "Eats, Shoots and Leaves."</pre>
+
+<p>... then a "whitespace tokenizer" with a <code>pattern</code> of <code>"\\S+"</code> produces...</p>
+
+<pre> Eats,
+ Shoots
+ and
+ Leaves.</pre>
+
+<p>... while a "word character tokenizer" with a <code>pattern</code> of <code>"\\w+"</code> produces...</p>
+
+<pre> Eats
+ Shoots
+ and
+ Leaves</pre>
+
+<p>... the difference being that the word character tokenizer skips over punctuation as well as whitespace when determining token boundaries.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new(_[labeled_params]_)"
+>new( <i>[labeled params]</i> )</a></h2>
+
+<pre> my $word_char_tokenizer = Lucy::Analysis::RegexTokenizer->new(
+ pattern => '\w+', # required
+ );</pre>
+
+<ul>
+<li><b>pattern</b> - A string specifying a Perl-syntax regular expression which should match one token. The default value is <code>\w+(?:[\x{2019}']\w+)*</code>, which matches "it's" as well as "it" and "O'Henry's" as well as "Henry".</li>
+</ul>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::RegexTokenizer isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStemmer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStemmer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStemmer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStemmer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,52 @@
+Title: Lucy::Analysis::SnowballStemmer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::SnowballStemmer - Reduce related words to a shared root.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $stemmer = Lucy::Analysis::SnowballStemmer->new( language => 'es' );
+
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stemmer ],
+ );</pre>
+
+<p>This class is a wrapper around the Snowball stemming library, so it supports the same languages.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>SnowballStemmer is an <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Analyzer</a> which reduces related words to a root form (using the "Snowball" stemming library). For instance, "horse", "horses", and "horsing" all become "hors" -- so that a search for 'horse' will also match documents containing 'horses' and 'horsing'.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new(_[labeled_params]_)"
+>new( <i>[labeled params]</i> )</a></h2>
+
+<pre> my $stemmer = Lucy::Analysis::SnowballStemmer->new( language => 'es' );</pre>
+
+<ul>
+<li><b>language</b> - A two-letter ISO code identifying a language supported by Snowball.</li>
+</ul>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::SnowballStemmer isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStopFilter.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStopFilter.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStopFilter.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/SnowballStopFilter.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,87 @@
+Title: Lucy::Analysis::SnowballStopFilter - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::SnowballStopFilter - Suppress a "stoplist" of common words.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $stopfilter = Lucy::Analysis::SnowballStopFilter->new(
+ language => 'fr',
+ );
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stopfilter, $stemmer ],
+ );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>A "stoplist" is collection of "stopwords": words which are common enough to be of little value when determining search results. For example, so many documents in English contain "the", "if", and "maybe" that it may improve both performance and relevance to block them.</p>
+
+<p>Before filtering stopwords:</p>
+
+<pre> ("i", "am", "the", "walrus")</pre>
+
+<p>After filtering stopwords:</p>
+
+<pre> ("walrus")</pre>
+
+<p>SnowballStopFilter provides default stoplists for several languages, courtesy of the Snowball project (<http://snowball.tartarus.org>), or you may supply your own.</p>
+
+<pre> |-----------------------|
+ | ISO CODE | LANGUAGE |
+ |-----------------------|
+ | da | Danish |
+ | de | German |
+ | en | English |
+ | es | Spanish |
+ | fi | Finnish |
+ | fr | French |
+ | hu | Hungarian |
+ | it | Italian |
+ | nl | Dutch |
+ | no | Norwegian |
+ | pt | Portuguese |
+ | sv | Swedish |
+ | ru | Russian |
+ |-----------------------|</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new(_[labeled_params]_)"
+>new( <i>[labeled params]</i> )</a></h2>
+
+<pre> my $stopfilter = Lucy::Analysis::SnowballStopFilter->new(
+ language => 'de',
+ );
+
+ # or...
+ my $stopfilter = Lucy::Analysis::SnowballStopFilter->new(
+ stoplist => \%stoplist,
+ );</pre>
+
+<ul>
+<li><b>stoplist</b> - A hash with stopwords as the keys.</li>
+
+<li><b>language</b> - The ISO code for a supported language.</li>
+</ul>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::SnowballStopFilter isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Analysis/StandardTokenizer.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Analysis/StandardTokenizer.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Analysis/StandardTokenizer.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Analysis/StandardTokenizer.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,49 @@
+Title: Lucy::Analysis::StandardTokenizer - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Analysis::StandardTokenizer - Split a string into tokens.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="SYNOPSIS"
+>SYNOPSIS</a></h1>
+
+<pre> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;
+
+ # Then... once you have a tokenizer, put it into a PolyAnalyzer:
+ my $polyanalyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [ $tokenizer, $normalizer, $stemmer ], );</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>Generically, "tokenizing" is a process of breaking up a string into an array of "tokens". For instance, the string "three blind mice" might be tokenized into "three", "blind", "mice".</p>
+
+<p>Lucy::Analysis::StandardTokenizer breaks up the text at the word boundaries defined in Unicode Standard Annex #29. It then returns those words that start with an alphabetic or numeric character.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="CONSTRUCTORS"
+>CONSTRUCTORS</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="new()"
+>new()</a></h2>
+
+<pre> my $tokenizer = Lucy::Analysis::StandardTokenizer->new;</pre>
+
+<p>Constructor. Takes no arguments.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="INHERITANCE"
+>INHERITANCE</a></h1>
+
+<p>Lucy::Analysis::StandardTokenizer isa <a href="../../Lucy/Analysis/Analyzer.html" class="podlinkpod"
+>Lucy::Analysis::Analyzer</a> isa Clownfish::Obj.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,52 @@
+Title: Lucy::Docs::Cookbook - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::Cookbook - Apache Lucy recipes.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>The Cookbook provides thematic documentation covering some of Apache Lucy's more sophisticated features.
+For a step-by-step introduction to Lucy,
+see <a href="../../Lucy/Docs/Tutorial.html" class="podlinkpod"
+>Lucy::Docs::Tutorial</a>.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Chapters"
+>Chapters</a></h2>
+
+<ul>
+<li><a href="../../Lucy/Docs/Cookbook/FastUpdates.html" class="podlinkpod"
+>Lucy::Docs::Cookbook::FastUpdates</a> - While index updates are fast on average,
+worst-case update performance may be significantly slower.
+To make index updates consistently quick,
+we must manually intervene to control the process of index segment consolidation.</li>
+
+<li><a href="../../Lucy/Docs/Cookbook/CustomQuery.html" class="podlinkpod"
+>Lucy::Docs::Cookbook::CustomQuery</a> - Explore Lucy's support for custom query types by creating a "PrefixQuery" class to handle trailing wildcards.</li>
+
+<li><a href="../../Lucy/Docs/Cookbook/CustomQueryParser.html" class="podlinkpod"
+>Lucy::Docs::Cookbook::CustomQueryParser</a> - Define your own custom search query syntax using Lucy::Search::QueryParser and <a href="../../Parse/RecDescent.html" class="podlinkpod"
+>Parse::RecDescent</a>.</li>
+</ul>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Materials"
+>Materials</a></h2>
+
+<p>Some of the recipes in the Cookbook reference the completed <a href="../../Lucy/Docs/Tutorial.html" class="podlinkpod"
+>Tutorial</a> application.
+These materials can be found in the <code>sample</code> directory at the root of the Lucy distribution:</p>
+
+<pre> sample/indexer.pl # indexing app
+ sample/search.cgi # search app
+ sample/us_constitution # corpus</pre>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQuery.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQuery.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQuery.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQuery.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,286 @@
+Title: Lucy::Docs::Cookbook::CustomQuery - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::Cookbook::CustomQuery - Sample subclass of Query.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="ABSTRACT"
+>ABSTRACT</a></h1>
+
+<p>Explore Apache Lucy's support for custom query types by creating a "PrefixQuery" class to handle trailing wildcards.</p>
+
+<pre> my $prefix_query = PrefixQuery->new(
+ field => 'content',
+ query_string => 'foo*',
+ );
+ my $hits = $searcher->hits( query => $prefix_query );
+ ...</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Query,_Compiler,_and_Matcher"
+>Query, Compiler, and Matcher</a></h1>
+
+<p>To add support for a new query type, we need three classes: a Query, a Compiler, and a Matcher.</p>
+
+<ul>
+<li>PrefixQuery - a subclass of <a href="../../../Lucy/Search/Query.html" class="podlinkpod"
+>Lucy::Search::Query</a>, and the only class that client code will deal with directly.</li>
+
+<li>PrefixCompiler - a subclass of <a href="../../../Lucy/Search/Compiler.html" class="podlinkpod"
+>Lucy::Search::Compiler</a>, whose primary role is to compile a PrefixQuery to a PrefixMatcher.</li>
+
+<li>PrefixMatcher - a subclass of <a href="../../../Lucy/Search/Matcher.html" class="podlinkpod"
+>Lucy::Search::Matcher</a>, which does the heavy lifting: it applies the query to individual documents and assigns a score to each match.</li>
+</ul>
+
+<p>The PrefixQuery class on its own isn't enough because a Query object's role is limited to expressing an abstract specification for the search. A Query is basically nothing but metadata; execution is left to the Query's companion Compiler and Matcher.</p>
+
+<p>Here's a simplified sketch illustrating how a Searcher's hits() method ties together the three classes.</p>
+
+<pre> sub hits {
+ my ( $self, $query ) = @_;
+ my $compiler = $query->make_compiler( searcher => $self );
+ my $matcher = $compiler->make_matcher(
+ reader => $self->get_reader,
+ need_score => 1,
+ );
+ my @hits = $matcher->capture_hits;
+ return \@hits;
+ }</pre>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="PrefixQuery"
+>PrefixQuery</a></h2>
+
+<p>Our PrefixQuery class will have two attributes: a query string and a field name.</p>
+
+<pre> package PrefixQuery;
+ use base qw( Lucy::Search::Query );
+ use Carp;
+ use Scalar::Util qw( blessed );
+
+ # Inside-out member vars and hand-rolled accessors.
+ my %query_string;
+ my %field;
+ sub get_query_string { my $self = shift; return $query_string{$$self} }
+ sub get_field { my $self = shift; return $field{$$self} }</pre>
+
+<p>PrefixQuery's constructor collects and validates the attributes.</p>
+
+<pre> sub new {
+ my ( $class, %args ) = @_;
+ my $query_string = delete $args{query_string};
+ my $field = delete $args{field};
+ my $self = $class->SUPER::new(%args);
+ confess("'query_string' param is required")
+ unless defined $query_string;
+ confess("Invalid query_string: '$query_string'")
+ unless $query_string =~ /\*\s*$/;
+ confess("'field' param is required")
+ unless defined $field;
+ $query_string{$$self} = $query_string;
+ $field{$$self} = $field;
+ return $self;
+ }</pre>
+
+<p>Since this is an inside-out class, we'll need a destructor:</p>
+
+<pre> sub DESTROY {
+ my $self = shift;
+ delete $query_string{$$self};
+ delete $field{$$self};
+ $self->SUPER::DESTROY;
+ }</pre>
+
+<p>The equals() method determines whether two Queries are logically equivalent:</p>
+
+<pre> sub equals {
+ my ( $self, $other ) = @_;
+ return 0 unless blessed($other);
+ return 0 unless $other->isa("PrefixQuery");
+ return 0 unless $field{$$self} eq $field{$$other};
+ return 0 unless $query_string{$$self} eq $query_string{$$other};
+ return 1;
+ }</pre>
+
+<p>The last thing we'll need is a make_compiler() factory method which kicks out a subclass of <a href="../../../Lucy/Search/Compiler.html" class="podlinkpod"
+>Compiler</a>.</p>
+
+<pre> sub make_compiler {
+ my ( $self, %args ) = @_;
+ my $subordinate = delete $args{subordinate};
+ my $compiler = PrefixCompiler->new( %args, parent => $self );
+ $compiler->normalize unless $subordinate;
+ return $compiler;
+ }</pre>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="PrefixCompiler"
+>PrefixCompiler</a></h2>
+
+<p>PrefixQuery's make_compiler() method will be called internally at search-time by objects which subclass <a href="../../../Lucy/Search/Searcher.html" class="podlinkpod"
+>Lucy::Search::Searcher</a> -- such as <a href="../../../Lucy/Search/IndexSearcher.html" class="podlinkpod"
+>IndexSearchers</a>.</p>
+
+<p>A Searcher is associated with a particular collection of documents. These documents may all reside in one index, as with IndexSearcher, or they may be spread out across multiple indexes on one or more machines, as with <a href="../../../LucyX/Remote/ClusterSearcher.html" class="podlinkpod"
+>LucyX::Remote::ClusterSearcher</a>.</p>
+
+<p>Searcher objects have access to certain statistical information about the collections they represent; for instance, a Searcher can tell you how many documents are in the collection...</p>
+
+<pre> my $maximum_number_of_docs_in_collection = $searcher->doc_max;</pre>
+
+<p>... or how many documents a specific term appears in:</p>
+
+<pre> my $term_appears_in_this_many_docs = $searcher->doc_freq(
+ field => 'content',
+ term => 'foo',
+ );</pre>
+
+<p>Such information can be used by sophisticated Compiler implementations to assign more or less heft to individual queries or sub-queries. However, we're not going to bother with weighting for this demo; we'll just assign a fixed score of 1.0 to each matching document.</p>
+
+<p>We don't need to write a constructor, as it will suffice to inherit new() from Lucy::Search::Compiler. The only method we need to implement for PrefixCompiler is make_matcher().</p>
+
+<pre> package PrefixCompiler;
+ use base qw( Lucy::Search::Compiler );
+
+ sub make_matcher {
+ my ( $self, %args ) = @_;
+ my $seg_reader = $args{reader};
+
+ # Retrieve low-level components LexiconReader and PostingListReader.
+ my $lex_reader
+ = $seg_reader->obtain("Lucy::Index::LexiconReader");
+ my $plist_reader
+ = $seg_reader->obtain("Lucy::Index::PostingListReader");
+
+ # Acquire a Lexicon and seek it to our query string.
+ my $substring = $self->get_parent->get_query_string;
+ $substring =~ s/\*.\s*$//;
+ my $field = $self->get_parent->get_field;
+ my $lexicon = $lex_reader->lexicon( field => $field );
+ return unless $lexicon;
+ $lexicon->seek($substring);
+
+ # Accumulate PostingLists for each matching term.
+ my @posting_lists;
+ while ( defined( my $term = $lexicon->get_term ) ) {
+ last unless $term =~ /^\Q$substring/;
+ my $posting_list = $plist_reader->posting_list(
+ field => $field,
+ term => $term,
+ );
+ if ($posting_list) {
+ push @posting_lists, $posting_list;
+ }
+ last unless $lexicon->next;
+ }
+ return unless @posting_lists;
+
+ return PrefixMatcher->new( posting_lists => \@posting_lists );
+ }</pre>
+
+<p>PrefixCompiler gets access to a <a href="../../../Lucy/Index/SegReader.html" class="podlinkpod"
+>SegReader</a> object when make_matcher() gets called. From the SegReader and its sub-components <a href="../../../Lucy/Index/LexiconReader.html" class="podlinkpod"
+>LexiconReader</a> and <a href="../../../Lucy/Index/PostingListReader.html" class="podlinkpod"
+>PostingListReader</a>, we acquire a <a href="../../../Lucy/Index/Lexicon.html" class="podlinkpod"
+>Lexicon</a>, scan through the Lexicon's unique terms, and acquire a <a href="../../../Lucy/Index/PostingList.html" class="podlinkpod"
+>PostingList</a> for each term that matches our prefix.</p>
+
+<p>Each of these PostingList objects represents a set of documents which match the query.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="PrefixMatcher"
+>PrefixMatcher</a></h2>
+
+<p>The Matcher subclass is the most involved.</p>
+
+<pre> package PrefixMatcher;
+ use base qw( Lucy::Search::Matcher );
+
+ # Inside-out member vars.
+ my %doc_ids;
+ my %tick;
+
+ sub new {
+ my ( $class, %args ) = @_;
+ my $posting_lists = delete $args{posting_lists};
+ my $self = $class->SUPER::new(%args);
+
+ # Cheesy but simple way of interleaving PostingList doc sets.
+ my %all_doc_ids;
+ for my $posting_list (@$posting_lists) {
+ while ( my $doc_id = $posting_list->next ) {
+ $all_doc_ids{$doc_id} = undef;
+ }
+ }
+ my @doc_ids = sort { $a <=> $b } keys %all_doc_ids;
+ $doc_ids{$$self} = \@doc_ids;
+
+ # Track our position within the array of doc ids.
+ $tick{$$self} = -1;
+
+ return $self;
+ }
+
+ sub DESTROY {
+ my $self = shift;
+ delete $doc_ids{$$self};
+ delete $tick{$$self};
+ $self->SUPER::DESTROY;
+ }</pre>
+
+<p>The doc ids must be in order, or some will be ignored; hence the <code>sort</code> above.</p>
+
+<p>In addition to the constructor and destructor, there are three methods that must be overridden.</p>
+
+<p>next() advances the Matcher to the next valid matching doc.</p>
+
+<pre> sub next {
+ my $self = shift;
+ my $doc_ids = $doc_ids{$$self};
+ my $tick = ++$tick{$$self};
+ return 0 if $tick >= scalar @$doc_ids;
+ return $doc_ids->[$tick];
+ }</pre>
+
+<p>get_doc_id() returns the current document id, or 0 if the Matcher is exhausted. (<a href="../../../Lucy/Docs/DocIDs.html" class="podlinkpod"
+>Document numbers</a> start at 1, so 0 is a sentinel.)</p>
+
+<pre> sub get_doc_id {
+ my $self = shift;
+ my $tick = $tick{$$self};
+ my $doc_ids = $doc_ids{$$self};
+ return $tick < scalar @$doc_ids ? $doc_ids->[$tick] : 0;
+ }</pre>
+
+<p>score() conveys the relevance score of the current match. We'll just return a fixed score of 1.0:</p>
+
+<pre> sub score { 1.0 }</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Usage"
+>Usage</a></h1>
+
+<p>To get a basic feel for PrefixQuery, insert the FlatQueryParser module described in <a href="../../../Lucy/Docs/Cookbook/CustomQueryParser.html" class="podlinkpod"
+>Lucy::Docs::Cookbook::CustomQueryParser</a> (which supports PrefixQuery) into the search.cgi sample app.</p>
+
+<pre> my $parser = FlatQueryParser->new( schema => $searcher->get_schema );
+ my $query = $parser->parse($q);</pre>
+
+<p>If you're planning on using PrefixQuery in earnest, though, you may want to change up analyzers to avoid stemming, because stemming -- another approach to prefix conflation -- is not perfectly compatible with prefix searches.</p>
+
+<pre> # Polyanalyzer with no SnowballStemmer.
+ my $analyzer = Lucy::Analysis::PolyAnalyzer->new(
+ analyzers => [
+ Lucy::Analysis::StandardTokenizer->new,
+ Lucy::Analysis::Normalizer->new,
+ ],
+ );</pre>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQueryParser.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQueryParser.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQueryParser.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/CustomQueryParser.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,212 @@
+Title: Lucy::Docs::Cookbook::CustomQueryParser - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::Cookbook::CustomQueryParser - Sample subclass of QueryParser.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="ABSTRACT"
+>ABSTRACT</a></h1>
+
+<p>Implement a custom search query language using a subclass of <a href="../../../Lucy/Search/QueryParser.html" class="podlinkpod"
+>Lucy::Search::QueryParser</a>.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="The_language"
+>The language</a></h1>
+
+<p>At first,
+our query language will support only simple term queries and phrases delimited by double quotes.
+For simplicity's sake,
+it will not support parenthetical groupings,
+boolean operators,
+or prepended plus/minus.
+The results for all subqueries will be unioned together -- i.e.
+joined using an OR -- which is usually the best approach for small-to-medium-sized document collections.</p>
+
+<p>Later,
+we'll add support for trailing wildcards.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Single-field_parser"
+>Single-field parser</a></h1>
+
+<p>Our initial parser implentation will generate queries against a single fixed field,
+"content",
+and it will analyze text using a fixed choice of English EasyAnalyzer.
+We won't subclass Lucy::Search::QueryParser just yet.</p>
+
+<pre> package FlatQueryParser;
+ use Lucy::Search::TermQuery;
+ use Lucy::Search::PhraseQuery;
+ use Lucy::Search::ORQuery;
+ use Carp;
+
+ sub new {
+ my $analyzer = Lucy::Analysis::EasyAnalyzer->new(
+ language => 'en',
+ );
+ return bless {
+ field => 'content',
+ analyzer => $analyzer,
+ }, __PACKAGE__;
+ }</pre>
+
+<p>Some private helper subs for creating TermQuery and PhraseQuery objects will help keep the size of our main parse() subroutine down:</p>
+
+<pre> sub _make_term_query {
+ my ( $self, $term ) = @_;
+ return Lucy::Search::TermQuery->new(
+ field => $self->{field},
+ term => $term,
+ );
+ }
+
+ sub _make_phrase_query {
+ my ( $self, $terms ) = @_;
+ return Lucy::Search::PhraseQuery->new(
+ field => $self->{field},
+ terms => $terms,
+ );
+ }</pre>
+
+<p>Our private _tokenize() method treats double-quote delimited material as a single token and splits on whitespace everywhere else.</p>
+
+<pre> sub _tokenize {
+ my ( $self, $query_string ) = @_;
+ my @tokens;
+ while ( length $query_string ) {
+ if ( $query_string =~ s/^\s+// ) {
+ next; # skip whitespace
+ }
+ elsif ( $query_string =~ s/^("[^"]*(?:"|$))// ) {
+ push @tokens, $1; # double-quoted phrase
+ }
+ else {
+ $query_string =~ s/(\S+)//;
+ push @tokens, $1; # single word
+ }
+ }
+ return \@tokens;
+ }</pre>
+
+<p>The main parsing routine creates an array of tokens by calling _tokenize(), runs the tokens through through the EasyAnalyzer, creates TermQuery or PhraseQuery objects according to how many tokens emerge from the EasyAnalyzer's split() method, and adds each of the sub-queries to the primary ORQuery.</p>
+
+<pre> sub parse {
+ my ( $self, $query_string ) = @_;
+ my $tokens = $self->_tokenize($query_string);
+ my $analyzer = $self->{analyzer};
+ my $or_query = Lucy::Search::ORQuery->new;
+
+ for my $token (@$tokens) {
+ if ( $token =~ s/^"// ) {
+ $token =~ s/"$//;
+ my $terms = $analyzer->split($token);
+ my $query = $self->_make_phrase_query($terms);
+ $or_query->add_child($phrase_query);
+ }
+ else {
+ my $terms = $analyzer->split($token);
+ if ( @$terms == 1 ) {
+ my $query = $self->_make_term_query( $terms->[0] );
+ $or_query->add_child($query);
+ }
+ elsif ( @$terms > 1 ) {
+ my $query = $self->_make_phrase_query($terms);
+ $or_query->add_child($query);
+ }
+ }
+ }
+
+ return $or_query;
+ }</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Multi-field_parser"
+>Multi-field parser</a></h1>
+
+<p>Most often, the end user will want their search query to match not only a single 'content' field, but also 'title' and so on. To make that happen, we have to turn queries such as this...</p>
+
+<pre> foo AND NOT bar</pre>
+
+<p>... into the logical equivalent of this:</p>
+
+<pre> (title:foo OR content:foo) AND NOT (title:bar OR content:bar)</pre>
+
+<p>Rather than continue with our own from-scratch parser class and write the routines to accomplish that expansion, we're now going to subclass Lucy::Search::QueryParser and take advantage of some of its existing methods.</p>
+
+<p>Our first parser implementation had the "content" field name and the choice of English EasyAnalyzer hard-coded for simplicity, but we don't need to do that once we subclass Lucy::Search::QueryParser. QueryParser's constructor -- which we will inherit, allowing us to eliminate our own constructor -- requires a Schema which conveys field and Analyzer information, so we can just defer to that.</p>
+
+<pre> package FlatQueryParser;
+ use base qw( Lucy::Search::QueryParser );
+ use Lucy::Search::TermQuery;
+ use Lucy::Search::PhraseQuery;
+ use Lucy::Search::ORQuery;
+ use PrefixQuery;
+ use Carp;
+
+ # Inherit new()</pre>
+
+<p>We're also going to jettison our _make_term_query() and _make_phrase_query() helper subs and chop our parse() subroutine way down. Our revised parse() routine will generate Lucy::Search::LeafQuery objects instead of TermQueries and PhraseQueries:</p>
+
+<pre> sub parse {
+ my ( $self, $query_string ) = @_;
+ my $tokens = $self->_tokenize($query_string);
+ my $or_query = Lucy::Search::ORQuery->new;
+ for my $token (@$tokens) {
+ my $leaf_query = Lucy::Search::LeafQuery->new( text => $token );
+ $or_query->add_child($leaf_query);
+ }
+ return $self->expand($or_query);
+ }</pre>
+
+<p>The magic happens in QueryParser's expand() method, which walks the ORQuery object we supply to it looking for LeafQuery objects, and calls expand_leaf() for each one it finds. expand_leaf() performs field-specific analysis, decides whether each query should be a TermQuery or a PhraseQuery, and if multiple fields are required, creates an ORQuery which mults out e.g. <code>foo</code> into <code>(title:foo OR content:foo)</code>.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Extending_the_query_language"
+>Extending the query language</a></h1>
+
+<p>To add support for trailing wildcards to our query language, we need to override expand_leaf() to accommodate PrefixQuery, while deferring to the parent class implementation on TermQuery and PhraseQuery.</p>
+
+<pre> sub expand_leaf {
+ my ( $self, $leaf_query ) = @_;
+ my $text = $leaf_query->get_text;
+ if ( $text =~ /\*$/ ) {
+ my $or_query = Lucy::Search::ORQuery->new;
+ for my $field ( @{ $self->get_fields } ) {
+ my $prefix_query = PrefixQuery->new(
+ field => $field,
+ query_string => $text,
+ );
+ $or_query->add_child($prefix_query);
+ }
+ return $or_query;
+ }
+ else {
+ return $self->SUPER::expand_leaf($leaf_query);
+ }
+ }</pre>
+
+<p>Ordinarily, those asterisks would have been stripped when running tokens through the EasyAnalyzer -- query strings containing "foo*" would produce TermQueries for the term "foo". Our override intercepts tokens with trailing asterisks and processes them as PrefixQueries before <code>SUPER::expand_leaf</code> can discard them, so that a search for "foo*" can match "food", "foosball", and so on.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Usage"
+>Usage</a></h1>
+
+<p>Insert our custom parser into the search.cgi sample app to get a feel for how it behaves:</p>
+
+<pre> my $parser = FlatQueryParser->new( schema => $searcher->get_schema );
+ my $query = $parser->parse( decode( 'UTF-8', $cgi->param('q') || '' ) );
+ my $hits = $searcher->hits(
+ query => $query,
+ offset => $offset,
+ num_wanted => $page_size,
+ );
+ ...</pre>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/Cookbook/FastUpdates.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,141 @@
+Title: Lucy::Docs::Cookbook::FastUpdates - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::Cookbook::FastUpdates - Near real-time index updates.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="ABSTRACT"
+>ABSTRACT</a></h1>
+
+<p>While index updates are fast on average,
+worst-case update performance may be significantly slower.
+To make index updates consistently quick,
+we must manually intervene to control the process of index segment consolidation.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="The_problem"
+>The problem</a></h1>
+
+<p>Ordinarily,
+modifying an index is cheap.
+New data is added to new segments,
+and the time to write a new segment scales more or less linearly with the number of documents added during the indexing session.</p>
+
+<p>Deletions are also cheap most of the time,
+because we don't remove documents immediately but instead mark them as deleted,
+and adding the deletion mark is cheap.</p>
+
+<p>However,
+as new segments are added and the deletion rate for existing segments increases,
+search-time performance slowly begins to degrade.
+At some point,
+it becomes necessary to consolidate existing segments,
+rewriting their data into a new segment.</p>
+
+<p>If the recycled segments are small,
+the time it takes to rewrite them may not be significant.
+Every once in a while,
+though,
+a large amount of data must be rewritten.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Procrastinating_and_playing_catch-up"
+>Procrastinating and playing catch-up</a></h1>
+
+<p>The simplest way to force fast index updates is to avoid rewriting anything.</p>
+
+<p>Indexer relies upon <a href="../../../Lucy/Index/IndexManager.html" class="podlinkpod"
+>IndexManager</a>'s recycle() method to tell it which segments should be consolidated.
+If we subclass IndexManager and override recycle() so that it always returns an empty array,
+we get consistently quick performance:</p>
+
+<pre> package NoMergeManager;
+ use base qw( Lucy::Index::IndexManager );
+ sub recycle { [] }
+
+ package main;
+ my $indexer = Lucy::Index::Indexer->new(
+ index => '/path/to/index',
+ manager => NoMergeManager->new,
+ );
+ ...
+ $indexer->commit;</pre>
+
+<p>However, we can't procrastinate forever. Eventually, we'll have to run an ordinary, uncontrolled indexing session, potentially triggering a large rewrite of lots of small and/or degraded segments:</p>
+
+<pre> my $indexer = Lucy::Index::Indexer->new(
+ index => '/path/to/index',
+ # manager => NoMergeManager->new,
+ );
+ ...
+ $indexer->commit;</pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Acceptable_worst-case_update_time,_slower_degradation"
+>Acceptable worst-case update time, slower degradation</a></h1>
+
+<p>Never merging anything at all in the main indexing process is probably overkill. Small segments are relatively cheap to merge; we just need to guard against the big rewrites.</p>
+
+<p>Setting a ceiling on the number of documents in the segments to be recycled allows us to avoid a mass proliferation of tiny, single-document segments, while still offering decent worst-case update speed:</p>
+
+<pre> package LightMergeManager;
+ use base qw( Lucy::Index::IndexManager );
+
+ sub recycle {
+ my $self = shift;
+ my $seg_readers = $self->SUPER::recycle(@_);
+ @$seg_readers = grep { $_->doc_max < 10 } @$seg_readers;
+ return $seg_readers;
+ }</pre>
+
+<p>However, we still have to consolidate every once in a while, and while that happens content updates will be locked out.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Background_merging"
+>Background merging</a></h1>
+
+<p>If it's not acceptable to lock out updates while the index consolidation process runs, the alternative is to move the consolidation process out of band, using Lucy::Index::BackgroundMerger.</p>
+
+<p>It's never safe to have more than one Indexer attempting to modify the content of an index at the same time, but a BackgroundMerger and an Indexer can operate simultaneously:</p>
+
+<pre> # Indexing process.
+ use Scalar::Util qw( blessed );
+ my $retries = 0;
+ while (1) {
+ eval {
+ my $indexer = Lucy::Index::Indexer->new(
+ index => '/path/to/index',
+ manager => LightMergeManager->new,
+ );
+ $indexer->add_doc($doc);
+ $indexer->commit;
+ };
+ last unless $@;
+ if ( blessed($@) and $@->isa("Lucy::Store::LockErr") ) {
+ # Catch LockErr.
+ warn "Couldn't get lock ($retries retries)";
+ $retries++;
+ }
+ else {
+ die "Write failed: $@";
+ }
+ }
+
+ # Background merge process.
+ my $manager = Lucy::Index::IndexManager->new;
+ $manager->set_write_lock_timeout(60_000);
+ my $bg_merger = Lucy::Index::BackgroundMerger->new(
+ index => '/path/to/index',
+ manager => $manager,
+ );
+ $bg_merger->commit;</pre>
+
+<p>The exception handling code becomes useful once you have more than one index modification process happening simultaneously. By default, Indexer tries several times to acquire a write lock over the span of one second, then holds it until commit() completes. BackgroundMerger handles most of its work without the write lock, but it does need it briefly once at the beginning and once again near the end. Under normal loads, the internal retry logic will resolve conflicts, but if it's not acceptable to miss an insert, you probably want to catch LockErr exceptions thrown by Indexer. In contrast, a LockErr from BackgroundMerger probably just needs to be logged.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/DevGuide.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/DevGuide.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/DevGuide.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/DevGuide.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,34 @@
+Title: Lucy::Docs::DevGuide - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::DevGuide - Quick-start guide to hacking on Apache Lucy.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<p>The Apache Lucy code base is organized into roughly four layers:</p>
+
+<pre> * Charmonizer - compiler and OS configuration probing.
+ * Clownfish - header files.
+ * C - implementation files.
+ * Host - binding language.</pre>
+
+<p>Charmonizer is a configuration prober which writes a single header file, "charmony.h", describing the build environment and facilitating cross-platform development. It's similar to Autoconf or Metaconfig, but written in pure C.</p>
+
+<p>The ".cfh" files within the Lucy core are Clownfish header files. Clownfish is a purpose-built, declaration-only language which superimposes a single-inheritance object model on top of C which is specifically designed to co-exist happily with variety of "host" languages and to allow limited run-time dynamic subclassing. For more information see the Clownfish docs, but if there's one thing you should know about Clownfish OO before you start hacking, it's that method calls are differentiated from functions by capitalization:</p>
+
+<pre> Indexer_Add_Doc <-- Method, typically uses dynamic dispatch.
+ Indexer_add_doc <-- Function, always a direct invocation.</pre>
+
+<p>The C files within the Lucy core are where most of Lucy's low-level functionality lies. They implement the interface defined by the Clownfish header files.</p>
+
+<p>The C core is intentionally left incomplete, however; to be usable, it must be bound to a "host" language. (In this context, even C is considered a "host" which must implement the missing pieces and be "bound" to the core.) Some of the binding code is autogenerated by Clownfish on a spec customized for each language. Other pieces are hand-coded in either C (using the host's C API) or the host language itself.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/DocIDs.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/DocIDs.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/DocIDs.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/DocIDs.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,38 @@
+Title: Lucy::Docs::DocIDs - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::DocIDs - Characteristics of Apache Lucy document ids.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="DESCRIPTION"
+>DESCRIPTION</a></h1>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Document_ids_are_signed_32-bit_integers"
+>Document ids are signed 32-bit integers</a></h2>
+
+<p>Document ids in Apache Lucy start at 1.
+Because 0 is never a valid doc id,
+we can use it as a sentinel value:</p>
+
+<pre> while ( my $doc_id = $posting_list->next ) {
+ ...
+ }</pre>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Document_ids_are_ephemeral"
+>Document ids are ephemeral</a></h2>
+
+<p>The document ids used by Lucy are associated with a single index snapshot. The moment an index is updated, the mapping of document ids to documents is subject to change.</p>
+
+<p>Since IndexReader objects represent a point-in-time view of an index, document ids are guaranteed to remain static for the life of the reader. However, because they are not permanent, Lucy document ids cannot be used as foreign keys to locate records in external data sources. If you truly need a primary key field, you must define it and populate it yourself.</p>
+
+<p>Furthermore, the order of document ids does not tell you anything about the sequence in which documents were added to the index.</p>
+
+</div>
Added: lucy/site/trunk/content/docs/test/Lucy/Docs/FileFormat.mdtext
URL: http://svn.apache.org/viewvc/lucy/site/trunk/content/docs/test/Lucy/Docs/FileFormat.mdtext?rev=1498162&view=auto
==============================================================================
--- lucy/site/trunk/content/docs/test/Lucy/Docs/FileFormat.mdtext (added)
+++ lucy/site/trunk/content/docs/test/Lucy/Docs/FileFormat.mdtext Sun Jun 30 17:42:54 2013
@@ -0,0 +1,172 @@
+Title: Lucy::Docs::FileFormat - Apache Lucy Documentation
+
+<div>
+<a name='___top' class='dummyTopAnchor' ></a>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="NAME"
+>NAME</a></h1>
+
+<p>Lucy::Docs::FileFormat - Overview of index file format.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="OVERVIEW"
+>OVERVIEW</a></h1>
+
+<p>It is not necessary to understand the current implementation details of the index file format in order to use Apache Lucy effectively,
+but it may be helpful if you are interested in tweaking for high performance,
+exotic usage,
+or debugging and development.</p>
+
+<p>On a file system,
+an index is a directory.
+The files inside have a hierarchical relationship: an index is made up of "segments",
+each of which is an independent inverted index with its own subdirectory; each segment is made up of several component parts.</p>
+
+<pre> [index]--|
+ |--snapshot_XXX.json
+ |--schema_XXX.json
+ |--write.lock
+ |
+ |--seg_1--|
+ | |--segmeta.json
+ | |--cfmeta.json
+ | |--cf.dat-------|
+ | |--[lexicon]
+ | |--[postings]
+ | |--[documents]
+ | |--[highlight]
+ | |--[deletions]
+ |
+ |--seg_2--|
+ | |--segmeta.json
+ | |--cfmeta.json
+ | |--cf.dat-------|
+ | |--[lexicon]
+ | |--[postings]
+ | |--[documents]
+ | |--[highlight]
+ | |--[deletions]
+ |
+ |--[...]--| </pre>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Write-once_philosophy"
+>Write-once philosophy</a></h1>
+
+<p>All segment directory names consist of the string "seg_" followed by a number in base 36: seg_1, seg_5m, seg_p9s2 and so on, with higher numbers indicating more recent segments. Once a segment is finished and committed, its name is never re-used and its files are never modified.</p>
+
+<p>Old segments become obsolete and can be removed when their data has been consolidated into new segments during the process of segment merging and optimization. A fully-optimized index has only one segment.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Top-level_entries"
+>Top-level entries</a></h1>
+
+<p>There are a handful of "top-level" files and directories which belong to the entire index rather than to a particular segment.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="snapshot_XXX.json"
+>snapshot_XXX.json</a></h2>
+
+<p>A "snapshot" file, e.g. <code>snapshot_m7p.json</code>, is list of index files and directories. Because index files, once written, are never modified, the list of entries in a snapshot defines a point-in-time view of the data in an index.</p>
+
+<p>Like segment directories, snapshot files also utilize the unique-base-36-number naming convention; the higher the number, the more recent the file. The appearance of a new snapshot file within the index directory constitutes an index update. While a new segment is being written new files may be added to the index directory, but until a new snapshot file gets written, a Searcher opening the index for reading won't know about them.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="schema_XXX.json"
+>schema_XXX.json</a></h2>
+
+<p>The schema file is a Schema object describing the index's format, serialized as JSON. It, too, is versioned, and a given snapshot file will reference one and only one schema file.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="locks"
+>locks</a></h2>
+
+<p>By default, only one indexing process may safely modify the index at any given time. Processes reserve an index by laying claim to the <code>write.lock</code> file within the <code>locks/</code> directory. A smattering of other lock files may be used from time to time, as well.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="A_segment's_component_parts"
+>A segment's component parts</a></h1>
+
+<p>By default, each segment has up to five logical components: lexicon, postings, document storage, highlight data, and deletions. Binary data from these components gets stored in virtual files within the "cf.dat" compound file; metadata is stored in a shared "segmeta.json" file.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="segmeta.json"
+>segmeta.json</a></h2>
+
+<p>The segmeta.json file is a central repository for segment metadata. In addition to information such as document counts and field numbers, it also warehouses arbitrary metadata on behalf of individual index components.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Lexicon"
+>Lexicon</a></h2>
+
+<p>Each indexed field gets its own lexicon in each segment. The exact files involved depend on the field's type, but generally speaking there will be two parts. First, there's a primary <code>lexicon-XXX.dat</code> file which houses a complete term list associating terms with corpus frequency statistics, postings file locations, etc. Second, one or more "lexicon index" files may be present which contain periodic samples from the primary lexicon file to facilitate fast lookups.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Postings"
+>Postings</a></h2>
+
+<p>"Posting" is a technical term from the field of <a href="../../Lucy/Docs/IRTheory.html" class="podlinkpod"
+>information retrieval</a>, defined as a single instance of a one term indexing one document. If you are looking at the index in the back of a book, and you see that "freedom" is referenced on pages 8, 86, and 240, that would be three postings, which taken together form a "posting list". The same terminology applies to an index in electronic form.</p>
+
+<p>Each segment has one postings file per indexed field. When a search is performed for a single term, first that term is looked up in the lexicon. If the term exists in the segment, the record in the lexicon will contain information about which postings file to look at and where to look.</p>
+
+<p>The first thing any posting record tells you is a document id. By iterating over all the postings associated with a term, you can find all the documents that match that term, a process which is analogous to looking up page numbers in a book's index. However, each posting record typically contains other information in addition to document id, e.g. the positions at which the term occurs within the field.</p>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Documents"
+>Documents</a></h2>
+
+<p>The document storage section is a simple database, organized into two files:</p>
+
+<ul>
+<li><b>documents.dat</b> - Serialized documents.</li>
+
+<li><b>documents.ix</b> - Document storage index, a solid array of 64-bit integers where each integer location corresponds to a document id, and the value at that location points at a file position in the documents.dat file.</li>
+</ul>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Highlight_data"
+>Highlight data</a></h2>
+
+<p>The files which store data used for excerpting and highlighting are organized similarly to the files used to store documents.</p>
+
+<ul>
+<li><b>highlight.dat</b> - Chunks of serialized highlight data, one per doc id.</li>
+
+<li><b>highlight.ix</b> - Highlight data index -- as with the <code>documents.ix</code> file, a solid array of 64-bit file pointers.</li>
+</ul>
+
+<h2><a class='u' href='#___top' title='click to go to top of document'
+name="Deletions"
+>Deletions</a></h2>
+
+<p>When a document is "deleted" from a segment, it is not actually purged right away; it is merely marked as "deleted" via a deletions file. Deletions files contains bit vectors with one bit for each document in the segment; if bit #254 is set then document 254 is deleted, and if that document turns up in a search it will be masked out.</p>
+
+<p>It is only when a segment's contents are rewritten to a new segment during the segment-merging process that deleted documents truly go away.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="Compound_Files"
+>Compound Files</a></h1>
+
+<p>If you peer inside an index directory, you won't actually find any files named "documents.dat", "highlight.ix", etc. unless there is an indexing process underway. What you will find instead is one "cf.dat" and one "cfmeta.json" file per segment.</p>
+
+<p>To minimize the need for file descriptors at search-time, all per-segment binary data files are concatenated together in "cf.dat" at the close of each indexing session. Information about where each file begins and ends is stored in <code>cfmeta.json</code>. When the segment is opened for reading, a single file descriptor per "cf.dat" file can be shared among several readers.</p>
+
+<h1><a class='u' href='#___top' title='click to go to top of document'
+name="A_Typical_Search"
+>A Typical Search</a></h1>
+
+<p>Here's a simplified narrative, dramatizing how a search for "freedom" against a given segment plays out:</p>
+
+<ol>
+<li>The searcher asks the relevant Lexicon Index, "Do you know anything about 'freedom'?" Lexicon Index replies, "Can't say for sure, but if the main Lexicon file does, 'freedom' is probably somewhere around byte 21008".</li>
+
+<li>The main Lexicon tells the searcher "One moment, let me scan our records... Yes, we have 2 documents which contain 'freedom'. You'll find them in seg_6/postings-4.dat starting at byte 66991."</li>
+
+<li>The Postings file says "Yep, we have 'freedom', all right! Document id 40 has 1 'freedom', and document 44 has 8. If you need to know more, like if any 'freedom' is part of the phrase 'freedom of speech', ask me about positions!</li>
+
+<li>If the searcher is only looking for 'freedom' in isolation, that's where it stops. It now knows enough to assign the documents scores against "freedom", with the 8-freedom document likely ranking higher than the single-freedom document.</li>
+</ol>
+
+</div>