You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Marvin Humphrey <ma...@rectangular.com> on 2006/04/03 07:51:54 UTC
Benchmarkers
Greets,
I've been putting together some indexing benchmarkers which compare
Lucene, KinoSearch, and Plucene using the Reuters 21578 corpus.
Before I publish any numbers, I'd like to make sure that my limited
Java experience hasn't unfairly hobbled the Lucene app. Does anyone
see anything in the source below that wouldn't be considered best
practice?
Note that before indexing, the corpus, which comes packaged as 22
SGML files, gets preprocessed into 19000 or so text files spread out
over 22 directories -- the idea being that we want to measure
indexers, not SGML parsers.
The other two indexers, plus the corpus extractor and a short README,
can be found at <http://www.rectangular.com/svn/kinosearch/trunk/t/
benchmarks/>.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.analysis.WhitespaceAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import java.io.File;
import java.io.BufferedReader;
import java.io.FileReader;
import java.text.DecimalFormat;
import java.util.Date;
import java.util.Vector;
import java.util.Collections;
public class LuceneIndexer {
private File corpusDir = new File("extracted_corpus");
private File indexDir = new File("lucene_index");
public LuceneIndexer() { }
public static void main (String[] args) throws Exception {
LuceneIndexer indexer = new LuceneIndexer();
// index all docs unless otherwise spec'd
int maxToIndex = args.length > 0 ?
Integer.parseInt(args[0]) : 0;
// verify that we're running from the right directory
String curDir = new File(".").getCanonicalPath();
if (!curDir.endsWith("benchmarks"))
throw new Exception("Must be run from benchmarks/ ");
// assemble the sorted list of article files
String[] fileList = indexer.buildFileList();
// start the clock and build the index
long start = new Date().getTime();
int numIndexed = indexer.buildIndex(fileList, maxToIndex);
// stop the clock and print a report
long end = new Date().getTime();
indexer.printReport(start, end, numIndexed);
}
// Return a lexically sorted list of all article files from all
subdirs.
private String[] buildFileList () throws Exception {
File[] articleDirs = corpusDir.listFiles();
Vector filePaths = new Vector();
for (int i = 0; i < articleDirs.length; i++) {
File[] articles = articleDirs[i].listFiles();
for (int j = 0; j < articles.length; j++) {
String path = articles[i].getCanonicalPath();
if (path.indexOf("article") == -1)
continue;
filePaths.add(path);
}
}
Collections.sort(filePaths);
return (String[])filePaths.toArray(new String[filePaths.size()]);
}
// Build an index, stopping at maxToIndex docs if maxToIndex > 0.
private int buildIndex (String[] fileList, int maxToIndex)
throws Exception {
IndexWriter writer = new IndexWriter(indexDir,
new WhitespaceAnalyzer(), true);
int docsSoFar = 0;
for (int i = 0; i < fileList.length; i++) {
// add content to index
File f = new File(fileList[i]);
Document doc = this.nextDoc(f);
writer.addDocument(doc);
// bail if we've reached spec'd number of docs
if (maxToIndex > 0 && ++docsSoFar == maxToIndex)
break;
}
// finish index
int numIndexed = writer.docCount();
writer.optimize();
writer.close();
return numIndexed;
}
// Retrieve an article, parse it, and return a Lucene Document.
private Document nextDoc(File f) throws Exception {
// the title is the first line, the body is the rest
BufferedReader br = new BufferedReader(new FileReader(f));
String title;
if ( (title = br.readLine()) == null)
throw new Exception("Failed to read title");
StringBuffer buf = new StringBuffer();
String str;
while ( (str = br.readLine()) != null )
buf.append( str );
br.close();
String body = buf.toString();
// add title and body to doc
Document doc = new Document();
Field titleField = new Field("title", title,
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
Field bodyField = new Field("body", body,
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
doc.add(titleField);
doc.add(bodyField);
return doc;
}
// Print out stats for this run.
private void printReport(long start, long end, int numIndexed) {
float secs = (float)(end - start) / 1000;
DecimalFormat format = new DecimalFormat("#,##0.00");
String secString = format.format(secs);
Package lucenePackage = org.apache.lucene.LucenePackage.get();
String version = lucenePackage.getSpecificationVersion();
System.out.println("Java Lucene " + version
+ " DOCS: " + numIndexed + " SECS: " + secString);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 3, 2006, at 7:08 AM, karl wettin wrote:
> And if possible, it would be very interesting to see results using -
> d64 and -d32. And different platforms.
I only have easy access to one machine running Java: my G4 1.67 MHz
laptop, running Mac OS X 10.4.5. I agree that it would be very
interesting to see results from other systems. If anyone else wants
to run these, you can grab the source from my subversion repository:
svn co http://www.rectangular.com/svn/kinosearch/trunk/t/
benchmarks benchmarks
You'll need the Reuters corpus (location specified in benchmarks/
README.txt). You'll also need Perl to run the script which
preprocesses the Reuters corpus.
If you're interested in running the benchmarks for KinoSearch, I
recommend using the CPAN shell to install it, as it has a few
dependencies and it's easier to let CPAN handle them. You'll need a
C compiler, as KinoSearch is not pure Perl. The latest official
release, 0.08, is available with the usual...
cpan> install KinoSearch
... which will find the tarball <CREAMYG/KinoSearch-0.08.tar.gz>.
However, for my tests, I'm using a developer's release, 0.09_03,
which is at least twice as fast as 0.08. 0.9 is also the first
version of KinoSearch with incremental indexing enabled. To get
0.09_03 from CPAN, type...
cpan> install CREAMYG/KinoSearch-0.09_03.tar.gz
... at the CPAN shell prompt.
If 0.09_03 hasn't made it your CPAN mirror yet, you can also get it
from <http://www.rectangular.com/kinosearch>.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by karl wettin <ka...@snigel.net>.
3 apr 2006 kl. 17.26 skrev karl wettin:
> Solaris: HP DL145, 1 x Dualcore Opteron 2.2 GHz, 4 GB of RAM
> Linux: HP DL140 with 2x 3.06GHz Xeon CPUs and 4GB of RAM
If you want me to, and package the benchmark tests in a way simple
for me to run them, I'll run them on these machines for you.
The Solaris can do -d64 and -d32.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by karl wettin <ka...@snigel.net>.
3 apr 2006 kl. 16.50 skrev Grant Ingersoll:
>> And if possible, it would be very interesting to see results using
>> -d64 and -d32. And different platforms. So far I've got best
>> results in decending order on Solaris, OS X and last(!) Linux.
>> Solaris is straight out amazing under heavy load. Might even do
>> the switch next time I buy a new workstation.
>>
> What are the configurations of you machines?
You ask about the hardware? It's been multiple and I have not
benchmarked anything.
I've been running the following hardware in an environemt with 130000
documents and an average of 3000 queries per minute 24/7, peeking at
10000.
Solaris: HP DL145, 1 x Dualcore Opteron 2.2 GHz, 4 GB of RAM
Linux: HP DL140 with 2x 3.06GHz Xeon CPUs and 4GB of RAM
The Solaris so totally outperforms the Linux, especially at high
load. The memory management is magic.
I can not prove anything, but it feels as I get more bang for the CPU
on PPC Apple JVM than on X86 SUN Linux JVM.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Grant Ingersoll <gs...@syr.edu>.
karl wettin wrote:
>
>
> And if possible, it would be very interesting to see results using
> -d64 and -d32. And different platforms. So far I've got best results
> in decending order on Solaris, OS X and last(!) Linux. Solaris is
> straight out amazing under heavy load. Might even do the switch next
> time I buy a new workstation.
>
What are the configurations of you machines?
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by karl wettin <ka...@snigel.net>.
3 apr 2006 kl. 15.57 skrev Yonik Seeley:
> - use enough heap so too much time isn't taken in GC
I recommend -XX:+AggressiveHeap.
And if possible, it would be very interesting to see results using -
d64 and -d32. And different platforms. So far I've got best results
in decending order on Solaris, OS X and last(!) Linux. Solaris is
straight out amazing under heavy load. Might even do the switch next
time I buy a new workstation.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 3, 2006, at 6:26 PM, Marvin Humphrey wrote:
>
> On Apr 3, 2006, at 5:43 PM, Doug Cutting wrote:
>
>> Marvin Humphrey wrote:
>>> Plucene is a Lucene 1.3 port, so it doesn't have
>>> max_buffered_docs -- but I can set merge_factor to 1000.
>>
>> I would not recommend that. With a merge factor that high you may
>> run out of file handles, and, moreover, I doubt that disks are
>> very efficient when reading from that many streams.
>
> Running out of filehandles is a solvable problem because you can
> set ulimit -n to whatever on OS X -- and you pretty much have to,
> since the default is 256.
>
> The streams issue is more complicated. N-way merges from disk tend
> to be IO-bound. The best I can do is try a couple numbers and see
> what works. IIRC, the number 100 has gone by on the Plucene
> mailing list as a good value.
The higher the better, it seems. Here's times to index 1000 docs:
merge_factor secs
10 141
30 123
100 107
250 100
1000 89
I suspect that Plucene is so CPU-bound that the IO doesn't come into
play.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 3, 2006, at 5:43 PM, Doug Cutting wrote:
> Marvin Humphrey wrote:
>> Plucene is a Lucene 1.3 port, so it doesn't have max_buffered_docs
>> -- but I can set merge_factor to 1000.
>
> I would not recommend that. With a merge factor that high you may
> run out of file handles, and, moreover, I doubt that disks are very
> efficient when reading from that many streams.
Running out of filehandles is a solvable problem because you can set
ulimit -n to whatever on OS X -- and you pretty much have to, since
the default is 256.
The streams issue is more complicated. N-way merges from disk tend
to be IO-bound. The best I can do is try a couple numbers and see
what works. IIRC, the number 100 has gone by on the Plucene mailing
list as a good value.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> Plucene is a Lucene 1.3 port, so it doesn't have max_buffered_docs --
> but I can set merge_factor to 1000.
I would not recommend that. With a merge factor that high you may run
out of file handles, and, moreover, I doubt that disks are very
efficient when reading from that many streams.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 3, 2006, at 6:57 AM, Yonik Seeley wrote:
> A couple of points:
> - Are all the lucene variations using the same index parameters?
> max buffered docs, index format (compound or not), mergeFactor, etc
> I personally use non-compound index format, max buffered docs=1000,
> mergeFactor=10
IndexWriter writer = new IndexWriter(indexDir,
new WhitespaceAnalyzer(), true);
+ writer.setMaxBufferedDocs(1000);
+ writer.setUseCompoundFile(false);
I'll set Lucene to use the non-compound format. KinoSearch only
supports the compound index format, but since it only writes one
segment per indexing session, each file only gets rewritten once and
that's not going to be much of a handicap. Plucene only uses the non-
compound format.
KinoSearch doesn't have max_buffered_docs or merge_factor settings,
since it uses a different merge model based on external sorting and
serialized postings. Currently, it keeps track of the amount of
memory consumed by the in-memory sort pool, and writes a run when
that number hits 20 MB. Version 0.09_02 uses its own external
sorting routine for the first time, so I can and probably should
adapt it use a max_buffered_docs variable, which it will need to poll
a lot less frequently. But that's an optimization for another day.
Plucene is a Lucene 1.3 port, so it doesn't have max_buffered_docs --
but I can set merge_factor to 1000.
> - reading in the file line by line probably isn't the fastest (esp
> when you just construct another big string out of it).
I'm addressing this issue in my reply to Doug.
> - Java settings:
> - use the 1.5 JVM if possible, it's much faster than 1.4 in my
> experience
Interestingly, 1.5 produces slightly inferior results on my G4. (I
know about the command line alias snafu, BTW: <http://
www.cs.princeton.edu/introcs/11hello/mac.html>).
I'll include results from both 1.4 and 1.5. I'll also include
results for a vanilla compile of Perl 5.8.8, which is definitely
faster than the Perl 5.8.6 Apple ships with OS X Tiger.
> - use "-server", it's much faster than "-client"
> - use enough heap so too much time isn't taken in GC
Okeedoke.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Yonik Seeley <ys...@gmail.com>.
Hi Marvin,
A couple of points:
- Are all the lucene variations using the same index parameters?
max buffered docs, index format (compound or not), mergeFactor, etc
I personally use non-compound index format, max buffered docs=1000,
mergeFactor=10
- reading in the file line by line probably isn't the fastest (esp
when you just construct another big string out of it).
- Java settings:
- use the 1.5 JVM if possible, it's much faster than 1.4 in my experience
- use "-server", it's much faster than "-client"
- use enough heap so too much time isn't taken in GC
Looking forward to some numbers!
-Yonik
http://incubator.apache.org/solr Solr, The Open Source Lucene Search Server
On 4/3/06, Marvin Humphrey <ma...@rectangular.com> wrote:
> Greets,
>
> I've been putting together some indexing benchmarkers which compare
> Lucene, KinoSearch, and Plucene using the Reuters 21578 corpus.
> Before I publish any numbers, I'd like to make sure that my limited
> Java experience hasn't unfairly hobbled the Lucene app. Does anyone
> see anything in the source below that wouldn't be considered best
> practice?
>
> Note that before indexing, the corpus, which comes packaged as 22
> SGML files, gets preprocessed into 19000 or so text files spread out
> over 22 directories -- the idea being that we want to measure
> indexers, not SGML parsers.
>
> The other two indexers, plus the corpus extractor and a short README,
> can be found at <http://www.rectangular.com/svn/kinosearch/trunk/t/
> benchmarks/>.
>
> Marvin Humphrey
> Rectangular Research
> http://www.rectangular.com/
>
>
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.analysis.WhitespaceAnalyzer;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field;
>
> import java.io.File;
> import java.io.BufferedReader;
> import java.io.FileReader;
> import java.text.DecimalFormat;
> import java.util.Date;
> import java.util.Vector;
> import java.util.Collections;
>
> public class LuceneIndexer {
> private File corpusDir = new File("extracted_corpus");
> private File indexDir = new File("lucene_index");
>
> public LuceneIndexer() { }
>
> public static void main (String[] args) throws Exception {
> LuceneIndexer indexer = new LuceneIndexer();
>
> // index all docs unless otherwise spec'd
> int maxToIndex = args.length > 0 ?
> Integer.parseInt(args[0]) : 0;
>
> // verify that we're running from the right directory
> String curDir = new File(".").getCanonicalPath();
> if (!curDir.endsWith("benchmarks"))
> throw new Exception("Must be run from benchmarks/ ");
>
> // assemble the sorted list of article files
> String[] fileList = indexer.buildFileList();
>
> // start the clock and build the index
> long start = new Date().getTime();
> int numIndexed = indexer.buildIndex(fileList, maxToIndex);
>
> // stop the clock and print a report
> long end = new Date().getTime();
> indexer.printReport(start, end, numIndexed);
> }
>
> // Return a lexically sorted list of all article files from all
> subdirs.
> private String[] buildFileList () throws Exception {
> File[] articleDirs = corpusDir.listFiles();
> Vector filePaths = new Vector();
> for (int i = 0; i < articleDirs.length; i++) {
> File[] articles = articleDirs[i].listFiles();
> for (int j = 0; j < articles.length; j++) {
> String path = articles[i].getCanonicalPath();
> if (path.indexOf("article") == -1)
> continue;
> filePaths.add(path);
> }
> }
> Collections.sort(filePaths);
> return (String[])filePaths.toArray(new String[filePaths.size()]);
> }
>
> // Build an index, stopping at maxToIndex docs if maxToIndex > 0.
> private int buildIndex (String[] fileList, int maxToIndex)
> throws Exception {
> IndexWriter writer = new IndexWriter(indexDir,
> new WhitespaceAnalyzer(), true);
>
> int docsSoFar = 0;
> for (int i = 0; i < fileList.length; i++) {
> // add content to index
> File f = new File(fileList[i]);
> Document doc = this.nextDoc(f);
> writer.addDocument(doc);
>
> // bail if we've reached spec'd number of docs
> if (maxToIndex > 0 && ++docsSoFar == maxToIndex)
> break;
> }
>
> // finish index
> int numIndexed = writer.docCount();
> writer.optimize();
> writer.close();
>
> return numIndexed;
> }
>
> // Retrieve an article, parse it, and return a Lucene Document.
> private Document nextDoc(File f) throws Exception {
> // the title is the first line, the body is the rest
> BufferedReader br = new BufferedReader(new FileReader(f));
> String title;
> if ( (title = br.readLine()) == null)
> throw new Exception("Failed to read title");
> StringBuffer buf = new StringBuffer();
> String str;
> while ( (str = br.readLine()) != null )
> buf.append( str );
> br.close();
> String body = buf.toString();
>
> // add title and body to doc
> Document doc = new Document();
> Field titleField = new Field("title", title,
> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
> Field bodyField = new Field("body", body,
> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
> doc.add(titleField);
> doc.add(bodyField);
>
> return doc;
> }
>
> // Print out stats for this run.
> private void printReport(long start, long end, int numIndexed) {
> float secs = (float)(end - start) / 1000;
> DecimalFormat format = new DecimalFormat("#,##0.00");
> String secString = format.format(secs);
> Package lucenePackage = org.apache.lucene.LucenePackage.get();
> String version = lucenePackage.getSpecificationVersion();
> System.out.println("Java Lucene " + version
> + " DOCS: " + numIndexed + " SECS: " + secString);
> }
> }
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 3, 2006, at 11:11 AM, Doug Cutting wrote:
> You might still, if you have time, try swapping in something like
> StopAnalyzer and/or turning off Field.Store.YES. The relative
> speeds of the various implementations may vary in interesting ways,
> since these paramters may emphasize different strengths and
> weaknesses.
I think the stored+vectored / notStored+notVectored comparison will
emphasize strengths and weaknesses of the various merge models.
> For example, per-token and per-term method calls may be more-or-
> less of a bottleneck in different implementations, similarly for
> per-character i/o when copying document content around.
True, and KinoSearch's Analyzers don't work like Lucene's -- there
isn't even a Token class. But that's a topic for another day.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Doug Cutting <cu...@apache.org>.
Doug Cutting wrote:
> Please make sure that analyzers are comparable between the various
> engines you benchmark.
I just went back and re-read what you're benchmarking, and they're all
versions of Lucene, so you're probably already using comparable
analyzers! Sorry for not noticing that the first time...
You might still, if you have time, try swapping in something like
StopAnalyzer and/or turning off Field.Store.YES. The relative speeds of
the various implementations may vary in interesting ways, since these
paramters may emphasize different strengths and weaknesses. For
example, per-token and per-term method calls may be more-or-less of a
bottleneck in different implementations, similarly for per-character i/o
when copying document content around.
Cheers,
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 3, 2006, at 10:36 AM, Doug Cutting wrote:
> Marvin Humphrey wrote:
>> IndexWriter writer = new IndexWriter(indexDir,
>> new WhitespaceAnalyzer(), true);
>
> Please make sure that analyzers are comparable between the various
> engines you benchmark. WhitespaceAnalyzer is efficient, but
> results in far more tokens and terms than, e.g., StopAnalyzer
> (alphabetic character sequences, lowercased, with a 35-word English
> stop list).
They're all using WhitespaceAnalyzer or the equivalent. KinoSearch
doesn't offer that class per se, but its Tokenizer class allows you
to specify an arbitrary regex matching one token.
# a WhitespaceAnalyzer in KinoSearch
my $tokenizer = KinoSearch::Analysis::Tokenizer->new(
token_re => qr/\S+/,
);
> Since tokens and terms are the atoms and elements of indexing,
> their counts are a dominant factor in performance.
>
> Increasing IndexWriter.setMaxBufferedDocs(100) or more will
> increase indexing speed by using more Java heap. 10 is the
> default. IndexWriter.setUseCompoundFile(false) will also increase
> indexing speed. I don't think increasing IndexWriter.setMergeFactor
> () should help much. and advise staying with the default (10).
> Folks used to set this as a surrogate for setMaxBufferedDocs before
> that was a separate paramter.
I'm addressing these issues in my reply to Yonik.
>
> You may need to specify a larger Java heap, with something like -
> Xmx500M. The default is around 64MB.
Great, I'll use -Xmx500M.
> Also, the -server option is almost always faster with Sun's JVM.
> Sun's 1.5 JVM is faster than their 1.4 JVM. I think IBMs JVM may
> be generally faster for indexing. The last I checked, one was
> fasteer for indexing and the other for searching, but I'm not
> certain which was which.
I'm running these on my G4 laptop.
>
>> private Document nextDoc(File f) throws Exception {
>> // the title is the first line, the body is the rest
>> BufferedReader br = new BufferedReader(new FileReader(f));
>> String title;
>> if ( (title = br.readLine()) == null)
>> throw new Exception("Failed to read title");
>> StringBuffer buf = new StringBuffer();
>> String str;
>> while ( (str = br.readLine()) != null )
>> buf.append( str );
>> br.close();
>> String body = buf.toString();
>> // add title and body to doc
>> Document doc = new Document();
>> Field titleField = new Field("title", title,
>> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
>> Field bodyField = new Field("body", body,
>> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
>> doc.add(titleField);
>> doc.add(bodyField);
>
> You can avoid some buffering by passing a Reader for the body text:
>
> Field bodyField = new Field("body", br,
> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
>
> The only rub is that you'll have to make sure that the FileReader
> is closed. So you could rewrite this method to be something like:
>
> private void indexFile(File f, IndexWriter writer) {
> BufferedReader br = new BufferedReader(new FileReader(f));
> try {
> Document doc = new Document();
>
> ... read title from br and add it to doc ...
>
> Field bodyField = new Field("body", br,
> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
> doc.add(bodyField);
>
> writer.addDocument(doc);
> } finally {
> br.close();
> }
> }
>
> Does that make sense?
It does. However, there's no constructor for Field which allows
Field.Store.YES, but uses a Reader instead of a String.
> Finally, I question your use of Field.Store.YES. Do you really to
> use Lucene to store the full content of your documents?
Yes. By default, all fields in KinoSearch are analyzed, stored, and
vectorized (with positions and offsets). This allows use of the
Highlighter with minimum fuss. Savvy users looking to shrink the
size of their indexes can override those defaults.
I'd originally omitted TermVectors from the benchmarking apps because
Plucene doesn't have them. But having KinoSearch and Lucene generate
them isn't going to slow them down enough that Plucene will become
competitive. It makes sense to generate two result sets, one with
the body stored and vectored, and one with the body neither stored
nor vectored. I'll use the Reader constructor for Lucene's unstored
version.
> Are you asking this of the other engines?
They were all even before. Now Plucene will have a slight advantage
in one config.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Benchmarkers
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> IndexWriter writer = new IndexWriter(indexDir,
> new WhitespaceAnalyzer(), true);
Please make sure that analyzers are comparable between the various
engines you benchmark. WhitespaceAnalyzer is efficient, but results in
far more tokens and terms than, e.g., StopAnalyzer (alphabetic character
sequences, lowercased, with a 35-word English stop list). Since tokens
and terms are the atoms and elements of indexing, their counts are a
dominant factor in performance.
Increasing IndexWriter.setMaxBufferedDocs(100) or more will increase
indexing speed by using more Java heap. 10 is the default.
IndexWriter.setUseCompoundFile(false) will also increase indexing speed.
I don't think increasing IndexWriter.setMergeFactor() should help
much. and advise staying with the default (10). Folks used to set this
as a surrogate for setMaxBufferedDocs before that was a separate paramter.
You may need to specify a larger Java heap, with something like
-Xmx500M. The default is around 64MB. Also, the -server option is
almost always faster with Sun's JVM. Sun's 1.5 JVM is faster than their
1.4 JVM. I think IBMs JVM may be generally faster for indexing. The
last I checked, one was fasteer for indexing and the other for
searching, but I'm not certain which was which.
> private Document nextDoc(File f) throws Exception {
> // the title is the first line, the body is the rest
> BufferedReader br = new BufferedReader(new FileReader(f));
> String title;
> if ( (title = br.readLine()) == null)
> throw new Exception("Failed to read title");
> StringBuffer buf = new StringBuffer();
> String str;
> while ( (str = br.readLine()) != null )
> buf.append( str );
> br.close();
> String body = buf.toString();
>
> // add title and body to doc
> Document doc = new Document();
> Field titleField = new Field("title", title,
> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
> Field bodyField = new Field("body", body,
> Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
> doc.add(titleField);
> doc.add(bodyField);
You can avoid some buffering by passing a Reader for the body text:
Field bodyField = new Field("body", br,
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
The only rub is that you'll have to make sure that the FileReader is
closed. So you could rewrite this method to be something like:
private void indexFile(File f, IndexWriter writer) {
BufferedReader br = new BufferedReader(new FileReader(f));
try {
Document doc = new Document();
... read title from br and add it to doc ...
Field bodyField = new Field("body", br,
Field.Store.YES, Field.Index.TOKENIZED, Field.TermVector.NO);
doc.add(bodyField);
writer.addDocument(doc);
} finally {
br.close();
}
}
Does that make sense?
Finally, I question your use of Field.Store.YES. Do you really to use
Lucene to store the full content of your documents? Are you asking this
of the other engines?
Cheers,
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org