You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stanislav Jordanov <st...@sirma.bg> on 2005/03/01 09:43:38 UTC
Re: Fast access to a random page of the search results.
// The test source code (second attempt).
// Just in case the .txt attachment does not pass through
// I am pasting the code here:
package index_test;
import org.apache.lucene.search.*;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.Directory;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import java.io.*;
import java.util.Enumeration;
import java.util.StringTokenizer;
import java.util.ArrayList;
import com.odi.util.query.QueryParseException;
public class Search {
public static void main(String[] args) throws Exception {
if (args.length != 1) {
throw new Exception("Usage: " + Search.class.getName() + "
<index dir>");
}
File indexDir = new File(args[0]);
if (!indexDir.exists() || !indexDir.isDirectory()) {
throw new Exception(indexDir + " is does not exist or is not a
directory.");
}
System.out.println("Using index: " + indexDir.getCanonicalPath());
BooleanQuery.setMaxClauseCount(Integer.MAX_VALUE);
search(indexDir);
}
public static void search(File indexDir) throws Exception {
Directory fsDir = FSDirectory.getDirectory(indexDir, false);
IndexSearcher is = null;
BufferedReader brdr = new BufferedReader(new
InputStreamReader(System.in));
String q;
Sort sort = null;
while (!(q = brdr.readLine()).equals("exit")) {
q = q.trim();
if (is == null || q.equals("newsearcher")) {
is = new IndexSearcher(fsDir);
if (q.equals("newsearcher")) {
continue;
}
}
if (q.startsWith("sort ")) {
StringTokenizer tkz = new StringTokenizer(q);
tkz.nextToken(); // skip the "sort" word
ArrayList<SortField> sortFields = new
ArrayList<SortField>();
while (tkz.hasMoreTokens()) {
String tok = tkz.nextToken();
boolean reverse = false;
if (tok.startsWith("-")) {
tok = tok.substring(1);
reverse = true;
}
sortFields.add(new SortField(tok, reverse));
}
sort = new Sort(sortFields.toArray(new SortField[0]));
System.out.println("Sorting by " + sort);
continue;
}
if (q.equals("nosort")) {
sort = null;
System.out.println("Sorting is off");
continue;
}
long startTs = System.currentTimeMillis();
Query query = null;
try {
query = QueryParser.parse(q, "qcontent", new
StandardAnalyzer(new String[0]));
}
catch (QueryParseException exn) {
exn.printStackTrace();
continue;
}
Hits hits = (sort != null ? is.search(query, sort) :
is.search(query));
int nHits = hits.length();//hc.nHits;
long stopTs = System.currentTimeMillis();
System.out.println("Found " + nHits + " document(s) that matched
query '" + q + "'");
System.out.println("Sorting by " + sort);
System.out.println("query executed in " + (stopTs - startTs) +
"ms");
if (nHits > 0) {
startTs = System.currentTimeMillis();
dummyMethod(hits.doc(nHits - nHits));
stopTs = System.currentTimeMillis();
System.out.println("Last doc accessed in " + (stopTs -
startTs)
+ "ms");
}
}
}
public static double dummyMethod(Document doc) {
return doc.getBoost();
}
private static void dumpDocument(Document doc) throws IOException {
System.out.println("<DOCUMENT-DUMP>");
for (Enumeration e = doc.fields(); e.hasMoreElements(); ) {
Field f = (Field) e.nextElement();
System.out.println(f.name() + " ::>> '" + f.stringValue() +
"'");
}
System.out.println("</DOCUMENT-DUMP>");
}
}
Indexing sit (stuff it) files
Posted by Luke Shannon <ls...@futurebrand.com>.
Hello;
I've almost completed my zip file indexer. I used the following to get an
InputStream for each file in the archive:
ZipFile zip = new ZipFile(new File(fileLocation));
ZipEntry zipEntry;
Enumeration files = zip.entries();
while (files.hasMoreElements()) {
zipEntry = (ZipEntry)files.nextElement();
//I have conditions here based on
zipEntry.getName().endsWith(".fileExtension) to determine which Document
Handler to use
//below is the inputstream I send the handler
InputStream in = zip.getInputStream(zipEntry);
}
So far this is looking ok (not quite done yet).
A request came in to index stuffit files. I'm hoping to be able to do
something similar as the above, but I haven't found a Java api to work with
this file type. Anyone have any experience with this?
Thanks,
Luke
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fast access to a random page of the search results.
Posted by Stanislav Jordanov <st...@sirma.bg>.
Thank you guys,
there's a good chance that I will have the management persuaded to drop the
'random access requirement'.
As you surely know, the management (usually) tends to be franticly
optimistic.
True to this trend, our management suggested us (the R&D team) that:
"... it is time to assume that we will have to modify the core Lucene engine
in order to achieve what we want."
Quite everyone in our craft likes to think of his(or her)self as fairly good
programming geek.
But honestly speeking, I'm lacking the confidence that I can all of a sudden
introduce a significant improvement in Lucene -
a far-from-trivial framework that has been evolving for the last 4 (or 5)
years.
Highly appreciating the work you've done:
Stenly
----- Original Message -----
From: "Doug Cutting" <cu...@apache.org>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Wednesday, March 02, 2005 12:04 AM
Subject: Re: Fast access to a random page of the search results.
> Daniel Naber wrote:
> > After fixing this I can reproduce the problem with a local index that
> > contains about 220.000 documents (700MB). Fetching the first document
> > takes for example 30ms, fetching the last one takes >100ms. Of course I
> > tested this with a query that returns many results (about 50.000).
> > Actually it happens even with the default sorting, no need to sort by
some
> > specific field.
>
> In part this is due to the fact that Hits first searches for the
> top-scoring 100 documents. Then, if you ask for a hit after that, it
> must re-query. In part this is also due to the fact that maintaining a
> queue of the top 50k hits is more expensive than maintaining a queue of
> the top 100 hits, so the second query is slower. And in part this could
> be caused by other things, such as that the highest ranking document
> might tend to be cached and not require disk io.
>
> One could perform profiling to determine which is the largest factor.
> Of these, only the first is really fixable: if you know you'll need hit
> 50k then you could tell this to Hits and have it perform only a single
> query. But the algorithmic cost of keeping the queue of the top 50k is
> the same as collecting all the hits and sorting them. So, in part,
> getting hits 49,990 through 50,000 is inherently slower than getting
> hits 0-10. We can minimize that, but not eliminate it.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fast access to a random page of the search results.
Posted by Doug Cutting <cu...@apache.org>.
Daniel Naber wrote:
> After fixing this I can reproduce the problem with a local index that
> contains about 220.000 documents (700MB). Fetching the first document
> takes for example 30ms, fetching the last one takes >100ms. Of course I
> tested this with a query that returns many results (about 50.000).
> Actually it happens even with the default sorting, no need to sort by some
> specific field.
In part this is due to the fact that Hits first searches for the
top-scoring 100 documents. Then, if you ask for a hit after that, it
must re-query. In part this is also due to the fact that maintaining a
queue of the top 50k hits is more expensive than maintaining a queue of
the top 100 hits, so the second query is slower. And in part this could
be caused by other things, such as that the highest ranking document
might tend to be cached and not require disk io.
One could perform profiling to determine which is the largest factor.
Of these, only the first is really fixable: if you know you'll need hit
50k then you could tell this to Hits and have it perform only a single
query. But the algorithmic cost of keeping the queue of the top 50k is
the same as collecting all the hits and sorting them. So, in part,
getting hits 49,990 through 50,000 is inherently slower than getting
hits 0-10. We can minimize that, but not eliminate it.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Fast access to a random page of the search results.
Posted by Daniel Naber <da...@t-online.de>.
On Tuesday 01 March 2005 19:15, Doug Cutting wrote:
> 'nHits - nHits' always equals zero. So you're actually printing the
> first document, not the last. The last document would be accessed with
> 'hits.doc(nHits)'.
After fixing this I can reproduce the problem with a local index that
contains about 220.000 documents (700MB). Fetching the first document
takes for example 30ms, fetching the last one takes >100ms. Of course I
tested this with a query that returns many results (about 50.000).
Actually it happens even with the default sorting, no need to sort by some
specific field.
Regards
Daniel
--
http://www.danielnaber.de
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Fast access to a random page of the search results.
Posted by Stanislav Jordanov <st...@sirma.bg>.
You're rihgt: nHits - nHits == 0 :)
But I did the right tests - it just happened that I've sent you a wrong
source.
I mean I performed the tests accessing the proper last doc: doc(nHits - 1)
then I switched to accessing the first hit, just to make sure (once again)
there is essential difference in access times.
And instead of wiping out the code fragment (nHits - 1) and replacing it
with 0, a replaced 1 with nHits.
That's how the resulting (nHits - nHits) code got posted.
Yes, the index is stored at a local hard drive.
Stenly
----- Original Message -----
From: "Doug Cutting" <cu...@apache.org>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Tuesday, March 01, 2005 8:15 PM
Subject: Re: Fast access to a random page of the search results.
> Stanislav Jordanov wrote:
> > startTs = System.currentTimeMillis();
> > dummyMethod(hits.doc(nHits - nHits));
> > stopTs = System.currentTimeMillis();
> > System.out.println("Last doc accessed in " + (stopTs -
> > startTs)
> > + "ms");
>
> 'nHits - nHits' always equals zero. So you're actually printing the
> first document, not the last. The last document would be accessed with
> 'hits.doc(nHits)'. Accessing the last document should not be much
> slower (or faster) than accessing the first.
>
> 200+ milliseconds to access a document does seem slow. Where is you
> index stored? On a local hard drive?
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Fast access to a random page of the search results.
Posted by Doug Cutting <cu...@apache.org>.
Stanislav Jordanov wrote:
> startTs = System.currentTimeMillis();
> dummyMethod(hits.doc(nHits - nHits));
> stopTs = System.currentTimeMillis();
> System.out.println("Last doc accessed in " + (stopTs -
> startTs)
> + "ms");
'nHits - nHits' always equals zero. So you're actually printing the
first document, not the last. The last document would be accessed with
'hits.doc(nHits)'. Accessing the last document should not be much
slower (or faster) than accessing the first.
200+ milliseconds to access a document does seem slow. Where is you
index stored? On a local hard drive?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org