You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by revati joshi <re...@yahoo.com> on 2006/02/24 18:41:33 UTC
Indexing speed
hi all,
I just wnted to know how to increase the speed of indexing of files .
I tried it by using Multithreading approach but couldn't get much better performance.
It was same as it is in usual sequential indexing.Is there any other approach to get better Indexing performance incase of large file structure.?
waiting for the reply.
regards,
Revati.
---------------------------------
What are the most popular cars? Find out at Yahoo! Autos
Re: Indexing speed
Posted by Doug Cutting <cu...@apache.org>.
revati joshi wrote:
> hi all,
> I just wnted to know how to increase the speed of indexing of files .
> I tried it by using Multithreading approach but couldn't get much better performance.
> It was same as it is in usual sequential indexing.Is there any other approach to get better Indexing performance incase of large file structure.?
It would be better to know more about how you're currently using Lucene.
For example, it is much faster to index many documents on a single
IndexWriter rather than to create a new IndexWriter per document. It's
also somewhat faster to not store document text in the index. Etc.
That said, the single parameter that can most improve things is the
number of buffered docs:
http://lucene.apache.org/java/docs/api/org/apache/lucene/index/IndexWriter.html#setMaxBufferedDocs(int)
Try increasing this to, e.g., 50.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: queryParser and sorting question
Posted by Enrique Lamas <en...@corp.ya.com>.
Thanks Chris, I'll try that.
Bye
----- Original Message -----
From: "Chris Hostetter" <ho...@fucit.org>
To: <ja...@lucene.apache.org>
Sent: Tuesday, July 25, 2006 3:46 AM
Subject: Re: queryParser and sorting question
1) subclass DefaultSimilarity so that tf/idf allways return either
0 or 1 (more info on this can be found in the archives).
2) sort on score, and then your specific field as a secondary sort.
Of the top of my head, that should give you what you want ... but there
may be something else about the score that you have to tweak with the
Similarity class ... Searcher.explain will help you check that you get
consistent scores in all of your potential cases.
: Date: Mon, 24 Jul 2006 15:19:32 +0200
: From: Enrique Lamas <en...@corp.ya.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: queryParser and sorting question
:
: Hi,
: I'm, trying to execute a query to find some words, and I'm using
:
: QueryParser queryParser = new MultiFieldQueryParser(new String[] {"tags",
: "title"}, ProcessConstants.analyzer);
: Query query = queryParser.parse("word1 word2 word3");
:
: I want to show the results sorted like this:
:
: first, documents containing all of the three words, and this documents
: sorted by one of the index fields.
: Next, documents containing two of the words, also sorted by the same field
: like before.
: And so on.
:
: ¿How can I do this?
:
: I wold like to avoid executing several queries of this kind:
: "word1 + word2 + word3"
: "word1 + word2 - word3"
: "word1 - word2 + word3"
: "word1 - word2 - word3"
: .....
:
: Thanks
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: queryParser and sorting question
Posted by Chris Hostetter <ho...@fucit.org>.
1) subclass DefaultSimilarity so that tf/idf allways return either
0 or 1 (more info on this can be found in the archives).
2) sort on score, and then your specific field as a secondary sort.
Of the top of my head, that should give you what you want ... but there
may be something else about the score that you have to tweak with the
Similarity class ... Searcher.explain will help you check that you get
consistent scores in all of your potential cases.
: Date: Mon, 24 Jul 2006 15:19:32 +0200
: From: Enrique Lamas <en...@corp.ya.com>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: queryParser and sorting question
:
: Hi,
: I'm, trying to execute a query to find some words, and I'm using
:
: QueryParser queryParser = new MultiFieldQueryParser(new String[] {"tags",
: "title"}, ProcessConstants.analyzer);
: Query query = queryParser.parse("word1 word2 word3");
:
: I want to show the results sorted like this:
:
: first, documents containing all of the three words, and this documents
: sorted by one of the index fields.
: Next, documents containing two of the words, also sorted by the same field
: like before.
: And so on.
:
: ¿How can I do this?
:
: I wold like to avoid executing several queries of this kind:
: "word1 + word2 + word3"
: "word1 + word2 - word3"
: "word1 - word2 + word3"
: "word1 - word2 - word3"
: .....
:
: Thanks
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
queryParser and sorting question
Posted by Enrique Lamas <en...@corp.ya.com>.
Hi,
I'm, trying to execute a query to find some words, and I'm using
QueryParser queryParser = new MultiFieldQueryParser(new String[] {"tags",
"title"}, ProcessConstants.analyzer);
Query query = queryParser.parse("word1 word2 word3");
I want to show the results sorted like this:
first, documents containing all of the three words, and this documents
sorted by one of the index fields.
Next, documents containing two of the words, also sorted by the same field
like before.
And so on.
¿How can I do this?
I wold like to avoid executing several queries of this kind:
"word1 + word2 + word3"
"word1 + word2 - word3"
"word1 - word2 + word3"
"word1 - word2 - word3"
.....
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Problem with reverse ordering
Posted by Enrique Lamas <en...@corp.ya.com>.
Hi,
I have an index with some fields, one of them is a String representing a
date in the format yyyyMMddHHmmss.
I store this field as a Keyword to be sortable.
My problem is that I obtain the results in the same order with
Hits hits = is.search(query, new Sort("formatted-date"));
than
Hits hits = is.search(query, new Sort("formatted-date", true));
doen anyone why this could be happening?
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing speed
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Here: http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html
----- Original Message ----
From: anton@orbita1.ru
To: java-user@lucene.apache.org
Sent: Monday, February 27, 2006 4:24:27 AM
Subject: RE: Indexing speed
maxBufferedDocs parameters. You can also look for my article about indexing
with Lucene (link in the Wiki), which includes code for playing with various
parameters and explains what's going on, etc.
Sorry, but where this link ?
Where placed your article ? Please, give me url.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing speed
Posted by an...@orbita1.ru.
maxBufferedDocs parameters. You can also look for my article about indexing
with Lucene (link in the Wiki), which includes code for playing with various
parameters and explains what's going on, etc.
Sorry, but where this link ?
Where placed your article ? Please, give me url.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Indexing speed
Posted by an...@orbita1.ru.
maxBufferedDocs parameters. You can also look for my article about indexing
with Lucene (link in the Wiki), which includes code for playing with various
parameters and explains what's going on, etc.
Sorry, but where this link ?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Indexing speed
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Revati,
If you have Lucene in Action, look at Chapter 2, ection 2.7 - http://www.lucenebook.com/search?query=index+performance+control
If you don't have LIA, look at IndexWriter class and its mergeFactor and maxBufferedDocs parameters. You can also look for my article about indexing with Lucene (link in the Wiki), which includes code for playing with various parameters and explains what's going on, etc.
Otis
----- Original Message ----
From: revati joshi <re...@yahoo.com>
To: java-user@lucene.apache.org
Sent: Fri 24 Feb 2006 12:41:33 PM EST
Subject: Indexing speed
hi all,
I just wnted to know how to increase the speed of indexing of files .
I tried it by using Multithreading approach but couldn't get much better performance.
It was same as it is in usual sequential indexing.Is there any other approach to get better Indexing performance incase of large file structure.?
waiting for the reply.
regards,
Revati.
---------------------------------
What are the most popular cars? Find out at Yahoo! Autos
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org