You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by spinergywmy <sp...@gmail.com> on 2006/11/02 08:29:58 UTC
reindex all files
Hi,
Basically what I want to do is to reindex all the files in particular
directory. I wonder how I can do this with Lucene search? And I noticed that
reindexing all the files will actually take a lot of time and resources, so
how I can optimize the preformance problem.
Hope there are examples that I can review on.
Thanks.
regards,
Wooi Meng
--
View this message in context: http://www.nabble.com/reindex-all-files-tf2558188.html#a7128924
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Warming up a Searcher
Posted by Yonik Seeley <yo...@apache.org>.
SolrCore.getSearcher() and registerSearcher() work together to do do warming.
If you want to try and rip that out of solr, remove any calls having
to do with autowarming (warming new caches from most-recently-used
items in old caches), and replace the static warming queries (defined
in an external configuration file to be run when warming) with your
own queries.
Or see if Solr will meet your needs as-is ;-)
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Warming up a Searcher
Posted by Simon Willnauer <si...@googlemail.com>.
I can't point out the actual position in solr but I bet if you spend
10 min looking into the solr source you will find a nice example how
to warm up a searcher.
regards simon
On 11/2/06, Aigner, Thomas <TA...@wescodist.com> wrote:
>
> I have seen numerous posts on warming up a searcher, but was wondering
> if someone could post their code that would spin off another thread to
> warm up a searcher, then switch to the new one when it is warmed up?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reindex all files
Posted by spinergywmy <sp...@gmail.com>.
Hi,
Thanks Erik.
I will take a look on it.
regards,
Wooi Meng
--
View this message in context: http://www.nabble.com/reindex-all-files-tf2558188.html#a7149611
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reindex all files
Posted by Erick Erickson <er...@gmail.com>.
I really, really, really recommend a copy of "Lucene in action" <G>.
Another rich source of examples is the unit test code that comes along with
the source for Lucene. You can download a copy of all the source and unit
tests from http://www.eng.lsu.edu/mirrors/apache/lucene/java/
Best
Erick
On 11/2/06, spinergywmy <sp...@gmail.com> wrote:
>
>
> Hi,
>
> Thanks Erik.
>
> I hope u can provide me a complete example cause I actually quite new
> to
> lucene search, sorry for the inconvenience.
>
> Thanks.
>
> regards,
> Wooi Meng
> --
> View this message in context:
> http://www.nabble.com/reindex-all-files-tf2558188.html#a7135033
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Warming up a Searcher
Posted by "Aigner, Thomas" <TA...@WescoDist.com>.
I have seen numerous posts on warming up a searcher, but was wondering
if someone could post their code that would spin off another thread to
warm up a searcher, then switch to the new one when it is warmed up?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reindex all files
Posted by spinergywmy <sp...@gmail.com>.
Hi,
Thanks Erik.
I hope u can provide me a complete example cause I actually quite new to
lucene search, sorry for the inconvenience.
Thanks.
regards,
Wooi Meng
--
View this message in context: http://www.nabble.com/reindex-all-files-tf2558188.html#a7135033
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: reindex all files
Posted by Erick Erickson <er...@gmail.com>.
There isn't enough of an explanation here to give you any meaningful answer
about performance. Do you have any evidence that the optimization will take
a long time? Have you run it before? Do you have any count of how many files
you're indexing? Do they have a complex structure? What are the performance
numbers you need? What performance are you seeing? The only real advice here
is "try some stuff and see". That said, here are some parameters that will
affect your indexing speed, see the javadocs for cautions about how to use
these.
setMergeFactor
setMaxBufferedDocs
setMaxMergeDocs
I'd advise you just try a few runs with the defaults and see whether
performance needs tweaking....
Here's some code for recursing a directory structure
private void index(File file) throws Exception {
// Boring stuff to only try to index files.
if (file.canRead()) {
if (file.isDirectory()) {
String[] files = file.list();
if (files != null) {
for (int i = 0; i < files.length; i++) {
index(new File(file, files[i]));
}
}
} else {
// Index the file
System.out.println("Indexing " + file);
// Here's the interesting part.
indexOneFile(file); // Here's where you'll do your
processing....
}
}
}
Call this with a new File("path to directory");
On 11/2/06, spinergywmy <sp...@gmail.com> wrote:
>
>
> Hi,
>
> Basically what I want to do is to reindex all the files in particular
> directory. I wonder how I can do this with Lucene search? And I noticed
> that
> reindexing all the files will actually take a lot of time and resources,
> so
> how I can optimize the preformance problem.
>
> Hope there are examples that I can review on.
>
> Thanks.
>
> regards,
> Wooi Meng
> --
> View this message in context:
> http://www.nabble.com/reindex-all-files-tf2558188.html#a7128924
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>