You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by senthil kumaran <ku...@gmail.com> on 2007/03/14 14:10:21 UTC
how to get approximate total matching
Hi.
I have more index directories (>6) all in GB,and searching my query with
single IndexSearcher to all indexes one after another.i.e. I create one
IndexSearcher for index1 and search over that.Finally I close that and
create new IndexSearcher for index2 and so on. If i get 200 total results
then i don't go to search other index directories and i print 200 results
and exit from search.
I need to get approximate total matching documents all over the indexes
without going to search in other indexes.
Please suggest me a easiest way to achieve this.
P.S: To avoid more memory usage and to reduce search timeI don't want to
search my query through all indexes if i got 200 results. MultiSearcher
create OOM error, so that I'm using single IndexSearcher.
Thanks in Advance
Senthil
Re: how to get approximate total matching
Posted by senthil kumaran <ku...@gmail.com>.
Hi,
Sorry for my delayed reply. I can't reproduce that OOM error with
MultiSearcher now.I'll try to reproduce that error and send you the details
that you asked.
I've tested to search my index with Multisearcher & separate
IndexSearcher.The search time by using Multisearcher is nearly equal the
time by separate IndexSearcher.But memory used is more for MultiSearcher.
My Index size is *7.4GB *having 17M documents. Total Search Time for
MultiSearcher = 1.1Secs and Max.Memory used = 50937920. And total search
time for Separate indexSearcher = 1.4Secs and Max.Memory used = 10039408.
Wild card query is used for search.
I need to control memory usage. Is there any possible ways to use
multisearcher with less memory usage?
(FYI- I don't use any sort.)
Thanks & Regards
Senthil
On 3/14/07, Erick Erickson <er...@gmail.com> wrote:
>
> How much memory are you allocating for your JVM? Because you're
> paying a huge search time penalty by opening and closing your
> searcher sequentially, it would be a good thing to not do this.
> But, as you say, if you're getting OOM errors, that's a problem.
>
> What is the total size of all your indexes? That would help folks
> give you better responses and perhaps suggest other ways of
> solving your problem.....
>
> Erick
>
> On 3/14/07, senthil kumaran <ku...@gmail.com> wrote:
> >
> > Hi.
> > I have more index directories (>6) all in GB,and searching my query
> > with
> > single IndexSearcher to all indexes one after another.i.e. I create one
> > IndexSearcher for index1 and search over that.Finally I close that and
> > create new IndexSearcher for index2 and so on. If i get 200 total
> results
> > then i don't go to search other index directories and i print 200
> results
> > and exit from search.
> > I need to get approximate total matching documents all over the
> > indexes
> > without going to search in other indexes.
> > Please suggest me a easiest way to achieve this.
> >
> > P.S: To avoid more memory usage and to reduce search timeI don't want to
> > search my query through all indexes if i got 200 results. MultiSearcher
> > create OOM error, so that I'm using single IndexSearcher.
> >
> >
> > Thanks in Advance
> > Senthil
> >
>
Re: how to get approximate total matching
Posted by Erick Erickson <er...@gmail.com>.
How much memory are you allocating for your JVM? Because you're
paying a huge search time penalty by opening and closing your
searcher sequentially, it would be a good thing to not do this.
But, as you say, if you're getting OOM errors, that's a problem.
What is the total size of all your indexes? That would help folks
give you better responses and perhaps suggest other ways of
solving your problem.....
Erick
On 3/14/07, senthil kumaran <ku...@gmail.com> wrote:
>
> Hi.
> I have more index directories (>6) all in GB,and searching my query
> with
> single IndexSearcher to all indexes one after another.i.e. I create one
> IndexSearcher for index1 and search over that.Finally I close that and
> create new IndexSearcher for index2 and so on. If i get 200 total results
> then i don't go to search other index directories and i print 200 results
> and exit from search.
> I need to get approximate total matching documents all over the
> indexes
> without going to search in other indexes.
> Please suggest me a easiest way to achieve this.
>
> P.S: To avoid more memory usage and to reduce search timeI don't want to
> search my query through all indexes if i got 200 results. MultiSearcher
> create OOM error, so that I'm using single IndexSearcher.
>
>
> Thanks in Advance
> Senthil
>
Re: how to get approximate total matching
Posted by Xiaocheng Luan <je...@yahoo.com>.
If I remember correctly, I once searched over 40G of indexes using multi-searcher with 512M max heap size, how much memory did you give the JVM?
Thanks,
Xiaocheng
senthil kumaran <ku...@gmail.com> wrote: Hi.
I have more index directories (>6) all in GB,and searching my query with
single IndexSearcher to all indexes one after another.i.e. I create one
IndexSearcher for index1 and search over that.Finally I close that and
create new IndexSearcher for index2 and so on. If i get 200 total results
then i don't go to search other index directories and i print 200 results
and exit from search.
I need to get approximate total matching documents all over the indexes
without going to search in other indexes.
Please suggest me a easiest way to achieve this.
P.S: To avoid more memory usage and to reduce search timeI don't want to
search my query through all indexes if i got 200 results. MultiSearcher
create OOM error, so that I'm using single IndexSearcher.
Thanks in Advance
Senthil
---------------------------------
Don't get soaked. Take a quick peek at the forecast
with theYahoo! Search weather shortcut.