You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Zhang, Lisheng" <Li...@BroadVision.com> on 2011/11/14 18:09:14 UTC

Upgrade lucene from 2.3.2 to 3.1.0

We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In Action" I learned
that we should "warm up" IndexSearcher and donot expect initial a few queries to be fast.
 
But due to our special app we cannot "warm up" (each query has to use a new IndexSearcher),
in lucene 2.3.2 this seems OK, Should we encounter obvious performance decrease in 3.1.0 
if keeping this way?
 
Thanks very much for helps, Lisheng

RE: Upgrade lucene from 2.3.2 to 3.1.0

Posted by "Zhang, Lisheng" <Li...@BroadVision.com>.
Yes, your point made very good sense, thanks very much for helps!

Lisheng

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Monday, November 14, 2011 5:53 PM
To: java-user@lucene.apache.org
Subject: Re: Upgrade lucene from 2.3.2 to 3.1.0


It's hard to estimate this in the abstract, I'm afraid you'll just
have to try it.

Best
Erick

On Mon, Nov 14, 2011 at 6:40 PM, Zhang, Lisheng
<Li...@broadvision.com> wrote:
> Our indexed data are around 200~300MB size (each folder), so it is
> still small?
>
> Could you roughly estimate how big the indexed data size (10GB?)
> needs to be, so that creating IndexReader each time could become a
> serious issue?
>
> Thanks very much for helps!
>
> Lisheng
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, November 14, 2011 3:20 PM
> To: java-user@lucene.apache.org
> Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0
>
>
>> Thanks for your reply!
>
> Thanks :-)
>
>> The reason why we cannot reuse IndexReader is that our server holds many
>> (>4000) independent index folders, each one corresponds to a separate URL. At
>> any time any folder can be queried, so we cannot hold all of them into
>> memory.
>
> So I expect the indexes very small. Reopening IndexReaders should also be fast with 3.x. I had another customer with a similar problem with no problems.
>
>> In lucene 2.3.2 query is fast even if we recreate IndexSearcher (therefore
>> Index-
>> Reader) each time. Yes I saw in source code even in 2.3.2 a lot of segement
>> information also are loaded into memory, and in 3.1.0 there seems a lot of
>> code changes in IndexReader initialization, so do we expect performance
>> change if we keep our way?
>>
>> Thanks very much for helps!
>>
>>
>> -----Original Message-----
>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Monday, November 14, 2011 2:29 PM
>> To: java-user@lucene.apache.org
>> Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0
>>
>>
>> Hi,
>>
>> > We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In
>> Action"
>> > I learned that we should "warm up" IndexSearcher and donot expect
>> > initial a few queries to be fast.
>>
>> This was always the case, not only since Lucene 2.9/3.0. You should warm your
>> searchers.
>>
>> > But due to our special app we cannot "warm up" (each query has to use
>> > a new IndexSearcher), in lucene 2.3.2 this seems OK, Should we
>> > encounter obvious performance decrease in 3.1.0 if keeping this way?
>>
>> This seems to be an XY problem. Why can you not reuse IndexReaders?
>>
>> Uwe
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Upgrade lucene from 2.3.2 to 3.1.0

Posted by Erick Erickson <er...@gmail.com>.
It's hard to estimate this in the abstract, I'm afraid you'll just
have to try it.

Best
Erick

On Mon, Nov 14, 2011 at 6:40 PM, Zhang, Lisheng
<Li...@broadvision.com> wrote:
> Our indexed data are around 200~300MB size (each folder), so it is
> still small?
>
> Could you roughly estimate how big the indexed data size (10GB?)
> needs to be, so that creating IndexReader each time could become a
> serious issue?
>
> Thanks very much for helps!
>
> Lisheng
>
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, November 14, 2011 3:20 PM
> To: java-user@lucene.apache.org
> Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0
>
>
>> Thanks for your reply!
>
> Thanks :-)
>
>> The reason why we cannot reuse IndexReader is that our server holds many
>> (>4000) independent index folders, each one corresponds to a separate URL. At
>> any time any folder can be queried, so we cannot hold all of them into
>> memory.
>
> So I expect the indexes very small. Reopening IndexReaders should also be fast with 3.x. I had another customer with a similar problem with no problems.
>
>> In lucene 2.3.2 query is fast even if we recreate IndexSearcher (therefore
>> Index-
>> Reader) each time. Yes I saw in source code even in 2.3.2 a lot of segement
>> information also are loaded into memory, and in 3.1.0 there seems a lot of
>> code changes in IndexReader initialization, so do we expect performance
>> change if we keep our way?
>>
>> Thanks very much for helps!
>>
>>
>> -----Original Message-----
>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Monday, November 14, 2011 2:29 PM
>> To: java-user@lucene.apache.org
>> Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0
>>
>>
>> Hi,
>>
>> > We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In
>> Action"
>> > I learned that we should "warm up" IndexSearcher and donot expect
>> > initial a few queries to be fast.
>>
>> This was always the case, not only since Lucene 2.9/3.0. You should warm your
>> searchers.
>>
>> > But due to our special app we cannot "warm up" (each query has to use
>> > a new IndexSearcher), in lucene 2.3.2 this seems OK, Should we
>> > encounter obvious performance decrease in 3.1.0 if keeping this way?
>>
>> This seems to be an XY problem. Why can you not reuse IndexReaders?
>>
>> Uwe
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Upgrade lucene from 2.3.2 to 3.1.0

Posted by "Zhang, Lisheng" <Li...@BroadVision.com>.
Our indexed data are around 200~300MB size (each folder), so it is 
still small?

Could you roughly estimate how big the indexed data size (10GB?) 
needs to be, so that creating IndexReader each time could become a
serious issue?

Thanks very much for helps!

Lisheng

-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, November 14, 2011 3:20 PM
To: java-user@lucene.apache.org
Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0


> Thanks for your reply!

Thanks :-)

> The reason why we cannot reuse IndexReader is that our server holds many
> (>4000) independent index folders, each one corresponds to a separate URL. At
> any time any folder can be queried, so we cannot hold all of them into
> memory.

So I expect the indexes very small. Reopening IndexReaders should also be fast with 3.x. I had another customer with a similar problem with no problems.

> In lucene 2.3.2 query is fast even if we recreate IndexSearcher (therefore
> Index-
> Reader) each time. Yes I saw in source code even in 2.3.2 a lot of segement
> information also are loaded into memory, and in 3.1.0 there seems a lot of
> code changes in IndexReader initialization, so do we expect performance
> change if we keep our way?
> 
> Thanks very much for helps!
> 
> 
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, November 14, 2011 2:29 PM
> To: java-user@lucene.apache.org
> Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0
> 
> 
> Hi,
> 
> > We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In
> Action"
> > I learned that we should "warm up" IndexSearcher and donot expect
> > initial a few queries to be fast.
> 
> This was always the case, not only since Lucene 2.9/3.0. You should warm your
> searchers.
> 
> > But due to our special app we cannot "warm up" (each query has to use
> > a new IndexSearcher), in lucene 2.3.2 this seems OK, Should we
> > encounter obvious performance decrease in 3.1.0 if keeping this way?
> 
> This seems to be an XY problem. Why can you not reuse IndexReaders?
> 
> Uwe
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Upgrade lucene from 2.3.2 to 3.1.0

Posted by Uwe Schindler <uw...@thetaphi.de>.
> Thanks for your reply!

Thanks :-)

> The reason why we cannot reuse IndexReader is that our server holds many
> (>4000) independent index folders, each one corresponds to a separate URL. At
> any time any folder can be queried, so we cannot hold all of them into
> memory.

So I expect the indexes very small. Reopening IndexReaders should also be fast with 3.x. I had another customer with a similar problem with no problems.

> In lucene 2.3.2 query is fast even if we recreate IndexSearcher (therefore
> Index-
> Reader) each time. Yes I saw in source code even in 2.3.2 a lot of segement
> information also are loaded into memory, and in 3.1.0 there seems a lot of
> code changes in IndexReader initialization, so do we expect performance
> change if we keep our way?
> 
> Thanks very much for helps!
> 
> 
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Monday, November 14, 2011 2:29 PM
> To: java-user@lucene.apache.org
> Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0
> 
> 
> Hi,
> 
> > We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In
> Action"
> > I learned that we should "warm up" IndexSearcher and donot expect
> > initial a few queries to be fast.
> 
> This was always the case, not only since Lucene 2.9/3.0. You should warm your
> searchers.
> 
> > But due to our special app we cannot "warm up" (each query has to use
> > a new IndexSearcher), in lucene 2.3.2 this seems OK, Should we
> > encounter obvious performance decrease in 3.1.0 if keeping this way?
> 
> This seems to be an XY problem. Why can you not reuse IndexReaders?
> 
> Uwe
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Upgrade lucene from 2.3.2 to 3.1.0

Posted by "Zhang, Lisheng" <Li...@BroadVision.com>.
Thanks for your reply!

The reason why we cannot reuse IndexReader is that our server holds many (>4000)
independent index folders, each one corresponds to a separate URL. At any time
any folder can be queried, so we cannot hold all of them into memory.

In lucene 2.3.2 query is fast even if we recreate IndexSearcher (therefore Index-
Reader) each time. Yes I saw in source code even in 2.3.2 a lot of segement 
information also are loaded into memory, and in 3.1.0 there seems a lot of code 
changes in IndexReader initialization, so do we expect performance change if we 
keep our way?

Thanks very much for helps!


-----Original Message-----
From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: Monday, November 14, 2011 2:29 PM
To: java-user@lucene.apache.org
Subject: RE: Upgrade lucene from 2.3.2 to 3.1.0


Hi,

> We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In Action"
> I learned that we should "warm up" IndexSearcher and donot expect initial a
> few queries to be fast.

This was always the case, not only since Lucene 2.9/3.0. You should warm your searchers.

> But due to our special app we cannot "warm up" (each query has to use a new
> IndexSearcher), in lucene 2.3.2 this seems OK, Should we encounter obvious
> performance decrease in 3.1.0 if keeping this way?

This seems to be an XY problem. Why can you not reuse IndexReaders?

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Upgrade lucene from 2.3.2 to 3.1.0

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

> We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In Action"
> I learned that we should "warm up" IndexSearcher and donot expect initial a
> few queries to be fast.

This was always the case, not only since Lucene 2.9/3.0. You should warm your searchers.

> But due to our special app we cannot "warm up" (each query has to use a new
> IndexSearcher), in lucene 2.3.2 this seems OK, Should we encounter obvious
> performance decrease in 3.1.0 if keeping this way?

This seems to be an XY problem. Why can you not reuse IndexReaders?

Uwe


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Upgrade lucene from 2.3.2 to 3.1.0

Posted by "Zhang, Lisheng" <Li...@BroadVision.com>.
Thanks very much for reminding, yes, we always rebuild index when upgrading lucene.

-----Original Message-----
From: earlhood@gmail.com [mailto:earlhood@gmail.com]On Behalf Of Earl
Hood
Sent: Monday, November 14, 2011 11:54 AM
To: java-user@lucene.apache.org
Subject: Re: Upgrade lucene from 2.3.2 to 3.1.0


On Mon, Nov 14, 2011 at 11:09 AM, Zhang, Lisheng
<Li...@broadvision.com> wrote:
> We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In Action" I learned
> that we should "warm up" IndexSearcher and donot expect initial a few queries to be fast.

Make sure to QA things first.  When we went from 2.4.1 to 3.0.2, we had to
rebuild our indexes since search behavior was not working as expected
with index data based on 2.4.1.

YMMV,

--ewh

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Upgrade lucene from 2.3.2 to 3.1.0

Posted by Earl Hood <ea...@earlhood.com>.
On Mon, Nov 14, 2011 at 11:09 AM, Zhang, Lisheng
<Li...@broadvision.com> wrote:
> We plan to upgrade lucene from 2.3.2 to 3.1.0, from reading "Lucene In Action" I learned
> that we should "warm up" IndexSearcher and donot expect initial a few queries to be fast.

Make sure to QA things first.  When we went from 2.4.1 to 3.0.2, we had to
rebuild our indexes since search behavior was not working as expected
with index data based on 2.4.1.

YMMV,

--ewh

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org