You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stuart Goldberg <sg...@fixflyer.com> on 2018/04/17 15:52:27 UTC
Help! - Max Segment name reached
We have an index that has run into this bug:
https://issues.apache.org/jira/browse/LUCENE-7999
Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4 and
cannot upgrade.
By looking at the code it seems that the last segment number counter is
persisted in segment_h. When creating a new segment, it names the segment
based on the persisted counter. If this counter is larger than
Integer.MAX_VALUE how can we recover this index.
Is there anything we can do?
Stuart M Goldberg
Senior Vice President of Software Develpment
FIX Flyer LLC
<http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS
E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
Re: Help! - Max Segment name reached
Posted by Michael McCandless <lu...@mikemccandless.com>.
Well I think as time goes on we'll see more and more people running into it
;)
But you really need to commit at a surprisingly high rate, and have a
surprisingly long lived index, to overflow the int that holds the segment
number. E.g. if you commit once per second, it should take ~68 years to
overflow the int.
Mike McCandless
http://blog.mikemccandless.com
On Tue, Apr 17, 2018 at 4:04 PM, Stuart Goldberg <sg...@fixflyer.com>
wrote:
> Thanks, I will try that.
>
> Why haven't more people run into this issue? The next segment number is
> persisted, so if an index has a long life it should eventually run into
> this problem.
>
> Stuart M Goldberg
> Senior Vice President of Software Develpment
> FIX Flyer LLC
> http://www.FIXFlyer.com/
> NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
> RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
> WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
> DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS
> INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
> RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS
> E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
> -----Original Message-----
> From: Uwe Schindler <uw...@thetaphi.de>
> Sent: Tuesday, April 17, 2018 4:02 PM
> To: java-user@lucene.apache.org
> Subject: Re: Help! - Max Segment name reached
>
> Hi,
>
> Create a new empty index in a new directory and use addIndex() using the
> other directory with the broken index.
>
> This will copy all segments but renumber them.
>
> Uwe
>
> Am April 17, 2018 3:52:27 PM UTC schrieb Stuart Goldberg <
> sgoldberg@fixflyer.com>:
> >We have an index that has run into this bug:
> >https://issues.apache.org/jira/browse/LUCENE-7999
> >
> >
> >
> >Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4
> >and cannot upgrade.
> >
> >
> >
> >By looking at the code it seems that the last segment number counter is
> >persisted in segment_h. When creating a new segment, it names the
> >segment based on the persisted counter. If this counter is larger than
> >Integer.MAX_VALUE how can we recover this index.
> >
> >
> >
> >Is there anything we can do?
> >
> >
> >
> >Stuart M Goldberg
> >
> >Senior Vice President of Software Develpment FIX Flyer LLC
> ><http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
> >
> >NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
> >RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
> >WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
> >DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
> >THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE
> >INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE
> >DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
> >
> >
>
> --
> Uwe Schindler
> Achterdiek 19, 28357 Bremen
> https://www.thetaphi.de
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Help! - Max Segment name reached
Posted by Stuart Goldberg <sg...@fixflyer.com>.
Thanks, I will try that.
Why haven't more people run into this issue? The next segment number is persisted, so if an index has a long life it should eventually run into this problem.
Stuart M Goldberg
Senior Vice President of Software Develpment
FIX Flyer LLC
http://www.FIXFlyer.com/
NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING, DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
-----Original Message-----
From: Uwe Schindler <uw...@thetaphi.de>
Sent: Tuesday, April 17, 2018 4:02 PM
To: java-user@lucene.apache.org
Subject: Re: Help! - Max Segment name reached
Hi,
Create a new empty index in a new directory and use addIndex() using the other directory with the broken index.
This will copy all segments but renumber them.
Uwe
Am April 17, 2018 3:52:27 PM UTC schrieb Stuart Goldberg <sg...@fixflyer.com>:
>We have an index that has run into this bug:
>https://issues.apache.org/jira/browse/LUCENE-7999
>
>
>
>Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4
>and cannot upgrade.
>
>
>
>By looking at the code it seems that the last segment number counter is
>persisted in segment_h. When creating a new segment, it names the
>segment based on the persisted counter. If this counter is larger than
>Integer.MAX_VALUE how can we recover this index.
>
>
>
>Is there anything we can do?
>
>
>
>Stuart M Goldberg
>
>Senior Vice President of Software Develpment FIX Flyer LLC
><http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
>
>NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
>RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
>DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
>THIS INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE
>INTENDED RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE
>DELETE THIS E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Help! - Max Segment name reached
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
Create a new empty index in a new directory and use addIndex() using the other directory with the broken index.
This will copy all segments but renumber them.
Uwe
Am April 17, 2018 3:52:27 PM UTC schrieb Stuart Goldberg <sg...@fixflyer.com>:
>We have an index that has run into this bug:
>https://issues.apache.org/jira/browse/LUCENE-7999
>
>
>
>Although this is reported to be fixed in Lucene 7.2, we are at 4.10.4
>and
>cannot upgrade.
>
>
>
>By looking at the code it seems that the last segment number counter is
>persisted in segment_h. When creating a new segment, it names the
>segment
>based on the persisted counter. If this counter is larger than
>Integer.MAX_VALUE how can we recover this index.
>
>
>
>Is there anything we can do?
>
>
>
>Stuart M Goldberg
>
>Senior Vice President of Software Develpment
>FIX Flyer LLC
> <http://www.FIXFlyer.com/> http://www.FIXFlyer.com/
>
>NOTICE TO RECIPIENT: THIS E- MAIL IS MEANT ONLY FOR THE INTENDED
>RECIPIENT(S) OF THE TRANSMISSION, AND CONTAINS CONFIDENTIAL INFORMATION
>WHICH IS PROPRIETARY TO FIX FLYER LLC ANY UNAUTHORIZED USE, COPYING,
>DISTRIBUTION, OR DISSEMINATION IS STRICTLY PROHIBITED. ALL RIGHTS TO
>THIS
>INFORMATION IS RESERVED BY FIX FLYER LLC. IF YOU ARE NOT THE INTENDED
>RECIPIENT, PLEASE CONTACT THE SENDER BY REPLY EMAIL AND PLEASE DELETE
>THIS
>E-MAIL FROM YOUR SYSTEM AND DESTROY ANY COPIES.
>
>
--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de