You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org> on 2026/08/02 14:03:57 UTC

[PR] GITHUB#7820: report which segment's .si could not be read [lucene]

serhiy-bzhezytskyy opened a new pull request, #16476:
URL: https://github.com/apache/lucene/pull/16476

   ### Description
   
   `CheckIndex` reports that a commit point could not be read, but not *which* segment is at fault:
   
   ```
   ERROR: could not read latest commit point from segments file "segments_3" in directory
   ```
   
   The information exists at the point of failure — a deleted `.si` arrives as a `NoSuchFileException` carrying the full path, a truncated one as a suppressed `EOFException` naming the `MemorySegmentIndexInput` — but `SegmentInfos#parseSegmentInfos` lets the codec's exception propagate untyped, so nothing downstream can tell which segment it was. This is the November 2023 comment on #7820:
   
   > the exception "does not make it clear which segment(s) are broken"
   
   After this change:
   
   ```
   ERROR: could not read segment "_1" referenced by the latest commit point in segments file
   "segments_3": its _1.si is missing or corrupt
   ```
   
   ### Changes
   
   The first commit is @gokaai's from #12872, rebased onto `main` with authorship intact. It introduces `CorruptSegmentInfoException` and throws it from `SegmentInfos` when a segment's `.si` cannot be read. Two hunks needed adjusting for `main`: `IOContext.READ` became `IOContext.READONCE`, and the wildcard import was replaced with explicit ones, which the ecj lint requires.
   
   The second commit finishes the two things asked for in that PR's review but not done there:
   
   - **The root cause was dropped.** The throw site caught the codec's failure and called the three-argument constructor, discarding it. @mikemccand asked on the PR: *"Can we somehow return the root cause exception here and include it in `CheckIndexException`"*. It is now passed, and the catch is `Exception | AssertionError` rather than `Exception`, per the same review: *"Corruption in `.si` can result in exotic exceptions"*.
   - **`segmentName` was unreachable.** It was package-private with no accessor, so nothing outside `org.apache.lucene.index` could read the name the exception exists to carry, and no test could assert on it. `getSegmentName()` is added, the six constructors are reduced to the one that is used, and both arguments are `Objects.requireNonNull`.
   
   On top of that, `CheckIndex` records the name in `Status#brokenSegmentName` and puts it in the message, and `TestCheckIndex#testCorruptSegmentInfoNamesTheSegment` covers a deleted and a truncated `.si`, asserting the segment name, the cause's type, and that the cause names `_1.si`. #12872 had no test for the exception it added.
   
   ### Relationship to #12872 and #16474
   
   #12872 has been open since 2023-12 and is no longer mergeable; the last human comment there is from 2024-02, after a `git` digression, and @mikemccand's *"I'll try to review soon!"* never got a follow-up. The pain it addresses is still there, so this carries it forward rather than leaving it. If @gokaai would rather finish it there, close this one — the point is the fix landing, not where.
   
   #16474 is the other half of the same thread: `exorciseIndex` throws `NullPointerException` instead of refusing when `Status.newSegments` is null. That is the piece @gokaai explicitly deferred:
   
   > Will create a new commit (or issue?) to add in fixes and unit tests for `exorciseIndex`
   
   The two are independent and can land in either order.
   
   ### Verification
   
   - `testCorruptSegmentInfoNamesTheSegment` fails without the fix, and fails again if the root cause is replaced with a synthetic one — so it checks the cause rather than just its presence
   - `:lucene:core:test` for `TestCheckIndex`, `TestTransactions`, `SegmentInfos*` and `IndexWriter*` passes (362 tests)
   - `:lucene:core:check` and `tidy` pass
   
   Verified on `main` only.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5291994480

   yes, i tested — the code cherry-picks onto `branch_10x` cleanly, only the CHANGES entry conflicts since on 10.x it goes in a different section.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand merged PR #16476:
URL: https://github.com/apache/lucene/pull/16476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5326785902

   Hi @romseygeek — tagging you since you've picked up my earlier PRs here and know the context. This one has been sitting a couple of weeks; it's synced to main now, and the workflow runs need releasing before CI will report on the new head. Happy to do whatever is needed to get it merged — what else can I improve or fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5518539531

   This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5242756232

   Thanks @mikemccand — both nits are addressed in `8935d53a`: the `SegmentInfo` declaration is now shrink-wrapped inside the loop just above the `try`, and the test uses the `== false` form. CI is green.
   
   Anything else I should improve here?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #16476:
URL: https://github.com/apache/lucene/pull/16476#discussion_r3711767422


##########
lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java:
##########
@@ -388,14 +389,25 @@ private static void parseSegmentInfos(
     }
 
     long totalDocs = 0;
-
+    SegmentInfo info;

Review Comment:
   Could we move this inside the `for` loop, just above the new `try {`?  (Shrink-wrap the scoping)



##########
lucene/core/src/java/org/apache/lucene/index/CheckIndex.java:
##########
@@ -991,7 +1009,7 @@ private Status.SegmentInfoStatus testSegment(
 
     final Version version = info.info.getVersion();
     if (info.info.maxDoc() <= 0) {
-      throw new CheckIndexException(" illegal number of documents: maxDoc=" + info.info.maxDoc());

Review Comment:
   Thank you for the clean-as-you-go.



##########
lucene/core/src/test/org/apache/lucene/index/TestTransactions.java:
##########
@@ -206,7 +206,10 @@ public void doWork() throws Throwable {
         } catch (Exception e) {
           // can be rethrown as RuntimeException if it happens during a close listener
           if (!e.getMessage().contains("on purpose")) {
-            throw e;
+            // Caught "on-purpose" IOException can be rethrown as CorruptSegmentInfoException
+            if (!(e instanceof CorruptSegmentInfoException)) {

Review Comment:
   `== false`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5997279039

   Thanks @serhiy-bzhezytskyy -- I merged to `main` for 11.0.0, and backported to `branch_10x` for 10.6.0!  Very exciting to finally improve the error message from `CheckIndex`.  Error message specifics especially matter when you are struggling with a corrupt index.  Thank you for attending to such detail.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5997506959

   @mikemccand thank you a lot!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5178127109

   Thank you for resurrecting this dying-on-the-vine PR @serhiy-bzhezytskyy!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] GITHUB#7820: report which segment's .si could not be read [lucene]

Posted by "serhiy-bzhezytskyy (via GitHub)" <gi...@apache.org>.
serhiy-bzhezytskyy commented on PR #16476:
URL: https://github.com/apache/lucene/pull/16476#issuecomment-5680818474

   @mikemccand rebased on main again today (all green). Both nits and the backport question above are answered -- anything else before this can merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org