You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by "paulirwin (via GitHub)" <gi...@apache.org> on 2026/05/01 18:00:52 UTC
[PR] Fix ArgumentOutOfRangeException in IrishLowerCaseFilter span slicing, #1150 [lucenenet]
paulirwin opened a new pull request, #1286:
URL: https://github.com/apache/lucenenet/pull/1286
- [X] You've read the [Contributor Guide](https://github.com/apache/lucenenet/blob/master/CONTRIBUTING.md) and [Code of Conduct](https://www.apache.org/foundation/policies/conduct.html).
- [X] You've included unit or integration tests for your change, where applicable.
- [X] You've included inline docs for your change, where applicable.
- [X] There's an open issue for the PR that you are making. If you'd like to propose a change, please [open an issue](https://github.com/apache/lucenenet/issues/new/choose) to discuss the change or find an existing issue.
<!-- Once all that is done, you're ready to go. Open the PR with the content below. -->
Summary of the changes (Less than 80 chars)
Fixes #1150
## Description
IrishLowerCaseFilter was incorrectly calculating the span length for the lowercase operation. When the n-/t-prothesis path runs (inserting a hyphen after 'n' or 't'), idx becomes 2 and the span length should be (chLen - idx), not chLen. This caused out-of-bounds span creation.
The fix calculates spanLen = chLen - idx and uses this corrected length for both the source and destination spans, as well as the spare buffer allocation.
Added a regression test using a custom TightBufferTokenStream that writes directly into CharTermAttribute.termBuffer to bypass the Oversize over-allocation all public tokenizer paths use, producing a buffer sized to exactly the post-prothesis term length. This reliably triggers the ArgumentOutOfRangeException on the buggy code.
AI: co-authored with Claude Code Opus 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ArgumentOutOfRangeException in IrishLowerCaseFilter span slicing, #1150 [lucenenet]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1286:
URL: https://github.com/apache/lucenenet/pull/1286#discussion_r3174524030
##########
src/Lucene.Net.Tests.Analysis.Common/Analysis/Ga/TestIrishLowerCaseFilter.cs:
##########
@@ -49,5 +51,111 @@ public virtual void TestEmptyTerm()
});
CheckOneTerm(a, "", "");
}
+
+ /// <summary>
+ /// Test that prothesis output is correct across all upper vowels and fadas,
+ /// and for both n- and t- prefixes.
+ /// </summary>
+ [Test, LuceneNetSpecific] // Issue #1150
+ public virtual void TestProthesisCorrectness()
+ {
+ Analyzer a = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
+ {
+ Tokenizer tokenizer = new KeywordTokenizer(reader);
+ return new TokenStreamComponents(tokenizer, new IrishLowerCaseFilter(tokenizer));
+ });
+
+ // Plain upper vowels
+ CheckOneTerm(a, "nAthair", "n-athair");
+ CheckOneTerm(a, "nEan", "n-ean");
+ CheckOneTerm(a, "nIasc", "n-iasc");
+ CheckOneTerm(a, "nOiche", "n-oiche");
+ CheckOneTerm(a, "nUll", "n-ull");
+ CheckOneTerm(a, "tAthair", "t-athair");
+ CheckOneTerm(a, "tEan", "t-ean");
+ CheckOneTerm(a, "tIasc", "t-iasc");
+ CheckOneTerm(a, "tOiche", "t-oiche");
+ CheckOneTerm(a, "tUll", "t-ull");
+
+ // Accented vowels (fadas)
+ CheckOneTerm(a, "nÁit", "n-áit");
+ CheckOneTerm(a, "nÉan", "n-éan");
+ CheckOneTerm(a, "nÍoc", "n-íoc");
+ CheckOneTerm(a, "nÓg", "n-óg");
+ CheckOneTerm(a, "nÚll", "n-úll");
+ CheckOneTerm(a, "tÁit", "t-áit");
+ CheckOneTerm(a, "tÉan", "t-éan");
+ CheckOneTerm(a, "tÍoc", "t-íoc");
+ CheckOneTerm(a, "tÓg", "t-óg");
+ CheckOneTerm(a, "tÚll", "t-úll");
+ }
+
+ /// <summary>
+ /// Regression test for issue #1150: ArgumentOutOfRangeException in IrishLowerCaseFilter.
+ ///
+ /// The bug: when the n-/t-prothesis path runs, idx is set to 2 but the span length was
+ /// left as chLen (full term length) instead of chLen - idx. This creates a span that reads
+ /// chArray[idx .. idx+chLen], which exceeds the buffer when the buffer is sized just large
+ /// enough for chLen chars.
+ ///
+ /// We reproduce by using a custom TokenStream that writes directly into the internal
+ /// CharTermAttributeImpl.termBuffer field, bypassing the Oversize over-allocation that
Review Comment:
The XML doc comment refers to `CharTermAttributeImpl.termBuffer`, but the type in this codebase is `CharTermAttribute` (defined in CharTermAttributeImpl.cs) and the `termBuffer` field is on `CharTermAttribute`. Updating the wording will avoid confusion for future readers.
```suggestion
/// CharTermAttribute.termBuffer field, bypassing the Oversize over-allocation that
```
##########
src/Lucene.Net.Tests.Analysis.Common/Analysis/Ga/TestIrishLowerCaseFilter.cs:
##########
@@ -49,5 +51,111 @@ public virtual void TestEmptyTerm()
});
CheckOneTerm(a, "", "");
}
+
+ /// <summary>
+ /// Test that prothesis output is correct across all upper vowels and fadas,
+ /// and for both n- and t- prefixes.
+ /// </summary>
+ [Test, LuceneNetSpecific] // Issue #1150
+ public virtual void TestProthesisCorrectness()
+ {
+ Analyzer a = Analyzer.NewAnonymous(createComponents: (fieldName, reader) =>
+ {
+ Tokenizer tokenizer = new KeywordTokenizer(reader);
+ return new TokenStreamComponents(tokenizer, new IrishLowerCaseFilter(tokenizer));
+ });
+
+ // Plain upper vowels
+ CheckOneTerm(a, "nAthair", "n-athair");
+ CheckOneTerm(a, "nEan", "n-ean");
+ CheckOneTerm(a, "nIasc", "n-iasc");
+ CheckOneTerm(a, "nOiche", "n-oiche");
+ CheckOneTerm(a, "nUll", "n-ull");
+ CheckOneTerm(a, "tAthair", "t-athair");
+ CheckOneTerm(a, "tEan", "t-ean");
+ CheckOneTerm(a, "tIasc", "t-iasc");
+ CheckOneTerm(a, "tOiche", "t-oiche");
+ CheckOneTerm(a, "tUll", "t-ull");
+
+ // Accented vowels (fadas)
+ CheckOneTerm(a, "nÁit", "n-áit");
+ CheckOneTerm(a, "nÉan", "n-éan");
+ CheckOneTerm(a, "nÍoc", "n-íoc");
+ CheckOneTerm(a, "nÓg", "n-óg");
+ CheckOneTerm(a, "nÚll", "n-úll");
+ CheckOneTerm(a, "tÁit", "t-áit");
+ CheckOneTerm(a, "tÉan", "t-éan");
+ CheckOneTerm(a, "tÍoc", "t-íoc");
+ CheckOneTerm(a, "tÓg", "t-óg");
+ CheckOneTerm(a, "tÚll", "t-úll");
+ }
+
+ /// <summary>
+ /// Regression test for issue #1150: ArgumentOutOfRangeException in IrishLowerCaseFilter.
+ ///
+ /// The bug: when the n-/t-prothesis path runs, idx is set to 2 but the span length was
+ /// left as chLen (full term length) instead of chLen - idx. This creates a span that reads
+ /// chArray[idx .. idx+chLen], which exceeds the buffer when the buffer is sized just large
+ /// enough for chLen chars.
+ ///
+ /// We reproduce by using a custom TokenStream that writes directly into the internal
+ /// CharTermAttributeImpl.termBuffer field, bypassing the Oversize over-allocation that
+ /// all public paths go through, so the buffer is sized to exactly the post-prothesis
+ /// term length. The buggy code then asks for chArray[2..2+chLen] which is 2 chars past
+ /// the end.
+ /// </summary>
+ [Test, LuceneNetSpecific] // Issue #1150
+ public virtual void TestProthesis_SpanBoundaryThrows()
+ {
+ // "nAthair" has length 7; after prothesis it becomes "n-athair" (length 8).
+ // We pre-size the buffer to exactly 8 so that chArray[2..2+8] = chArray[2..10]
+ // exceeds the buffer and throws ArgumentOutOfRangeException on the buggy code.
+ const string input = "nAthair";
+ const string expected = "n-athair";
+
+ var source = new TightBufferTokenStream(input);
+ var filter = new IrishLowerCaseFilter(source);
+ AssertTokenStreamContents(filter, new[] { expected });
Review Comment:
The test name `TestProthesis_SpanBoundaryThrows` is misleading: the test asserts successful output and should pass without throwing. Consider renaming it to reflect that this is a regression test ensuring the prothesis span boundary does *not* throw (or that it produces the expected token with a tight buffer).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix ArgumentOutOfRangeException in IrishLowerCaseFilter span slicing, #1150 [lucenenet]
Posted by "paulirwin (via GitHub)" <gi...@apache.org>.
paulirwin merged PR #1286:
URL: https://github.com/apache/lucenenet/pull/1286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org