[PR] Back CharArraySet and CharArrayDictionary with J2N collections, #1478 [lucenenet]

["paulirwin (via GitHub)" <gi...@apache.org>]

paulirwin opened a new pull request, #1496:
URL: https://github.com/apache/lucenenet/pull/1496

   - [X] You've read the [Contributor Guide](https://github.com/apache/lucenenet/blob/master/CONTRIBUTING.md) and [Code of Conduct](https://www.apache.org/foundation/policies/conduct.html).
   - [X] You've included unit or integration tests for your change, where applicable.
   - [X] You've included inline docs for your change, where applicable.
   - [X] There's an open issue for the PR that you are making. If you'd like to propose a change, please [open an issue](https://github.com/apache/lucenenet/issues/new/choose) to discuss the change or find an existing issue.
   
   Back CharArraySet and CharArrayDictionary with J2N collections
   
   Fixes #1478
   
   ## Description
   
   The previous implementation was a custom open-addressing hash table of `char[]` keys (about 6,000 lines across `CharArraySet.cs` and `CharArrayMap.cs`). The set was built on the dictionary through internal interfaces, and keys could be passed as `char[]`, `char[]` with offset/length, `ICharSequence`, or any object converted with `ToString()`. There was no `ReadOnlySpan<char>` support.
   
   This PR backs both types with J2N collections and uses J2N's `SpanAlternateLookup<char>` for allocation-free lookups by `ReadOnlySpan<char>`. This works on all target frameworks.
   
   ### Case folding
   
   The issue suggested `OrdinalIgnoreCase`, but that folds to uppercase and matches a different set of strings than upstream Lucene. For example, the Kelvin sign and Ohm sign stop matching `k` and omega, and dotless `ı`, long `ſ` and final `ς` start matching `i`, `s` and `σ`. To avoid changing behavior, case-insensitive collections use a new internal `CharArrayComparer` that implements J2N's `ISpanAlternateEqualityComparer<char, string>`:
   
   - Keys match if their invariant lowercase per code point matches, which is the same as Java's `Character.toLowerCase` and the previous implementation.
   - Keys are stored lowercased, as in upstream Lucene, so enumeration, `ToString()` and `Equals()` return the lowercased form.
   - Folding goes through `CharacterUtils`, so the LUCENE_30 (Java 4) surrogate behavior is preserved.
   - There is a fast path for the common case where the probe is already lowercase, and an inline ASCII path for hashing.
   
   ### Testing approach
   
   The first commit adds `TestCharArrayCollectionsBehavior`, 70 tests that only use APIs shared by the old and new implementations. They were run against the old implementation before any production code changed. They cover case folding (including the edge characters above, supplementary characters and lone surrogates), randomized comparisons against a reference model at the current version and LUCENE_30, stored-key form, copy, read-only, null and set-operation semantics. Five tests that cover bugs in the old implementation were ignored in that commit, after confirming each one failed. All 70 pass on the new implementation.
   
   ### Breaking API changes
   
   - Both types are now `sealed`.
   - Removed the `char[]`, `char[]` + offset/length, `ICharSequence` and generic/object (`ToString()` conversion) overloads, and added `ReadOnlySpan<char>` overloads. Whole `char[]` arguments still compile because they convert implicitly to `ReadOnlySpan<char>`.
   - Replaced `CharArraySet.Copy()`, `CharArrayDictionary.Copy()` and the `ToCharArraySet()` / `ToCharArrayDictionary()` instance methods with copy constructors. The extension methods remain, closed over `string`.
   - Removed the obsolete `EMPTY_SET`, `UnmodifiableSet()`, `UnmodifiableMap()` and `ContainsAll()` members.
   - `CharArrayDictionary<TValue>.Keys` is now a read-only `KeyCollection` instead of a `CharArraySet`, so `Keys.Clear()` throws instead of clearing the dictionary.
   - Removed the dictionary enumerator's `CurrentKey`, `CurrentKeyString`, `CurrentKeyCharSequence`, `CurrentValue` and `SetValue()` members. Use `Current`, or the indexer to replace a value; replacing the value of an existing key does not invalidate enumerators.
   - `Clear()` no longer invalidates enumerators, which is consistent with `Dictionary<TKey, TValue>` in .NET Core 3.0+.
   - `AsReadOnly()` now returns a live view instead of a snapshot.
   - Removed the `StopFilter.MakeStopSet<T1>()` overloads in favor of `IList<string>`.
   
   ### Bugs fixed
   
   - `ToCharArrayDictionary(matchVersion, ignoreCase)` returned an empty dictionary.
   - `ValueCollection.Contains()` always returned `false`.
   - `CharArrayDictionary<TValue>.GetHashCode()` depended on entry order and table capacity.
   - The `ToCharArraySet(matchVersion)` extension dropped `ignoreCase` for a `CharArraySet` source.
   - Read-only views kept a stale `Count`.
   
   ### Other changes
   
   - Migrated the 42 production call sites that used `(char[], int, int)` overloads (`StopFilter`, `KeepWordFilter`, `CommonGramsFilter`, the compound filters, `GreekStemmer`, `KStemmer`, `HTMLStripCharFilter`, `SuggestStopFilter`, etc.) to `buffer.AsSpan(offset, length)`.
   - Removed the internal `CharacterUtils.CodePointAt(ICharSequence, int)` members, which only existed for the old `CharArrayDictionary<TValue>`.
   - Ported tests keep their upstream names. Where an upstream assertion no longer applies, a `LUCENENET` comment explains why. Tests for removed `char[]` / `ICharSequence` / `object` overloads were removed.
   - 
   ### Performance
   
   A quick microbenchmark of `CharArraySet.Contains` against `master` (English stop words, term-buffer-style `char[]` slices, net10.0, Apple Silicon). Both versions allocate nothing per lookup:
   
   | Lookup | Before | After |
   |---|---|---|
   | Case sensitive | 4.8 ns | 4.5 ns |
   | Ignore case, lowercase input | 23.8 ns | 8.2 ns |
   | Ignore case, title-case input | 23.5 ns | 13.3 ns |
   
   AI: written by Claude Code, Opus 5.5.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Back CharArraySet and CharArrayDictionary with J2N collections, #1478 [lucenenet]

["NightOwl888 (via GitHub) via dev" <de...@lucenenet.apache.org>]

NightOwl888 merged PR #1496:
URL: https://github.com/apache/lucenenet/pull/1496


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org