[jira] [Updated] (LUCENENET-616) Make Collections from Lucene.Net.Support into a 1st Class Feature

["Shad Storhaug (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENENET-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shad Storhaug updated LUCENENET-616:
------------------------------------
    Description: 
The collection types in {{Lucene.Net.Support}} were originally sourced to support Lucene.Net itself. While they were made public, they were not considered to be features that would be used by anyone except for advanced users.

However, it has become clear by user reports that some parts of Lucene's design require specialized collections that don't exist in the .NET Framework in order to properly function (see [LUCENENET-612|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-612] and [LUCENENET-615|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-615]. .NET users are generally not familiar with these specialized collection types and assume that when {{IDictionary<TKey, TValue>}} is required by an API that using {{Dictionary<TKey, TValue>}} is their best choice. We need to improve documentation and increase visibility of the specialized collection types in order to help them along.

Some of the existing collections are composed of other nested collection objects and should be replaced with a lower-level implementation, if possible. Since these are breaking API changes, they should be done before the official release of Lucene.Net 4.8.0.

Additionally, many parts of Lucene.NET expect the collections to:
 * Be structurally equal
 * Format their contents in the ToString() method

The safest and most thorough way to achieve this is to replace the usage of all built-in .NET collections with collections that we own that implement this functionality. .NET provides interfaces to help achieve this:
 * {{IStructuralEquatable}}
 * {{IFormattable}}

But neither interface is implemented in any of the built-in collections in .NET.
h2. *Requirements for Lucene.NET Collections*

We must have the following collections for internal use within Lucene.NET, and each should also be made available to end users.
 * {{J2N.Collections.Generic.List<T>}}
 ** Subclass of {{System.Collections.Generic.List<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.HashSet<T>}}
 ** Subclass of {{System.Collections.Generic.List<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.SortedSet<T>}}
 ** Subclass of {{System.Collections.Generic.SortedSet<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.SortedDictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.SortedDictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.LinkedDictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Preserves insertion order across adds/deletes
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.PriorityQueue<T>}}
 ** Migrate {{Lucene.Net.Support.PriorityQueue<T>}} implementation, but clean up the API for .NET (AddAll() > AddRange(), etc.)

 * {{J2N.Collections.Generic.IdentityDictionary<TKey, TValue>}} (See {{Lucene.Net.Support.IdentityHashMap<TKey, TValue>}})
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.IdentityHashSet<T>}} (See {{Lucene.Net.Support.IdentityHashSet<T>}})
 ** Subclass of {{System.Collections.Generic.HashSet<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{ConcurrentHashSet<T>}} - (See LUCENENET-634)

We should have the following collections for internal use within Lucene.NET.
 * {{J2N.Collections.Generic.LinkedHashSet<T>}}
 ** Same interface as {{System.Collections.Generic.List<T>}}
 ** Preserves insertion order across adds/deletes
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Runtime.CompilerServices.ConditionalWeakTable<TKey, TValue>}} (See LUCENENET-636)

h3. Special Cases

There are a few uses where the above collections won't suffice because they use low-level methods that don't exist in .NET collections. However, these are closed systems and we don't have to worry about supporting end users in these cases.
 * {{Lucene.Net.TestFramework.Analysis.MockCharFilter}} > Use C5's {{TreeDictionary<TKey, TValue}}
 * {{Lucene.Net.Grouping.AbstractGroupFacetCollector}} > Use C5's {{TreeSet<T>}}
 * {{Lucene.Net.Highlighter.PostingsHighlight.PostingsHighlighter}} > Use C5's {{TreeSet<T>}}

Note that C5's {{TreeSet<T>}} does not currently support {{ISet<T>}}, and we will need to push our implementation of it back to them to support this change.

Structural Equality

J2N has {{J2N.Collections.Generic.ListEqualityComparer<T>}}, {{J2N.Collections.Generic.SetEqualityComparer<T>}}, {{J2N.Collections.Generic.DictionaryEqualityComparer<TKey, TValue>}}, and {{J2N.Collections.StructuralEqualityComparer<T>}} that implement the structural equality behavior for each type of collection. These types should be passed in through each collection's constructor, as each has a {{Default}} and an {{Aggressive}} implementation.

{{Default}} is more efficient and should be used in all closed scenarios within Lucene.NET (where the collection is not passed in from outside of the class). {{Aggressive}} is designed to interoperate with built-in .NET collections.

See [these minimal examples of implementations|https://github.com/NightOwl888/J2N/tree/a50d41668cf459f3fe827dff5f1746422155c4f6/src/J2N.TestFramework/Util].

Collection Formatting

J2N has a {{StringFormatter}} class that can be used to format collections, as follows.
{code:c#}
// IFormattable
public string ToString(IFormatProvider provider)
{
    return string.Format(provider, "{0}", this);
}

public override string ToString()
{
    return ToString(StringFormatter.CurrentCulture);
}
{code}
Nullable Keys

Use the attached {{NullableKey<T>}} and {{NullableKeyDictionary<TKey, TValue>}} implementations. {{NullableKeyDictionary<TKey, TValue>}} is intended to be a base class for all nullable key dictionary implementations. The backing dictionary (the one we are trying to mimic) is then passed in through its constructor.

  was:
The collection types in {{Lucene.Net.Support}} were originally sourced to support Lucene.Net itself. While they were made public, they were not considered to be features that would be used by anyone except for advanced users.

However, it has become clear by user reports that some parts of Lucene's design require specialized collections that don't exist in the .NET Framework in order to properly function (see [LUCENENET-612|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-612] and [LUCENENET-615|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-615]. .NET users are generally not familiar with these specialized collection types and assume that when {{IDictionary<TKey, TValue>}} is required by an API that using {{Dictionary<TKey, TValue>}} is their best choice. We need to improve documentation and increase visibility of the specialized collection types in order to help them along.
 # The collection types should be moved to a new {{Lucene.Net.Collections.Specialized}} namespace
 # The collection types should stay in the {{Support}} folder, but be moved into a {{Collections/Specialized}} subfolder. The {{Collections}} class needs to be renamed to prevent a naming conflict - I suggest following the same approach that was done in [ICU4N's Support folder|https://github.com/NightOwl888/ICU4N/tree/master/src/ICU4N/Support], by making many of them into extension methods.
 # The collection types should be named consistently according to .NET conventions, for example {{HashMap}} should be renamed {{HashDictionary}}
 # We should ensure we have a full set of tests for each collection, and that each collection is fully implemented
 # All collections must implement the expected interfaces from {{System.Collections}}, for example a generic dictionary should implement {{System.Collections.Generic.IDictionary<TKey, TValue>}}
 # The documentation for each collection should not just indicate where it was grabbed from, but explain fully how it behaves and how its behavior differs from other collection types
 # APIs where it is known that specialized collections were intended to be used should also be documented to indicate the choices the end user has so they don't automatically assume they are constrained by the .NET framework's collection types
 # There should be a "namespace document" in {{Lucene.Net.Collections.Generic}} that lists all of the collection types and a summary of their behavior, so they can be comparison shopped easily

Since these are breaking API changes, they should be done before the official release of Lucene.Net 4.8.0.

Generally speaking, collections were fully implemented and tested to prevent odd bugs from creeping into Lucene.Net but it wouldn't hurt to review to ensure that is the case.


> Make Collections from Lucene.Net.Support into a 1st Class Feature
> -----------------------------------------------------------------
>
>                 Key: LUCENENET-616
>                 URL: https://issues.apache.org/jira/browse/LUCENENET-616
>             Project: Lucene.Net
>          Issue Type: Improvement
>          Components: Lucene.Net Core
>    Affects Versions: Lucene.Net 4.8.0
>            Reporter: Shad Storhaug
>            Assignee: Shad Storhaug
>            Priority: Major
>
> The collection types in {{Lucene.Net.Support}} were originally sourced to support Lucene.Net itself. While they were made public, they were not considered to be features that would be used by anyone except for advanced users.
> However, it has become clear by user reports that some parts of Lucene's design require specialized collections that don't exist in the .NET Framework in order to properly function (see [LUCENENET-612|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-612] and [LUCENENET-615|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-615]. .NET users are generally not familiar with these specialized collection types and assume that when {{IDictionary<TKey, TValue>}} is required by an API that using {{Dictionary<TKey, TValue>}} is their best choice. We need to improve documentation and increase visibility of the specialized collection types in order to help them along.
> Some of the existing collections are composed of other nested collection objects and should be replaced with a lower-level implementation, if possible. Since these are breaking API changes, they should be done before the official release of Lucene.Net 4.8.0.
> Additionally, many parts of Lucene.NET expect the collections to:
>  * Be structurally equal
>  * Format their contents in the ToString() method
> The safest and most thorough way to achieve this is to replace the usage of all built-in .NET collections with collections that we own that implement this functionality. .NET provides interfaces to help achieve this:
>  * {{IStructuralEquatable}}
>  * {{IFormattable}}
> But neither interface is implemented in any of the built-in collections in .NET.
> h2. *Requirements for Lucene.NET Collections*
> We must have the following collections for internal use within Lucene.NET, and each should also be made available to end users.
>  * {{J2N.Collections.Generic.List<T>}}
>  ** Subclass of {{System.Collections.Generic.List<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.HashSet<T>}}
>  ** Subclass of {{System.Collections.Generic.List<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.SortedSet<T>}}
>  ** Subclass of {{System.Collections.Generic.SortedSet<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.SortedDictionary<TKey, TValue>}}
>  ** Same interface as {{System.Collections.Generic.SortedDictionary<TKey, TValue>}}
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.LinkedDictionary<TKey, TValue>}}
>  ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Preserves insertion order across adds/deletes
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.PriorityQueue<T>}}
>  ** Migrate {{Lucene.Net.Support.PriorityQueue<T>}} implementation, but clean up the API for .NET (AddAll() > AddRange(), etc.)
>  * {{J2N.Collections.Generic.IdentityDictionary<TKey, TValue>}} (See {{Lucene.Net.Support.IdentityHashMap<TKey, TValue>}})
>  ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.IdentityHashSet<T>}} (See {{Lucene.Net.Support.IdentityHashSet<T>}})
>  ** Subclass of {{System.Collections.Generic.HashSet<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{ConcurrentHashSet<T>}} - (See LUCENENET-634)
> We should have the following collections for internal use within Lucene.NET.
>  * {{J2N.Collections.Generic.LinkedHashSet<T>}}
>  ** Same interface as {{System.Collections.Generic.List<T>}}
>  ** Preserves insertion order across adds/deletes
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Runtime.CompilerServices.ConditionalWeakTable<TKey, TValue>}} (See LUCENENET-636)
> h3. Special Cases
> There are a few uses where the above collections won't suffice because they use low-level methods that don't exist in .NET collections. However, these are closed systems and we don't have to worry about supporting end users in these cases.
>  * {{Lucene.Net.TestFramework.Analysis.MockCharFilter}} > Use C5's {{TreeDictionary<TKey, TValue}}
>  * {{Lucene.Net.Grouping.AbstractGroupFacetCollector}} > Use C5's {{TreeSet<T>}}
>  * {{Lucene.Net.Highlighter.PostingsHighlight.PostingsHighlighter}} > Use C5's {{TreeSet<T>}}
> Note that C5's {{TreeSet<T>}} does not currently support {{ISet<T>}}, and we will need to push our implementation of it back to them to support this change.
> Structural Equality
> J2N has {{J2N.Collections.Generic.ListEqualityComparer<T>}}, {{J2N.Collections.Generic.SetEqualityComparer<T>}}, {{J2N.Collections.Generic.DictionaryEqualityComparer<TKey, TValue>}}, and {{J2N.Collections.StructuralEqualityComparer<T>}} that implement the structural equality behavior for each type of collection. These types should be passed in through each collection's constructor, as each has a {{Default}} and an {{Aggressive}} implementation.
> {{Default}} is more efficient and should be used in all closed scenarios within Lucene.NET (where the collection is not passed in from outside of the class). {{Aggressive}} is designed to interoperate with built-in .NET collections.
> See [these minimal examples of implementations|https://github.com/NightOwl888/J2N/tree/a50d41668cf459f3fe827dff5f1746422155c4f6/src/J2N.TestFramework/Util].
> Collection Formatting
> J2N has a {{StringFormatter}} class that can be used to format collections, as follows.
> {code:c#}
> // IFormattable
> public string ToString(IFormatProvider provider)
> {
>     return string.Format(provider, "{0}", this);
> }
> public override string ToString()
> {
>     return ToString(StringFormatter.CurrentCulture);
> }
> {code}
> Nullable Keys
> Use the attached {{NullableKey<T>}} and {{NullableKeyDictionary<TKey, TValue>}} implementations. {{NullableKeyDictionary<TKey, TValue>}} is intended to be a base class for all nullable key dictionary implementations. The backing dictionary (the one we are trying to mimic) is then passed in through its constructor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)