You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucenenet.apache.org by ni...@apache.org on 2021/03/28 17:52:07 UTC
[lucenenet] branch master updated (eff154a -> adc0e6a)
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git.
from eff154a Tweaked messaging so that brands are properly communicated. Also fixed padding a bit on one case.
new 4be60a7 docs: Lucene.Net/Index/package.md: Fixed broken formatting and links (see #284, #300)
new 421b961 docs: Lucene.Net: Fixed remaining broken formatting and links (see #284, #300)
new bffd9b0 docs: Lucene.Net.Analysis.Common: Fixed broken formatting and links (see #284, #300)
new c8a0f00 docs: Lucene.Net.ICU: Restructured documentation and fixed broken formatting and links (see #284, #300)
new 3832015 docs.ps1: Update the version in toc/toc.yml on build
new adc0e6a docs: Changed breadcrumb link to "_api" so we can update it without also updating "_rel", which all of the styles and js links use.
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../Analysis/CharFilter/package.md | 8 +-
.../Analysis/Cjk/package.md | 14 +-
.../Analysis/Cn/package.md | 15 +-
.../Analysis/Compound/package.md | 119 +++++----
.../Analysis/Standard/Std31/package.md | 4 +-
.../Analysis/Standard/Std34/package.md | 4 +-
.../Analysis/Standard/Std36/package.md | 4 +-
.../Analysis/Standard/Std40/package.md | 4 +-
.../Analysis/Standard/package.md | 80 +++----
.../Collation/TokenAttributes/package.md | 4 +-
src/Lucene.Net.Analysis.ICU/overview.md | 265 ++++++++++++---------
src/Lucene.Net/Codecs/Lucene40/package.md | 121 ++++------
src/Lucene.Net/Codecs/Lucene41/package.md | 117 ++++-----
src/Lucene.Net/Codecs/Lucene42/package.md | 124 ++++------
src/Lucene.Net/Codecs/Lucene45/package.md | 127 ++++------
src/Lucene.Net/Codecs/Lucene46/package.md | 130 ++++------
src/Lucene.Net/Index/package.md | 154 +++++++++---
src/Lucene.Net/Search/Payloads/package.md | 7 +-
src/Lucene.Net/Search/Similarities/package.md | 33 ++-
src/Lucene.Net/Search/Spans/package.md | 39 ++-
src/Lucene.Net/Search/package.md | 155 +++++++-----
src/Lucene.Net/Util/Fst/package.md | 110 +++++----
src/Lucene.Net/Util/Packed/package.md | 25 +-
src/dotnet/Lucene.Net.ICU/Lucene.Net.ICU.csproj | 18 +-
src/dotnet/Lucene.Net.ICU/overview.md | 66 +++++
.../LuceneApiDocs/partials/breadcrumb.tmpl.partial | 2 +-
websites/apidocs/docfx.global.subsite.json | 5 +-
websites/apidocs/docfx.icu.json | 12 +-
websites/apidocs/docfx.json | 1 +
websites/apidocs/docs.ps1 | 12 +-
websites/apidocs/index.md | 4 +-
31 files changed, 991 insertions(+), 792 deletions(-)
create mode 100644 src/dotnet/Lucene.Net.ICU/overview.md
[lucenenet] 06/06: docs: Changed breadcrumb link to "_api" so we
can update it without also updating "_rel",
which all of the styles and js links use.
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit adc0e6a433a17156835bc5bb88fd395bc06bdd15
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Mar 28 20:59:23 2021 +0700
docs: Changed breadcrumb link to "_api" so we can update it without also updating "_rel", which all of the styles and js links use.
---
.../apidocs/Templates/LuceneApiDocs/partials/breadcrumb.tmpl.partial | 2 +-
websites/apidocs/docfx.global.subsite.json | 5 +++--
websites/apidocs/docs.ps1 | 5 +++++
3 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/websites/apidocs/Templates/LuceneApiDocs/partials/breadcrumb.tmpl.partial b/websites/apidocs/Templates/LuceneApiDocs/partials/breadcrumb.tmpl.partial
index 3a669f0..b12eba1 100644
--- a/websites/apidocs/Templates/LuceneApiDocs/partials/breadcrumb.tmpl.partial
+++ b/websites/apidocs/Templates/LuceneApiDocs/partials/breadcrumb.tmpl.partial
@@ -4,7 +4,7 @@
<div class="container hide-when-search">
<ul class="level0 breadcrumb">
<li>
- <a href="{{_rel}}">API</a>
+ <a href="{{_api}}">API</a>
<span id="breadcrumb">
<ul class="breadcrumb">
<li>{{_tocTitle}}</li>
diff --git a/websites/apidocs/docfx.global.subsite.json b/websites/apidocs/docfx.global.subsite.json
index f9025e5..fafaf8e 100644
--- a/websites/apidocs/docfx.global.subsite.json
+++ b/websites/apidocs/docfx.global.subsite.json
@@ -1,3 +1,4 @@
-{
- "_rel": "https://lucenenet.apache.org/docs/4.8.0-beta00009/"
+{
+ "_rel": "https://lucenenet.apache.org/docs/4.8.0-beta00009/",
+ "_api": "https://lucenenet.apache.org/docs/4.8.0-beta00009/"
}
diff --git a/websites/apidocs/docs.ps1 b/websites/apidocs/docs.ps1
index 5de4cae..390da4d 100644
--- a/websites/apidocs/docs.ps1
+++ b/websites/apidocs/docs.ps1
@@ -54,6 +54,7 @@ $ToolsFolder = Join-Path -Path $ApiDocsFolder -ChildPath "tools";
$CliIndexPath = Join-Path -Path $RepoRoot -ChildPath "src\dotnet\tools\lucene-cli\docs\index.md";
$TocPath1 = Join-Path -Path $ApiDocsFolder -ChildPath "toc.yml"
$TocPath2 = Join-Path -Path $ApiDocsFolder -ChildPath "toc\toc.yml"
+$BreadcrumbPath = Join-Path -Path $ApiDocsFolder -ChildPath "docfx.global.subsite.json"
#ensure the /build/tools folder
New-Item $ToolsFolder -type directory -force
@@ -192,6 +193,10 @@ if ($? -and $DisableBuild -eq $false) {
(Get-Content -Path $TocPath1 -Raw) -Replace '(?<=lucenenet\.apache\.org\/docs\/)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $TocPath1
(Get-Content -Path $TocPath2 -Raw) -Replace '(?<=lucenenet\.apache\.org\/docs\/)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $TocPath2
+ # Update the API link to the latest LuceneNetVersion
+ # Note we don't update _rel because that is used for styles and js
+ (Get-Content -Path $BreadcrumbPath -Raw) -Replace '(?<="_api":\s*?"https?\:\/\/lucenenet\.apache\.org\/docs\/)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $BreadcrumbPath
+
foreach ($proj in $DocFxJsonMeta) {
$projFile = Join-Path -Path $ApiDocsFolder $proj
[lucenenet] 04/06: docs: Lucene.Net.ICU: Restructured documentation
and fixed broken formatting and links (see #284, #300)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit c8a0f00f0c92c0c55d3bb1ed0b6f26133774eb5e
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Mar 28 20:36:57 2021 +0700
docs: Lucene.Net.ICU: Restructured documentation and fixed broken formatting and links (see #284, #300)
---
.../Collation/TokenAttributes/package.md | 4 +-
src/Lucene.Net.Analysis.ICU/overview.md | 265 ++++++++++++---------
src/dotnet/Lucene.Net.ICU/Lucene.Net.ICU.csproj | 18 +-
src/dotnet/Lucene.Net.ICU/overview.md | 66 +++++
websites/apidocs/docfx.icu.json | 12 +-
websites/apidocs/docfx.json | 1 +
websites/apidocs/index.md | 4 +-
7 files changed, 239 insertions(+), 131 deletions(-)
diff --git a/src/Lucene.Net.Analysis.ICU/Collation/TokenAttributes/package.md b/src/Lucene.Net.Analysis.ICU/Collation/TokenAttributes/package.md
index 4c6ef88..e429c8b 100644
--- a/src/Lucene.Net.Analysis.ICU/Collation/TokenAttributes/package.md
+++ b/src/Lucene.Net.Analysis.ICU/Collation/TokenAttributes/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Collation.TokenAttributes
summary: *content
---
@@ -20,4 +20,4 @@ summary: *content
limitations under the License.
-->
-Custom <xref:Lucene.Net.Util.AttributeImpl> for indexing collation keys as index terms.
\ No newline at end of file
+Custom <xref:Lucene.Net.Util.Attribute> for indexing collation keys as index terms.
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.ICU/overview.md b/src/Lucene.Net.Analysis.ICU/overview.md
index 95609f8..59d49d6 100644
--- a/src/Lucene.Net.Analysis.ICU/overview.md
+++ b/src/Lucene.Net.Analysis.ICU/overview.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Icu
summary: *content
---
@@ -22,16 +22,26 @@ summary: *content
<!-- :Post-Release-Update-Version.LUCENE_XY: - several mentions in this file -->
This module exposes functionality from
-[ICU](http://site.icu-project.org/) to Apache Lucene. ICU4J is a Java
-library that enhances Java's internationalization support by improving
+[ICU](http://site.icu-project.org/) to Apache Lucene. ICU4N is a .NET
+library that enhances .NET's internationalization support by improving
performance, keeping current with the Unicode Standard, and providing richer
-APIs.
+APIs.
+
+> [!NOTE]
+> The <xref:Lucene.Net.Analysis.Icu> namespace was ported from Lucene 7.1.0 to get a more up-to-date version of Unicode than what shipped with Lucene 4.8.0.
+
+> [!NOTE]
+> Since the .NET platform doesn't provide a BreakIterator class (or similar), the functionality that utilizes it was consolidated from Java Lucene's analyzers-icu package, <xref:Lucene.Net.Analysis.Common> and <xref:Lucene.Net.Highlighter> into this unified package.
+> [!WARNING]
+> While ICU4N's BreakIterator has customizable rules, its default behavior is not the same as the one in the JDK. When using any features of this package outside of the <xref:Lucene.Net.Analysis.Icu> namespace, they will behave differently than they do in Java Lucene and the rules may need some tweaking to fit your needs. See the [Break Rules](http://userguide.icu-project.org/boundaryanalysis/break-rules) ICU documentation for details on how to customize `ICU4N.Text.RuleBaseBreakIterator`.
+
+
For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis> package documentation.
This module exposes the following functionality:
-* [Text Segmentation](#segmentation): Tokenizes text based on
+* [Text Segmentation](#text-segmentation): Tokenizes text based on
properties and rules defined in Unicode.
* [Collation](#collation): Compare strings according to the
@@ -40,18 +50,18 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
* [Normalization](#normalization): Converts text to a unique,
equivalent form.
-* [Case Folding](#casefolding): Removes case distinctions with
+* [Case Folding](#case-folding): Removes case distinctions with
Unicode's Default Caseless Matching algorithm.
-* [Search Term Folding](#searchfolding): Removes distinctions
+* [Search Term Folding](#search-term-folding): Removes distinctions
(such as accent marks) between similar characters for a loose or fuzzy search.
-* [Text Transformation](#transform): Transforms Unicode text in
+* [Text Transformation](#text-transform): Transforms Unicode text in
a context-sensitive fashion: e.g. mapping Traditional to Simplified Chinese
* * *
-# [Text Segmentation]()
+# Text Segmentation
Text Segmentation (Tokenization) divides document and query text into index terms (typically words). Unicode provides special properties and rules so that this can be done in a manner that works well with most languages.
@@ -66,26 +76,26 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
### Tokenizing multilanguage text
- /**
- * This tokenizer will work well in general for most languages.
- */
- Tokenizer tokenizer = new ICUTokenizer(reader);
+```cs
+// This tokenizer will work well in general for most languages.
+Tokenizer tokenizer = new ICUTokenizer(reader);
+```
* * *
-# [Collation]()
+# Collation
- `ICUCollationKeyAnalyzer` converts each token into its binary `CollationKey` using the provided `Collator`, allowing it to be stored as an index term.
+ <xref:Lucene.Net.Collation.ICUCollationKeyAnalyzer> converts each token into its binary `CollationKey` using the provided `Collator`, allowing it to be stored as an index term.
- `ICUCollationKeyAnalyzer` depends on ICU4J to produce the `CollationKey`s.
+ <xref:Lucene.Net.Collation.ICUCollationKeyAnalyzer> depends on ICU4N to produce the `CollationKey`s.
## Use Cases
* Efficient sorting of terms in languages that use non-Unicode character
- orderings. (Lucene Sort using a Locale can be very slow.)
+ orderings. (Lucene Sort using a CultureInfo can be very slow.)
* Efficient range queries over fields that contain terms in languages that
- use non-Unicode character orderings. (Range queries using a Locale can be
+ use non-Unicode character orderings. (Range queries using a CultureInfo can be
very slow.)
* Effective Locale-specific normalization (case differences, diacritics, etc.).
@@ -97,80 +107,99 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
### Farsi Range Queries
- Collator collator = Collator.getInstance(new ULocale("ar"));
- ICUCollationKeyAnalyzer analyzer = new ICUCollationKeyAnalyzer(Version.LUCENE_48, collator);
- RAMDirectory ramDir = new RAMDirectory();
- IndexWriter writer = new IndexWriter(ramDir, new IndexWriterConfig(Version.LUCENE_48, analyzer));
- Document doc = new Document();
- doc.add(new Field("content", "\u0633\u0627\u0628",
- Field.Store.YES, Field.Index.ANALYZED));
- writer.addDocument(doc);
- writer.close();
- IndexSearcher is = new IndexSearcher(ramDir, true);
-
- QueryParser aqp = new QueryParser(Version.LUCENE_48, "content", analyzer);
- aqp.setAnalyzeRangeTerms(true);
-
- // Unicode order would include U+0633 in [ U+062F - U+0698 ], but Farsi
- // orders the U+0698 character before the U+0633 character, so the single
- // indexed Term above should NOT be returned by a ConstantScoreRangeQuery
- // with a Farsi Collator (or an Arabic one for the case when Farsi is not
- // supported).
- ScoreDoc[] result
- = is.search(aqp.parse("[ \u062F TO \u0698 ]"), null, 1000).scoreDocs;
- assertEquals("The index Term should not be included.", 0, result.length);
+```cs
+const LuceneVersion matchVersion = LuceneVersion.LUCENE_48;
+Collator collator = Collator.GetInstance(new UCultureInfo("ar"));
+ICUCollationKeyAnalyzer analyzer = new ICUCollationKeyAnalyzer(matchVersion, collator);
+RAMDirectory ramDir = new RAMDirectory();
+using IndexWriter writer = new IndexWriter(ramDir, new IndexWriterConfig(matchVersion, analyzer));
+writer.AddDocument(new Document {
+ new TextField("content", "\u0633\u0627\u0628", Field.Store.YES)
+});
+using IndexReader reader = writer.GetReader(applyAllDeletes: true);
+writer.Dispose();
+IndexSearcher searcher = new IndexSearcher(reader);
+
+QueryParser queryParser = new QueryParser(matchVersion, "content", analyzer)
+{
+ AnalyzeRangeTerms = true
+};
+
+// Unicode order would include U+0633 in [ U+062F - U+0698 ], but Farsi
+// orders the U+0698 character before the U+0633 character, so the single
+// indexed Term above should NOT be returned by a ConstantScoreRangeQuery
+// with a Farsi Collator (or an Arabic one for the case when Farsi is not
+// supported).
+ScoreDoc[] result = searcher.Search(queryParser.Parse("[ \u062F TO \u0698 ]"), null, 1000).ScoreDocs;
+Assert.AreEqual(0, result.Length, "The index Term should not be included.");
+```
### Danish Sorting
- Analyzer analyzer
- = new ICUCollationKeyAnalyzer(Version.LUCENE_48, Collator.getInstance(new ULocale("da", "dk")));
- RAMDirectory indexStore = new RAMDirectory();
- IndexWriter writer = new IndexWriter(indexStore, new IndexWriterConfig(Version.LUCENE_48, analyzer));
- String[] tracer = new String[] { "A", "B", "C", "D", "E" };
- String[] data = new String[] { "HAT", "HUT", "H\u00C5T", "H\u00D8T", "HOT" };
- String[] sortedTracerOrder = new String[] { "A", "E", "B", "D", "C" };
- for (int i = 0 ; i < data.length="" ;="" ++i)="" {="" document="" doc="new" document();="" doc.add(new="" field("tracer",="" tracer[i],="" field.store.yes,="" field.index.no));="" doc.add(new="" field("contents",="" data[i],="" field.store.no,="" field.index.analyzed));="" writer.adddocument(doc);="" }="" writer.close();="" indexsearcher="" searcher="new" indexsearcher(indexstore,="" true);="" sort="" sort="new" sort();="" sort.setsort(new="" sortfield("contents",="" sortfield.stri [...]
+```cs
+const LuceneVersion matchVersion = LuceneVersion.LUCENE_48;
+Analyzer analyzer = new ICUCollationKeyAnalyzer(matchVersion, Collator.GetInstance(new UCultureInfo("da-dk")));
+string indexPath = Path.Combine(Path.GetTempPath(), Path.GetFileNameWithoutExtension(Path.GetTempFileName()));
+Directory dir = FSDirectory.Open(indexPath);
+using IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig(matchVersion, analyzer));
+string[] tracer = new string[] { "A", "B", "C", "D", "E" };
+string[] data = new string[] { "HAT", "HUT", "H\u00C5T", "H\u00D8T", "HOT" };
+string[] sortedTracerOrder = new string[] { "A", "E", "B", "D", "C" };
+for (int i = 0; i < data.Length; ++i)
+{
+ writer.AddDocument(new Document
+ {
+ new StringField("tracer", tracer[i], Field.Store.YES),
+ new TextField("contents", data[i], Field.Store.NO)
+ });
+}
+using IndexReader reader = writer.GetReader(applyAllDeletes: true);
+writer.Dispose();
+IndexSearcher searcher = new IndexSearcher(reader);
+Sort sort = new Sort();
+sort.SetSort(new SortField("contents", SortFieldType.STRING));
+Query query = new MatchAllDocsQuery();
+ScoreDoc[] result = searcher.Search(query, null, 1000, sort).ScoreDocs;
+for (int i = 0; i < result.Length; ++i)
+{
+ Document doc = searcher.Doc(result[i].Doc);
+ Assert.AreEqual(sortedTracerOrder[i], doc.GetValues("tracer")[0]);
+}
+```
### Turkish Case Normalization
- Collator collator = Collator.getInstance(new ULocale("tr", "TR"));
- collator.setStrength(Collator.PRIMARY);
- Analyzer analyzer = new ICUCollationKeyAnalyzer(Version.LUCENE_48, collator);
- RAMDirectory ramDir = new RAMDirectory();
- IndexWriter writer = new IndexWriter(ramDir, new IndexWriterConfig(Version.LUCENE_48, analyzer));
- Document doc = new Document();
- doc.add(new Field("contents", "DIGY", Field.Store.NO, Field.Index.ANALYZED));
- writer.addDocument(doc);
- writer.close();
- IndexSearcher is = new IndexSearcher(ramDir, true);
- QueryParser parser = new QueryParser(Version.LUCENE_48, "contents", analyzer);
- Query query = parser.parse("d\u0131gy"); // U+0131: dotless i
- ScoreDoc[] result = is.search(query, null, 1000).scoreDocs;
- assertEquals("The index Term should be included.", 1, result.length);
+```cs
+const LuceneVersion matchVersion = LuceneVersion.LUCENE_48;
+Collator collator = Collator.GetInstance(new UCultureInfo("tr-TR"));
+collator.Strength = CollationStrength.Primary;
+Analyzer analyzer = new ICUCollationKeyAnalyzer(matchVersion, collator);
+string indexPath = Path.Combine(Path.GetTempPath(), Path.GetFileNameWithoutExtension(Path.GetTempFileName()));
+Directory dir = FSDirectory.Open(indexPath);
+using IndexWriter writer = new IndexWriter(dir, new IndexWriterConfig(matchVersion, analyzer));
+writer.AddDocument(new Document {
+ new TextField("contents", "DIGY", Field.Store.NO)
+});
+using IndexReader reader = writer.GetReader(applyAllDeletes: true);
+writer.Dispose();
+IndexSearcher searcher = new IndexSearcher(reader);
+QueryParser parser = new QueryParser(matchVersion, "contents", analyzer);
+Query query = parser.Parse("d\u0131gy"); // U+0131: dotless i
+ScoreDoc[] result = searcher.Search(query, null, 1000).ScoreDocs;
+Assert.AreEqual(1, result.Length, "The index Term should be included.");
+```
## Caveats and Comparisons
- __WARNING:__ Make sure you use exactly the same `Collator` at index and query time -- `CollationKey`s are only comparable when produced by the same `Collator`. Since {@link java.text.RuleBasedCollator}s are not independently versioned, it is unsafe to search against stored `CollationKey`s unless the following are exactly the same (best practice is to store this information with the index and check that they remain the same at query time):
-
-1. JVM vendor
-
-2. JVM version, including patch version
-
-3. The language (and country and variant, if specified) of the Locale
- used when constructing the collator via
- {@link java.text.Collator#getInstance(java.util.Locale)}.
-
-4. The collation strength used - see {@link java.text.Collator#setStrength(int)}
-
- `ICUCollationKeyAnalyzer` uses ICU4J's `Collator`, which makes its version available, thus allowing collation to be versioned independently from the JVM. `ICUCollationKeyAnalyzer` is also significantly faster and generates significantly shorter keys than `CollationKeyAnalyzer`. See [http://site.icu-project.org/charts/collation-icu4j-sun](http://site.icu-project.org/charts/collation-icu4j-sun) for key generation timing and key length comparisons between ICU4J and `java.text.Collator` ove [...]
+ `ICUCollationKeyAnalyzer` uses ICU4N's `Collator`, which makes its version available, thus allowing collation to be versioned independently from the .NET target framework. `ICUCollationKeyAnalyzer` is also fast.
- `CollationKey`s generated by `java.text.Collator`s are not compatible with those those generated by ICU Collators. Specifically, if you use `CollationKeyAnalyzer` to generate index terms, do not use `ICUCollationKeyAnalyzer` on the query side, or vice versa.
+ `SortKey`s generated by `CompareInfo`s are not compatible with those those generated by ICU Collators. Specifically, if you use `CollationKeyAnalyzer` to generate index terms, do not use `ICUCollationKeyAnalyzer` on the query side, or vice versa.
* * *
-# [Normalization]()
+# Normalization
- `ICUNormalizer2Filter` normalizes term text to a [Unicode Normalization Form](http://unicode.org/reports/tr15/), so that [equivalent](http://en.wikipedia.org/wiki/Unicode_equivalence) forms are standardized to a unique form.
+ <xref:Lucene.Net.Analysis.Icu.ICUNormalizer2Filter> normalizes term text to a [Unicode Normalization Form](http://unicode.org/reports/tr15/), so that [equivalent](http://en.wikipedia.org/wiki/Unicode_equivalence) forms are standardized to a unique form.
## Use Cases
@@ -183,18 +212,16 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
### Normalizing text to NFC
- /**
- * Normalizer2 objects are unmodifiable and immutable.
- */
- Normalizer2 normalizer = Normalizer2.getInstance(null, "nfc", Normalizer2.Mode.COMPOSE);
- /**
- * This filter will normalize to NFC.
- */
- TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer, normalizer);
+```cs
+// Normalizer2 objects are unmodifiable and immutable.
+Normalizer2 normalizer = Normalizer2.GetInstance(null, "nfc", Normalizer2Mode.Compose);
+// This filter will normalize to NFC.
+TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer, normalizer);
+```
* * *
-# [Case Folding]()
+# Case Folding
Default caseless matching, or case-folding is more than just conversion to lowercase. For example, it handles cases such as the Greek sigma, so that "Μάϊος" and "ΜΆΪΟΣ" will match correctly.
@@ -211,14 +238,14 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
### Lowercasing text
- /**
- * This filter will case-fold and normalize to NFKC.
- */
- TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer);
+```cs
+// This filter will case-fold and normalize to NFKC.
+TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer);
+```
* * *
-# [Search Term Folding]()
+# Search Term Folding
Search term folding removes distinctions (such as accent marks) between similar characters. It is useful for a fuzzy or loose search.
@@ -233,15 +260,15 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
### Removing accents
- /**
- * This filter will case-fold, remove accents and other distinctions, and
- * normalize to NFKC.
- */
- TokenStream tokenstream = new ICUFoldingFilter(tokenizer);
+```cs
+// This filter will case-fold, remove accents and other distinctions, and
+// normalize to NFKC.
+TokenStream tokenstream = new ICUFoldingFilter(tokenizer);
+```
* * *
-# [Text Transformation]()
+# Text Transformation
ICU provides text-transformation functionality via its Transliteration API. This allows you to transform text in a variety of ways, taking context into account.
@@ -257,36 +284,36 @@ For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis>
### Convert Traditional to Simplified
- /**
- * This filter will map Traditional Chinese to Simplified Chinese
- */
- TokenStream tokenstream = new ICUTransformFilter(tokenizer, Transliterator.getInstance("Traditional-Simplified"));
+```cs
+// This filter will map Traditional Chinese to Simplified Chinese
+TokenStream tokenstream = new ICUTransformFilter(tokenizer, Transliterator.GetInstance("Traditional-Simplified"));
+```
### Transliterate Serbian Cyrillic to Serbian Latin
- /**
- * This filter will map Serbian Cyrillic to Serbian Latin according to BGN rules
- */
- TokenStream tokenstream = new ICUTransformFilter(tokenizer, Transliterator.getInstance("Serbian-Latin/BGN"));
+```cs
+// This filter will map Serbian Cyrillic to Serbian Latin according to BGN rules
+TokenStream tokenstream = new ICUTransformFilter(tokenizer, Transliterator.GetInstance("Serbian-Latin/BGN"));
+```
* * *
-# [Backwards Compatibility]()
+# Backwards Compatibility
- This module exists to provide up-to-date Unicode functionality that supports the most recent version of Unicode (currently 6.3). However, some users who wish for stronger backwards compatibility can restrict <xref:Lucene.Net.Analysis.Icu.ICUNormalizer2Filter> to operate on only a specific Unicode Version by using a {@link com.ibm.icu.text.FilteredNormalizer2}.
+ This module exists to provide up-to-date Unicode functionality that supports the most recent version of Unicode (currently 8.0). However, some users who wish for stronger backwards compatibility can restrict <xref:Lucene.Net.Analysis.Icu.ICUNormalizer2Filter> to operate on only a specific Unicode Version by using a FilteredNormalizer2.
## Example Usages
### Restricting normalization to Unicode 5.0
- /**
- * This filter will do NFC normalization, but will ignore any characters that
- * did not exist as of Unicode 5.0. Because of the normalization stability policy
- * of Unicode, this is an easy way to force normalization to a specific version.
- */
- Normalizer2 normalizer = Normalizer2.getInstance(null, "nfc", Normalizer2.Mode.COMPOSE);
- UnicodeSet set = new UnicodeSet("[:age=5.0:]");
- // see FilteredNormalizer2 docs, the set should be frozen or performance will suffer
- set.freeze();
- FilteredNormalizer2 unicode50 = new FilteredNormalizer2(normalizer, set);
- TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer, unicode50);
\ No newline at end of file
+```cs
+// This filter will do NFC normalization, but will ignore any characters that
+// did not exist as of Unicode 5.0. Because of the normalization stability policy
+// of Unicode, this is an easy way to force normalization to a specific version.
+Normalizer2 normalizer = Normalizer2.GetInstance(null, "nfc", Normalizer2Mode.Compose);
+UnicodeSet set = new UnicodeSet("[:age=5.0:]");
+// see FilteredNormalizer2 docs, the set should be frozen or performance will suffer
+set.Freeze();
+FilteredNormalizer2 unicode50 = new FilteredNormalizer2(normalizer, set);
+TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer, unicode50);
+```
\ No newline at end of file
diff --git a/src/dotnet/Lucene.Net.ICU/Lucene.Net.ICU.csproj b/src/dotnet/Lucene.Net.ICU/Lucene.Net.ICU.csproj
index 050614a..0c8bc3b 100644
--- a/src/dotnet/Lucene.Net.ICU/Lucene.Net.ICU.csproj
+++ b/src/dotnet/Lucene.Net.ICU/Lucene.Net.ICU.csproj
@@ -37,18 +37,26 @@
<ItemGroup>
<Compile Include="..\..\Lucene.Net.Analysis.Common\Analysis\Th\**\*.cs" LinkBase="Analysis\Th" />
- <EmbeddedResource Include="..\..\Lucene.Net.Analysis.Common\Analysis\Th\stopwords.txt" Link="Analysis\Th\stopwords.txt" />
- <EmbeddedResource Include="Support\*.brk" />
<Compile Include="..\..\Lucene.Net.Analysis.Common\Analysis\Util\CharArrayIterator.cs" Link="Analysis\Util\CharArrayIterator.cs" />
<Compile Include="..\..\Lucene.Net.Analysis.Common\Analysis\Util\SegmentingTokenizerBase.cs" Link="Analysis\Util\SegmentingTokenizerBase.cs" />
<Compile Include="..\..\Lucene.Net.Analysis.ICU\Analysis\**\*.cs" LinkBase="Analysis" />
- <EmbeddedResource Include="..\..\Lucene.Net.Analysis.ICU\Analysis\**\*.nrm" LinkBase="Analysis" />
- <EmbeddedResource Include="..\..\Lucene.Net.Analysis.ICU\Analysis\**\*.brk" LinkBase="Analysis" />
<Compile Include="..\..\Lucene.Net.Analysis.ICU\Collation\**\*.cs" LinkBase="Collation" />
<Compile Include="..\..\Lucene.Net.Highlighter\PostingsHighlight\**\*.cs" LinkBase="Search\PostingsHighlight" />
+ <Compile Include="..\..\Lucene.Net.Highlighter\VectorHighlight\BreakIteratorBoundaryScanner.cs" Link="Search\VectorHighlight\BreakIteratorBoundaryScanner.cs" />
+ </ItemGroup>
+
+ <ItemGroup Label="Embedded Resources">
+ <EmbeddedResource Include="..\..\Lucene.Net.Analysis.Common\Analysis\Th\stopwords.txt" Link="Analysis\Th\stopwords.txt" />
+ <EmbeddedResource Include="Support\*.brk" />
+ <EmbeddedResource Include="..\..\Lucene.Net.Analysis.ICU\Analysis\**\*.nrm" LinkBase="Analysis" />
+ <EmbeddedResource Include="..\..\Lucene.Net.Analysis.ICU\Analysis\**\*.brk" LinkBase="Analysis" />
<EmbeddedResource Include="..\..\Lucene.Net.Highlighter\PostingsHighlight\**\*.brk" LinkBase="Search\PostingsHighlight" />
<None Remove="Support\*.brk" />
- <Compile Include="..\..\Lucene.Net.Highlighter\VectorHighlight\BreakIteratorBoundaryScanner.cs" Link="Search\VectorHighlight\BreakIteratorBoundaryScanner.cs" />
+ </ItemGroup>
+
+ <ItemGroup Label="Documentation">
+ <None Include="..\..\Lucene.Net.Analysis.Common\Analysis\Th\**\*.md" LinkBase="Analysis\Th" />
+ <None Include="..\..\Lucene.Net.Highlighter\PostingsHighlight\**\*.md" LinkBase="Search\PostingsHighlight" />
</ItemGroup>
<ItemGroup>
diff --git a/src/dotnet/Lucene.Net.ICU/overview.md b/src/dotnet/Lucene.Net.ICU/overview.md
new file mode 100644
index 0000000..fd982db
--- /dev/null
+++ b/src/dotnet/Lucene.Net.ICU/overview.md
@@ -0,0 +1,66 @@
+---
+uid: Lucene.Net.ICU
+title: Lucene.Net.ICU
+summary: *content
+---
+
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+This module exposes functionality from
+[ICU](http://site.icu-project.org/) to Apache Lucene. ICU4N is a .NET
+library that enhances .NET's internationalization support by improving
+performance, keeping current with the Unicode Standard, and providing richer
+APIs.
+
+> [!NOTE]
+> Since the .NET platform doesn't provide a BreakIterator class (or similar), the functionality that utilizes it was consolidated from Java Lucene's analyzers-icu package, <xref:Lucene.Net.Analysis.Common> and <xref:Lucene.Net.Highlighter> into this unified package.
+> [!WARNING]
+> While ICU4N's BreakIterator has customizable rules, its default behavior is not the same as the one in the JDK. When using any features of this package outside of the <xref:Lucene.Net.Analysis.Icu> namespace, they will behave differently than they do in Java Lucene and the rules may need some tweaking to fit your needs. See the [Break Rules](http://userguide.icu-project.org/boundaryanalysis/break-rules) ICU documentation for details on how to customize `ICU4N.Text.RuleBaseBreakIterator`.
+
+
+
+This module exposes the following functionality:
+
+* [Text Analysis](xref:Lucene.Net.Analysis.Icu): For an introduction to Lucene's analysis API, see the <xref:Lucene.Net.Analysis> package documentation.
+
+ * [Text Segmentation](xref:Lucene.Net.Analysis.Icu#text-segmentation): Tokenizes text based on
+ properties and rules defined in Unicode.
+
+ * [Collation](xref:Lucene.Net.Analysis.Icu#collation): Compare strings according to the
+ conventions and standards of a particular language, region or country.
+
+ * [Normalization](xref:Lucene.Net.Analysis.Icu#normalization): Converts text to a unique,
+ equivalent form.
+
+ * [Case Folding](xref:Lucene.Net.Analysis.Icu#case-folding): Removes case distinctions with
+ Unicode's Default Caseless Matching algorithm.
+
+ * [Search Term Folding](xref:Lucene.Net.Analysis.Icu#search-term-folding): Removes distinctions
+ (such as accent marks) between similar characters for a loose or fuzzy search.
+
+ * [Text Transformation](xref:Lucene.Net.Analysis.Icu#text-transformation): Transforms Unicode text in
+ a context-sensitive fashion: e.g. mapping Traditional to Simplified Chinese
+
+ * [Thai Language Analysis](xref:Lucene.Net.Analysis.Th)
+
+* Unicode Highlighter Support
+
+ * [Postings Highlighter](xref:Lucene.Net.Search.PostingsHighlight): Highlighter implementation that uses offsets from postings lists.
+
+ * [Vector Highlighter](xref:Lucene.Net.Search.VectorHighlight.BreakIteratorBoundaryScanner): An implementation of IBoundaryScanner for use with the vector highlighter in the [Lucene.Net.Highlighter module](../highlighter/Lucene.Net.Search.Highlight.html).
+
diff --git a/websites/apidocs/docfx.icu.json b/websites/apidocs/docfx.icu.json
index 89c4691..2049a53 100644
--- a/websites/apidocs/docfx.icu.json
+++ b/websites/apidocs/docfx.icu.json
@@ -1,4 +1,4 @@
-{
+{
"metadata": [
{
"src": [
@@ -20,7 +20,13 @@
}
],
"build": {
- "content": [
+ "content": [
+ {
+ "files": [
+ "overview.md"
+ ],
+ "src": "../../src/dotnet/Lucene.Net.ICU"
+ },
{
"files": [
"**.yml",
@@ -36,7 +42,7 @@
"src": "toc"
}
],
- "overwrite": [
+ "overwrite": [
{
"files": [
"**/package.md",
diff --git a/websites/apidocs/docfx.json b/websites/apidocs/docfx.json
index d6dca5c..3a2bad0 100644
--- a/websites/apidocs/docfx.json
+++ b/websites/apidocs/docfx.json
@@ -523,6 +523,7 @@
"Lucene.Net.Analysis.Morfologik/overview.md",
"Lucene.Net.Analysis.OpenNLP/overview.md",
"Lucene.Net.Highlighter/overview.md",
+ "Lucene.Net.ICU/overview.md",
"Lucene.Net.Grouping/package.md",
"Lucene.Net.QueryParser/overview.md",
"Lucene.Net.Sandbox/overview.md",
diff --git a/websites/apidocs/index.md b/websites/apidocs/index.md
index 777265e..d57aefb 100644
--- a/websites/apidocs/index.md
+++ b/websites/apidocs/index.md
@@ -49,12 +49,12 @@ on some of the conceptual or inner details of Lucene:
- <xref:Lucene.Net.Analysis.Stempel> - Analyzer for indexing Polish
- [Lucene.Net.Benchmark](xref:Lucene.Net.Benchmarks) - System for benchmarking Lucene
- <xref:Lucene.Net.Classification> - Classification module for Lucene
-- <xref:Lucene.Net.Codecs> - Lucene codecs and postings formats
+- [Lucene.Net.Codecs](api/codecs/overview.html) - Lucene codecs and postings formats
- [Lucene.Net.Expressions](xref:Lucene.Net.Expressions) - Dynamically computed values to sort/facet/search on based on a pluggable grammar
- [Lucene.Net.Facet](xref:Lucene.Net.Facet) - Faceted indexing and search capabilities
- <xref:Lucene.Net.Grouping> - Collectors for grouping search results
- <xref:Lucene.Net.Search.Highlight> - Highlights search keywords in results
-- <xref:Lucene.Net.Analysis.Icu> - Specialized ICU (International Components for Unicode) Analyzers and Highlighters
+- <xref:Lucene.Net.ICU> - Specialized ICU (International Components for Unicode) Analyzers and Highlighters
- <xref:Lucene.Net.Join> - Index-time and Query-time joins for normalized content
- [Lucene.Net.Memory](xref:Lucene.Net.Index.Memory) - Single-document in-memory index implementation
- <xref:Lucene.Net.Misc> - Index tools and other miscellaneous code
[lucenenet] 05/06: docs.ps1: Update the version in toc/toc.yml on
build
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 38320152930cc419cfa504411f997ba19f45e678
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Mar 28 20:39:04 2021 +0700
docs.ps1: Update the version in toc/toc.yml on build
---
websites/apidocs/docs.ps1 | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/websites/apidocs/docs.ps1 b/websites/apidocs/docs.ps1
index 3180087..5de4cae 100644
--- a/websites/apidocs/docs.ps1
+++ b/websites/apidocs/docs.ps1
@@ -52,7 +52,9 @@ $RepoRoot = (get-item $PSScriptFilePath).Directory.Parent.Parent.FullName;
$ApiDocsFolder = Join-Path -Path $RepoRoot -ChildPath "websites\apidocs";
$ToolsFolder = Join-Path -Path $ApiDocsFolder -ChildPath "tools";
$CliIndexPath = Join-Path -Path $RepoRoot -ChildPath "src\dotnet\tools\lucene-cli\docs\index.md";
-$TocPath = Join-Path -Path $ApiDocsFolder -ChildPath "toc.yml"
+$TocPath1 = Join-Path -Path $ApiDocsFolder -ChildPath "toc.yml"
+$TocPath2 = Join-Path -Path $ApiDocsFolder -ChildPath "toc\toc.yml"
+
#ensure the /build/tools folder
New-Item $ToolsFolder -type directory -force
@@ -187,7 +189,8 @@ if ($? -and $DisableBuild -eq $false) {
(Get-Content -Path $CliIndexPath -Raw) -Replace '(?<=--version\s)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $CliIndexPath
# Update our TOC to the latest LuceneNetVersion
- (Get-Content -Path $TocPath -Raw) -Replace '(?<=lucenenet\.apache\.org\/docs\/)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $TocPath
+ (Get-Content -Path $TocPath1 -Raw) -Replace '(?<=lucenenet\.apache\.org\/docs\/)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $TocPath1
+ (Get-Content -Path $TocPath2 -Raw) -Replace '(?<=lucenenet\.apache\.org\/docs\/)\d+?\.\d+?\.\d+?(?:\.\d+?)?(?:-\w+)?', $LuceneNetVersion | Set-Content -Path $TocPath2
foreach ($proj in $DocFxJsonMeta) {
$projFile = Join-Path -Path $ApiDocsFolder $proj
[lucenenet] 03/06: docs: Lucene.Net.Analysis.Common: Fixed broken
formatting and links (see #284, #300)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit bffd9b066f3236789ba2871cf6f61e769eb7f044
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Mar 28 20:33:44 2021 +0700
docs: Lucene.Net.Analysis.Common: Fixed broken formatting and links (see #284, #300)
---
.../Analysis/CharFilter/package.md | 8 +-
.../Analysis/Cjk/package.md | 14 ++-
.../Analysis/Cn/package.md | 15 ++-
.../Analysis/Compound/package.md | 119 ++++++++++++---------
.../Analysis/Standard/Std31/package.md | 4 +-
.../Analysis/Standard/Std34/package.md | 4 +-
.../Analysis/Standard/Std36/package.md | 4 +-
.../Analysis/Standard/Std40/package.md | 4 +-
.../Analysis/Standard/package.md | 80 +++++++-------
9 files changed, 146 insertions(+), 106 deletions(-)
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/CharFilter/package.md b/src/Lucene.Net.Analysis.Common/Analysis/CharFilter/package.md
index 929671f..b3ddf5b 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/CharFilter/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/CharFilter/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.CharFilters
summary: *content
---
@@ -24,10 +24,10 @@ summary: *content
CharFilters are chainable filters that normalize text before tokenization and provide mappings between normalized text offsets and the corresponding offset in the original text.
-<h2>CharFilter offset mappings</h2>
+## CharFilter offset mappings
CharFilters modify an input stream via a series of substring replacements (including deletions and insertions) to produce an output stream. There are three possible replacement cases: the replacement string has the same length as the original substring; the replacement is shorter; and the replacement is longer. In the latter two cases (when the replacement has a different length than the original), one or more offset correction mappings are required.
- When the replacement is shorter than the original (e.g. when the replacement is the empty string), a single offset correction mapping should be added at the replacement's end offset in the output stream. The `cumulativeDiff` parameter to the `addOffCorrectMapping()` method will be the sum of all previous replacement offset adjustments, with the addition of the difference between the lengths of the original substring and the replacement string (a positive value).
+ When the replacement is shorter than the original (e.g. when the replacement is the empty string), a single offset correction mapping should be added at the replacement's end offset in the output stream. The `cumulativeDiff` parameter to the `AddOffCorrectMap()` method will be the sum of all previous replacement offset adjustments, with the addition of the difference between the lengths of the original substring and the replacement string (a positive value).
- When the replacement is longer than the original (e.g. when the original is the empty string), you should add as many offset correction mappings as the difference between the lengths of the replacement string and the original substring, starting at the end offset the original substring would have had in the output stream. The `cumulativeDiff` parameter to the `addOffCorrectMapping()` method will be the sum of all previous replacement offset adjustments, with the addition of the differen [...]
\ No newline at end of file
+ When the replacement is longer than the original (e.g. when the original is the empty string), you should add as many offset correction mappings as the difference between the lengths of the replacement string and the original substring, starting at the end offset the original substring would have had in the output stream. The `cumulativeDiff` parameter to the `AddOffCorrectMap()` method will be the sum of all previous replacement offset adjustments, with the addition of the difference b [...]
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Cjk/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Cjk/package.md
index 8f45e50..946ee7d 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Cjk/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Cjk/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Cjk
summary: *content
---
@@ -23,4 +23,14 @@ summary: *content
Analyzer for Chinese, Japanese, and Korean, which indexes bigrams.
This analyzer generates bigram terms, which are overlapping groups of two adjacent Han, Hiragana, Katakana, or Hangul characters.
- Three analyzers are provided for Chinese, each of which treats Chinese text in a different way. * ChineseAnalyzer (in the analyzers/cn package): Index unigrams (individual Chinese characters) as a token. * CJKAnalyzer (in this package): Index bigrams (overlapping groups of two adjacent Chinese characters) as tokens. * SmartChineseAnalyzer (in the analyzers/smartcn package): Index words (attempt to segment Chinese text into words) as tokens. Example phrase： "我是中国人" 1. ChineseAnalyzer: 我－ [...]
\ No newline at end of file
+ Three analyzers are provided for Chinese, each of which treats Chinese text in a different way.
+
+* ChineseAnalyzer (in the Lucene.Net.Analysis.Cn namespace): Index unigrams (individual Chinese characters) as a token.
+* CJKAnalyzer (in the Lucene.Net.Analysis.Cjk namespace): Index bigrams (overlapping groups of two adjacent Chinese characters) as tokens.
+* SmartChineseAnalyzer (in the Lucene.Net.Analysis.SmartCn package): Index words (attempt to segment Chinese text into words) as tokens.
+
+Example phrase： "我是中国人"
+
+1. ChineseAnalyzer: 我－是－中－国－人
+2. CJKAnalyzer: 我是－是中－中国－国人
+3. SmartChineseAnalyzer: 我－是－中国－人
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Cn/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Cn/package.md
index a0a8f74..aa00067 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Cn/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Cn/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Cn
summary: *content
---
@@ -22,4 +22,15 @@ summary: *content
Analyzer for Chinese, which indexes unigrams (individual chinese characters).
- Three analyzers are provided for Chinese, each of which treats Chinese text in a different way. * StandardAnalyzer: Index unigrams (individual Chinese characters) as a token. * CJKAnalyzer (in the analyzers/cjk package): Index bigrams (overlapping groups of two adjacent Chinese characters) as tokens. * SmartChineseAnalyzer (in the analyzers/smartcn package): Index words (attempt to segment Chinese text into words) as tokens. Example phrase： "我是中国人" 1. StandardAnalyzer: 我－是－中－国－人 2. CJKA [...]
\ No newline at end of file
+ Three analyzers are provided for Chinese, each of which treats Chinese text in a different way.
+
+* StandardAnalyzer: Index unigrams (individual Chinese characters) as a token.
+* CJKAnalyzer (in the Lucene.Net.Analysis.Cjk namespace): Index bigrams (overlapping groups of two adjacent Chinese characters) as tokens.
+* SmartChineseAnalyzer (in the Lucene.Net.Analysis.SmartCn package): Index words (attempt to segment Chinese text into words) as tokens.
+
+
+Example phrase： "我是中国人"
+
+1. StandardAnalyzer: 我－是－中－国－人
+2. CJKAnalyzer: 我是－是中－中国－国人
+3. SmartChineseAnalyzer: 我－是－中国－人
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Compound/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Compound/package.md
index a56f95c..6560069 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Compound/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Compound/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Compound
summary: *content
---
@@ -147,56 +147,77 @@ This decision matrix should help you:
### Examples
- public void testHyphenationCompoundWordsDE() throws Exception {
- String[] dict = { "Rind", "Fleisch", "Draht", "Schere", "Gesetz",
- "Aufgabe", "Überwachung" };
-
- Reader reader = new FileReader("de_DR.xml");
-
- HyphenationTree hyphenator = HyphenationCompoundWordTokenFilter
- .getHyphenationTree(reader);
-
+```cs
+const LuceneVersion matchVersion = LuceneVersion.LUCENE_48;
+
+[Test]
+public void TestHyphenationCompoundWordsDE()
+{
+ string[] dictionary = {
+ "Rind", "Fleisch", "Draht", "Schere", "Gesetz",
+ "Aufgabe", "Überwachung" };
+
+ using Stream stream = new FileStream("de_DR.xml", FileMode.Open);
+
+ HyphenationTree hyphenator = HyphenationCompoundWordTokenFilter.GetHyphenationTree(stream);
+
HyphenationCompoundWordTokenFilter tf = new HyphenationCompoundWordTokenFilter(
- new WhitespaceTokenizer(new StringReader(
- "Rindfleischüberwachungsgesetz Drahtschere abba")), hyphenator,
- dict, CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ matchVersion
+
+ new WhitespaceTokenizer(
+ matchVersion,
+ new StringReader("Rindfleischüberwachungsgesetz Drahtschere abba")),
+ hyphenator,
+ dictionary,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
- CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE, false);
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE,
+ onlyLongestMatch: false);
+
+ ICharTermAttribute t = tf.AddAttribute<ICharTermAttribute>();
+ while (tf.IncrementToken())
+ {
+ Console.WriteLine(t);
+ }
+}
+
+[Test]
+public void TestHyphenationCompoundWordsWithoutDictionaryDE()
+{
+ using Stream stream = new FileStream("de_DR.xml", FileMode.Open);
+
+ HyphenationTree hyphenator = HyphenationCompoundWordTokenFilter.GetHyphenationTree(stream);
- CharTermAttribute t = tf.addAttribute(CharTermAttribute.class);
- while (tf.incrementToken()) {
- System.out.println(t);
- }
- }
-
- public void testHyphenationCompoundWordsWithoutDictionaryDE() throws Exception {
- Reader reader = new FileReader("de_DR.xml");
-
- HyphenationTree hyphenator = HyphenationCompoundWordTokenFilter
- .getHyphenationTree(reader);
-
HyphenationCompoundWordTokenFilter tf = new HyphenationCompoundWordTokenFilter(
- new WhitespaceTokenizer(new StringReader(
- "Rindfleischüberwachungsgesetz Drahtschere abba")), hyphenator);
-
- CharTermAttribute t = tf.addAttribute(CharTermAttribute.class);
- while (tf.incrementToken()) {
- System.out.println(t);
- }
- }
-
- public void testDumbCompoundWordsSE() throws Exception {
- String[] dict = { "Bil", "Dörr", "Motor", "Tak", "Borr", "Slag", "Hammar",
- "Pelar", "Glas", "Ögon", "Fodral", "Bas", "Fiol", "Makare", "Gesäll",
- "Sko", "Vind", "Rute", "Torkare", "Blad" };
-
+ new WhitespaceTokenizer(matchVersion,
+ new StringReader("Rindfleischüberwachungsgesetz Drahtschere abba")),
+ hyphenator);
+
+ ICharTermAttribute t = tf.AddAttribute<ICharTermAttribute>();
+ while (tf.IncrementToken())
+ {
+ Console.WriteLine(t);
+ }
+}
+
+[Test]
+public void TestDumbCompoundWordsSE()
+{
+ string[] dictionary = {
+ "Bil", "Dörr", "Motor", "Tak", "Borr", "Slag", "Hammar",
+ "Pelar", "Glas", "Ögon", "Fodral", "Bas", "Fiol", "Makare", "Gesäll",
+ "Sko", "Vind", "Rute", "Torkare", "Blad" };
+
DictionaryCompoundWordTokenFilter tf = new DictionaryCompoundWordTokenFilter(
- new WhitespaceTokenizer(
- new StringReader(
- "Bildörr Bilmotor Biltak Slagborr Hammarborr Pelarborr Glasögonfodral Basfiolsfodral Basfiolsfodralmakaregesäll Skomakare Vindrutetorkare Vindrutetorkarblad abba")),
- dict);
- CharTermAttribute t = tf.addAttribute(CharTermAttribute.class);
- while (tf.incrementToken()) {
- System.out.println(t);
- }
- }
\ No newline at end of file
+ new WhitespaceTokenizer(
+ matchVersion,
+ new StringReader(
+ "Bildörr Bilmotor Biltak Slagborr Hammarborr Pelarborr Glasögonfodral Basfiolsfodral Basfiolsfodralmakaregesäll Skomakare Vindrutetorkare Vindrutetorkarblad abba")),
+ dictionary);
+ ICharTermAttribute t = tf.AddAttribute<ICharTermAttribute>();
+ while (tf.IncrementToken())
+ {
+ Console.WriteLine(t);
+ }
+}
+```
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std31/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std31/package.md
index 8901325..3894608 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std31/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std31/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Standard.Std31
summary: *content
---
@@ -20,4 +20,4 @@ summary: *content
limitations under the License.
-->
-Backwards-compatible implementation to match [#LUCENE_31](xref:Lucene.Net.Util.Version)
\ No newline at end of file
+Backwards-compatible implementation to match [LuceneVersion.LUCENE_31](xref:Lucene.Net.Util.LuceneVersion#Lucene_Net_Util_LuceneVersion_LUCENE_31)
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std34/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std34/package.md
index 0c8297f..c49592d 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std34/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std34/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Standard.Std34
summary: *content
---
@@ -20,4 +20,4 @@ summary: *content
limitations under the License.
-->
-Backwards-compatible implementation to match [#LUCENE_34](xref:Lucene.Net.Util.Version)
\ No newline at end of file
+Backwards-compatible implementation to match [LuceneVersion.LUCENE_34](xref:Lucene.Net.Util.LuceneVersion#Lucene_Net_Util_LuceneVersion_LUCENE_34)
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std36/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std36/package.md
index c12cfaa..50b81da 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std36/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std36/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Standard.Std36
summary: *content
---
@@ -20,4 +20,4 @@ summary: *content
limitations under the License.
-->
-Backwards-compatible implementation to match [#LUCENE_36](xref:Lucene.Net.Util.Version)
\ No newline at end of file
+Backwards-compatible implementation to match [LuceneVersion.LUCENE_36](xref:Lucene.Net.Util.LuceneVersion#Lucene_Net_Util_LuceneVersion_LUCENE_36)
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std40/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std40/package.md
index 62c466b..50f60cb 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std40/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Standard/Std40/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Standard.Std40
summary: *content
---
@@ -20,4 +20,4 @@ summary: *content
limitations under the License.
-->
-Backwards-compatible implementation to match [#LUCENE_40](xref:Lucene.Net.Util.Version)
\ No newline at end of file
+Backwards-compatible implementation to match [LuceneVersion.LUCENE_40](xref:Lucene.Net.Util.LuceneVersion#Lucene_Net_Util_LuceneVersion_LUCENE_40)
\ No newline at end of file
diff --git a/src/Lucene.Net.Analysis.Common/Analysis/Standard/package.md b/src/Lucene.Net.Analysis.Common/Analysis/Standard/package.md
index 9a3f1d1..d2202e3 100644
--- a/src/Lucene.Net.Analysis.Common/Analysis/Standard/package.md
+++ b/src/Lucene.Net.Analysis.Common/Analysis/Standard/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Analysis.Standard
summary: *content
---
@@ -22,43 +22,41 @@ summary: *content
Fast, general-purpose grammar-based tokenizers.
-The `org.apache.lucene.analysis.standard` package contains three fast grammar-based tokenizers constructed with JFlex:
-
-* <xref:Lucene.Net.Analysis.Standard.StandardTokenizer>:
- as of Lucene 3.1, implements the Word Break rules from the Unicode Text
- Segmentation algorithm, as specified in
- [Unicode Standard Annex #29](http://unicode.org/reports/tr29/).
- Unlike `UAX29URLEmailTokenizer`, URLs and email addresses are
- __not__ tokenized as single tokens, but are instead split up into
- tokens according to the UAX#29 word break rules.
-
- [StandardAnalyzer](xref:Lucene.Net.Analysis.Standard.StandardAnalyzer) includes
- [StandardTokenizer](xref:Lucene.Net.Analysis.Standard.StandardTokenizer),
- [StandardFilter](xref:Lucene.Net.Analysis.Standard.StandardFilter),
- [LowerCaseFilter](xref:Lucene.Net.Analysis.Core.LowerCaseFilter)
- and [StopFilter](xref:Lucene.Net.Analysis.Core.StopFilter).
- When the `Version` specified in the constructor is lower than
- 3.1, the [ClassicTokenizer](xref:Lucene.Net.Analysis.Standard.ClassicTokenizer)
- implementation is invoked.
-
-* [ClassicTokenizer](xref:Lucene.Net.Analysis.Standard.ClassicTokenizer):
- this class was formerly (prior to Lucene 3.1) named
- `StandardTokenizer`. (Its tokenization rules are not
- based on the Unicode Text Segmentation algorithm.)
- [ClassicAnalyzer](xref:Lucene.Net.Analysis.Standard.ClassicAnalyzer) includes
- [ClassicTokenizer](xref:Lucene.Net.Analysis.Standard.ClassicTokenizer),
- [StandardFilter](xref:Lucene.Net.Analysis.Standard.StandardFilter),
- [LowerCaseFilter](xref:Lucene.Net.Analysis.Core.LowerCaseFilter)
- and [StopFilter](xref:Lucene.Net.Analysis.Core.StopFilter).
-
-* [UAX29URLEmailTokenizer](xref:Lucene.Net.Analysis.Standard.UAX29URLEmailTokenizer):
- implements the Word Break rules from the Unicode Text Segmentation
- algorithm, as specified in
- [Unicode Standard Annex #29](http://unicode.org/reports/tr29/).
- URLs and email addresses are also tokenized according to the relevant RFCs.
-
- [UAX29URLEmailAnalyzer](xref:Lucene.Net.Analysis.Standard.UAX29URLEmailAnalyzer) includes
- [UAX29URLEmailTokenizer](xref:Lucene.Net.Analysis.Standard.UAX29URLEmailTokenizer),
- [StandardFilter](xref:Lucene.Net.Analysis.Standard.StandardFilter),
- [LowerCaseFilter](xref:Lucene.Net.Analysis.Core.LowerCaseFilter)
- and [StopFilter](xref:Lucene.Net.Analysis.Core.StopFilter).
\ No newline at end of file
+The <xref:Lucene.Net.Analysis.Standard> package contains three fast grammar-based tokenizers constructed with JFlex:
+
+* <xref:Lucene.Net.Analysis.Standard.StandardTokenizer>:
+ as of Lucene 3.1, implements the Word Break rules from the Unicode Text
+ Segmentation algorithm, as specified in
+ [Unicode Standard Annex #29](http://unicode.org/reports/tr29/).
+ Unlike `UAX29URLEmailTokenizer`, URLs and email addresses are
+ __not__ tokenized as single tokens, but are instead split up into
+ tokens according to the UAX#29 word break rules.<br/><br/>
+ [StandardAnalyzer](xref:Lucene.Net.Analysis.Standard.StandardAnalyzer) includes
+ [StandardTokenizer](xref:Lucene.Net.Analysis.Standard.StandardTokenizer),
+ [StandardFilter](xref:Lucene.Net.Analysis.Standard.StandardFilter),
+ [LowerCaseFilter](xref:Lucene.Net.Analysis.Core.LowerCaseFilter)
+ and [StopFilter](xref:Lucene.Net.Analysis.Core.StopFilter).
+ When the `LuceneVersion` specified in the constructor is lower than
+ 3.1, the [ClassicTokenizer](xref:Lucene.Net.Analysis.Standard.ClassicTokenizer)
+ implementation is invoked.
+
+* [ClassicTokenizer](xref:Lucene.Net.Analysis.Standard.ClassicTokenizer):
+ this class was formerly (prior to Lucene 3.1) named
+ `StandardTokenizer`. (Its tokenization rules are not
+ based on the Unicode Text Segmentation algorithm.)
+ [ClassicAnalyzer](xref:Lucene.Net.Analysis.Standard.ClassicAnalyzer) includes
+ [ClassicTokenizer](xref:Lucene.Net.Analysis.Standard.ClassicTokenizer),
+ [StandardFilter](xref:Lucene.Net.Analysis.Standard.StandardFilter),
+ [LowerCaseFilter](xref:Lucene.Net.Analysis.Core.LowerCaseFilter)
+ and [StopFilter](xref:Lucene.Net.Analysis.Core.StopFilter).
+
+* [UAX29URLEmailTokenizer](xref:Lucene.Net.Analysis.Standard.UAX29URLEmailTokenizer):
+ implements the Word Break rules from the Unicode Text Segmentation
+ algorithm, as specified in
+ [Unicode Standard Annex #29](http://unicode.org/reports/tr29/).
+ URLs and email addresses are also tokenized according to the relevant RFCs.<br/><br/>
+ [UAX29URLEmailAnalyzer](xref:Lucene.Net.Analysis.Standard.UAX29URLEmailAnalyzer) includes
+ [UAX29URLEmailTokenizer](xref:Lucene.Net.Analysis.Standard.UAX29URLEmailTokenizer),
+ [StandardFilter](xref:Lucene.Net.Analysis.Standard.StandardFilter),
+ [LowerCaseFilter](xref:Lucene.Net.Analysis.Core.LowerCaseFilter)
+ and [StopFilter](xref:Lucene.Net.Analysis.Core.StopFilter).
\ No newline at end of file
[lucenenet] 02/06: docs: Lucene.Net: Fixed remaining broken
formatting and links (see #284, #300)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 421b96141bd96234b768d4e47ca5abfb5d18e2b8
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Mar 28 20:32:40 2021 +0700
docs: Lucene.Net: Fixed remaining broken formatting and links (see #284, #300)
---
src/Lucene.Net/Codecs/Lucene40/package.md | 121 ++++++++------------
src/Lucene.Net/Codecs/Lucene41/package.md | 117 ++++++++-----------
src/Lucene.Net/Codecs/Lucene42/package.md | 124 ++++++++-------------
src/Lucene.Net/Codecs/Lucene45/package.md | 127 ++++++++-------------
src/Lucene.Net/Codecs/Lucene46/package.md | 130 +++++++++------------
src/Lucene.Net/Search/Payloads/package.md | 7 +-
src/Lucene.Net/Search/Similarities/package.md | 33 ++++--
src/Lucene.Net/Search/Spans/package.md | 39 ++++---
src/Lucene.Net/Search/package.md | 155 ++++++++++++++++----------
src/Lucene.Net/Util/Fst/package.md | 110 ++++++++++--------
src/Lucene.Net/Util/Packed/package.md | 25 +++--
11 files changed, 473 insertions(+), 515 deletions(-)
diff --git a/src/Lucene.Net/Codecs/Lucene40/package.md b/src/Lucene.Net/Codecs/Lucene40/package.md
index 250ca84..4ed138d 100644
--- a/src/Lucene.Net/Codecs/Lucene40/package.md
+++ b/src/Lucene.Net/Codecs/Lucene40/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Codecs.Lucene40
summary: *content
---
@@ -24,59 +24,51 @@ Lucene 4.0 file format.
# Apache Lucene - Index File Formats
-<div>
-* [Introduction](#introduction)
+* [Introduction](#introduction)
-* [Definitions](#definitions)
+* [Definitions](#definitions)
- * [Inverted Indexing](#inverted-indexing)
+ * [Inverted Indexing](#inverted-indexing)
- * [Types of Fields](#types-of-fields)
+ * [Types of Fields](#types-of-fields)
- * [Segments](#segments)
+ * [Segments](#segments)
- * [Document Numbers](#document-numbers)
+ * [Document Numbers](#document-numbers)
-* [Index Structure Overview](#overview)
+* [Index Structure Overview](#index-structure-overview)
-* [File Naming](#file-naming)
+* [File Naming](#file-naming)
-* [Summary of File Extensions](#file-names)
+* [Summary of File Extensions](#summary-of-file-extensions)
- * [Lock File](#lock-file)
+ * [Lock File](#lock-file)
- * [History](#history)
+ * [History](#history)
- * [Limitations](#limitations)
+ * [Limitations](#limitations)
-</div>
## Introduction
-<div>
-
This document defines the index file formats used in this version of Lucene. If you are using a different version of Lucene, please consult the copy of `docs/` that was distributed with the version you are using.
-Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations). If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
+Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations) including this implementation in .NET. If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
As Lucene evolves, this document should evolve. Versions of Lucene in different programming languages should endeavor to agree on file formats, and generate new versions of this document.
-</div>
-
## Definitions
-<div>
-
The fundamental concepts in Lucene are index, document, field and term.
An index contains a sequence of documents.
-* A document is a sequence of fields.
+* A document is a sequence of fields.
-* A field is a named sequence of terms.
+* A field is a named sequence of terms.
-* A term is a sequence of bytes.
+* A term is a sequence of bytes.
The same sequence of bytes in two different fields is considered a different term. Thus terms are represented as a pair: the string naming the field, and the bytes within the field.
@@ -90,7 +82,7 @@ In Lucene, fields may be _stored_, in which case their text is stored in the ind
The text of a field may be _tokenized_ into terms to be indexed, or the text of a field may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for certain identifier fields to be indexed literally.
-See the [Field](xref:Lucene.Net.Documents.Field) java docs for more information on Fields.
+See the [Field](xref:Lucene.Net.Documents.Field) docs for more information on Fields.
### Segments
@@ -108,90 +100,74 @@ Internally, Lucene refers to documents by an integer _document number_. The firs
Note that a document's number may change, so caution should be taken when storing these numbers outside of Lucene. In particular, numbers may change in the following situations:
-*
-
-The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the external [...]
-
-*
-
-When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
+* The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the extern [...]
-</div>
+* When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
## Index Structure Overview
-<div>
-
Each segment index maintains the following:
-* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
+* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
This contains metadata about a segment, such as the number of documents,
what files it uses,
-* [Field names](xref:Lucene.Net.Codecs.Lucene40.Lucene40FieldInfosFormat).
+* [Field names](xref:Lucene.Net.Codecs.Lucene40.Lucene40FieldInfosFormat).
This contains the set of field names used in the index.
-* [Stored Field values](xref:Lucene.Net.Codecs.Lucene40.Lucene40StoredFieldsFormat).
+* [Stored Field values](xref:Lucene.Net.Codecs.Lucene40.Lucene40StoredFieldsFormat).
This contains, for each document, a list of attribute-value pairs, where the attributes
are field names. These are used to store auxiliary information about the document, such as
its title, url, or an identifier to access a database. The set of stored fields are what is
returned for each hit when searching. This is keyed by document number.
-* [Term dictionary](xref:Lucene.Net.Codecs.Lucene40.Lucene40PostingsFormat).
+* [Term dictionary](xref:Lucene.Net.Codecs.Lucene40.Lucene40PostingsFormat).
A dictionary containing all of the terms used in all of the
indexed fields of all of the documents. The dictionary also contains the number
of documents which contain the term, and pointers to the term's frequency and
proximity data.
-* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene40.Lucene40PostingsFormat).
+* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene40.Lucene40PostingsFormat).
For each term in the dictionary, the numbers of all the
documents that contain that term, and the frequency of the term in that
document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
-* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene40.Lucene40PostingsFormat).
+* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene40.Lucene40PostingsFormat).
For each term in the dictionary, the positions that the
term occurs in each document. Note that this will not exist if all fields in
all documents omit position data.
-* [Normalization factors](xref:Lucene.Net.Codecs.Lucene40.Lucene40NormsFormat).
+* [Normalization factors](xref:Lucene.Net.Codecs.Lucene40.Lucene40NormsFormat).
For each field in each document, a value is stored
that is multiplied into the score for hits on that field.
-* [Term Vectors](xref:Lucene.Net.Codecs.Lucene40.Lucene40TermVectorsFormat).
+* [Term Vectors](xref:Lucene.Net.Codecs.Lucene40.Lucene40TermVectorsFormat).
For each field in each document, the term vector (sometimes
called document vector) may be stored. A term vector consists of term text and
term frequency. To add Term Vectors to your index see the
[Field](xref:Lucene.Net.Documents.Field) constructors
-* [Per-document values](xref:Lucene.Net.Codecs.Lucene40.Lucene40DocValuesFormat).
+* [Per-document values](xref:Lucene.Net.Codecs.Lucene40.Lucene40DocValuesFormat).
Like stored values, these are also keyed by document
number, but are generally intended to be loaded into main memory for fast
access. Whereas stored values are generally intended for summary results from
searches, per-document values are useful for things like scoring factors.
-* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
+* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
An optional file indicating which documents are deleted.
Details on each of these are provided in their linked pages.
-</div>
-
## File Naming
-<div>
-
All files belonging to a segment have the same name with varying extensions. The extensions correspond to the different file formats described below. When using the Compound File format (default in 1.4 and greater) these files (except for the Segment info file, the Lock file, and Deleted documents file) are collapsed into a single .cfs file (see below for details)
Typically, all segments in an index are stored in a single directory, although this is not required.
-As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is segments_1, then segments_2, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
-
-</div>
+As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is `segments_1`, then `segments_2`, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
## Summary of File Extensions
-<div>
-
The following table summarizes the names and extensions of the files in Lucene:
<table cellspacing="1" cellpadding="4">
@@ -288,13 +264,12 @@ systems that frequently run out of file handles.</td>
<td>Info about what files are deleted</td>
</tr>
</table>
-</div>
## Lock File
The write lock, which is stored in the index directory by default, is named
-"write.lock". If the lock directory is different from the index directory then
-the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix
+`write.lock`. If the lock directory is different from the index directory then
+the write lock will be named `XXXX-write.lock` where XXXX is a unique prefix
derived from the full path to the index directory. When this file is present, a
writer is currently modifying the index (adding or removing documents). This
lock file ensures that only one writer is modifying the index at a time.
@@ -303,50 +278,50 @@ lock file ensures that only one writer is modifying the index at a time.
Compatibility notes are provided in this document, describing how file formats have changed from prior versions:
-* In version 2.1, the file format was changed to allow lock-less commits (ie,
+* In version 2.1, the file format was changed to allow lock-less commits (ie,
no more commit lock). The change is fully backwards compatible: you can open a
pre-2.1 index for searching or adding/deleting of docs. When the new segments
file is saved (committed), it will be written in the new file format (meaning
no specific "upgrade" process is needed). But note that once a commit has
occurred, pre-2.1 Lucene will not be able to read the index.
-* In version 2.3, the file format was changed to allow segments to share a
+* In version 2.3, the file format was changed to allow segments to share a
single set of doc store (vectors & stored fields) files. This allows for
faster indexing in certain cases. The change is fully backwards compatible (in
the same way as the lock-less commits change in 2.1).
-* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
+* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
Java's modified UTF-8. See [
LUCENE-510](http://issues.apache.org/jira/browse/LUCENE-510) for details.
-* In version 2.9, an optional opaque Map<String,String> CommitUserData
+* In version 2.9, an optional opaque `IDictionary<string, string>` CommitUserData
may be passed to IndexWriter's commit methods (and later retrieved), which is
-recorded in the segments_N file. See [
-LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
+recorded in the `segments_N` file. See
+[LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
diagnostics were added to each segment written recording details about why it
was written (due to flush, merge; which OS/JRE was used; etc.). See issue
[LUCENE-1654](http://issues.apache.org/jira/browse/LUCENE-1654) for details.
-* In version 3.0, compressed fields are no longer written to the index (they
+* In version 3.0, compressed fields are no longer written to the index (they
can still be read, but on merge the new segment will write them, uncompressed).
See issue [LUCENE-1960](http://issues.apache.org/jira/browse/LUCENE-1960)
for details.
-* In version 3.1, segments records the code version that created them. See
+* In version 3.1, segments records the code version that created them. See
[LUCENE-2720](http://issues.apache.org/jira/browse/LUCENE-2720) for details.
Additionally segments track explicitly whether or not they have term vectors.
See [LUCENE-2811](http://issues.apache.org/jira/browse/LUCENE-2811)
for details.
-* In version 3.2, numeric fields are written as natively to stored fields
+* In version 3.2, numeric fields are written as natively to stored fields
file, previously they were stored in text format only.
-* In version 3.4, fields can omit position data while still indexing term
+* In version 3.4, fields can omit position data while still indexing term
frequencies.
-* In version 4.0, the format of the inverted index became extensible via
+* In version 4.0, the format of the inverted index became extensible via
the [Codec](xref:Lucene.Net.Codecs.Codec) api. Fast per-document storage
-({@code DocValues}) was introduced. Normalization factors need no longer be a
+(`DocValues`) was introduced. Normalization factors need no longer be a
single byte, they can be any [NumericDocValues](xref:Lucene.Net.Index.NumericDocValues).
Terms need not be unicode strings, they can be any byte sequence. Term offsets
can optionally be indexed into the postings lists. Payloads can be stored in the
@@ -354,8 +329,4 @@ term vectors.
## Limitations
-<div>
-
-Lucene uses a Java `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#methods) values which have no limit.
-
-</div>
\ No newline at end of file
+Lucene uses a .NET `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#Lucene_Net_Store_DataOutput_WriteVInt32_System_Int32_) values which have no limit.
\ No newline at end of file
diff --git a/src/Lucene.Net/Codecs/Lucene41/package.md b/src/Lucene.Net/Codecs/Lucene41/package.md
index 50fd77a..0c0dd15 100644
--- a/src/Lucene.Net/Codecs/Lucene41/package.md
+++ b/src/Lucene.Net/Codecs/Lucene41/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Codecs.Lucene41
summary: *content
---
@@ -24,59 +24,51 @@ Lucene 4.1 file format.
# Apache Lucene - Index File Formats
-<div>
-* [Introduction](#introduction)
+* [Introduction](#introduction)
-* [Definitions](#definitions)
+* [Definitions](#definitions)
- * [Inverted Indexing](#inverted-indexing)
+ * [Inverted Indexing](#inverted-indexing)
- * [Types of Fields](#types-of-fields)
+ * [Types of Fields](#types-of-fields)
- * [Segments](#segments)
+ * [Segments](#segments)
- * [Document Numbers](#document-numbers)
+ * [Document Numbers](#document-numbers)
-* [Index Structure Overview](#overview)
+* [Index Structure Overview](#index-structure-overview)
-* [File Naming](#file-naming)
+* [File Naming](#file-naming)
-* [Summary of File Extensions](#file-names)
+* [Summary of File Extensions](#summary-of-file-extensions)
- * [Lock File](#lock-file)
+ * [Lock File](#lock-file)
- * [History](#history)
+ * [History](#history)
- * [Limitations](#limitations)
+ * [Limitations](#limitations)
-</div>
## Introduction
-<div>
-
This document defines the index file formats used in this version of Lucene. If you are using a different version of Lucene, please consult the copy of `docs/` that was distributed with the version you are using.
-Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations). If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
+Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations) including this implementation in .NET. If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
As Lucene evolves, this document should evolve. Versions of Lucene in different programming languages should endeavor to agree on file formats, and generate new versions of this document.
-</div>
-
## Definitions
-<div>
-
The fundamental concepts in Lucene are index, document, field and term.
An index contains a sequence of documents.
-* A document is a sequence of fields.
+* A document is a sequence of fields.
-* A field is a named sequence of terms.
+* A field is a named sequence of terms.
-* A term is a sequence of bytes.
+* A term is a sequence of bytes.
The same sequence of bytes in two different fields is considered a different term. Thus terms are represented as a pair: the string naming the field, and the bytes within the field.
@@ -90,7 +82,7 @@ In Lucene, fields may be _stored_, in which case their text is stored in the ind
The text of a field may be _tokenized_ into terms to be indexed, or the text of a field may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for certain identifier fields to be indexed literally.
-See the [Field](xref:Lucene.Net.Documents.Field) java docs for more information on Fields.
+See the [Field](xref:Lucene.Net.Documents.Field) docs for more information on Fields.
### Segments
@@ -108,90 +100,74 @@ Internally, Lucene refers to documents by an integer _document number_. The firs
Note that a document's number may change, so caution should be taken when storing these numbers outside of Lucene. In particular, numbers may change in the following situations:
-*
-
-The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the external [...]
-
-*
-
-When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
+* The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the extern [...]
-</div>
+* When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
## Index Structure Overview
-<div>
-
Each segment index maintains the following:
-* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
+* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
This contains metadata about a segment, such as the number of documents,
what files it uses,
-* [Field names](xref:Lucene.Net.Codecs.Lucene40.Lucene40FieldInfosFormat).
+* [Field names](xref:Lucene.Net.Codecs.Lucene40.Lucene40FieldInfosFormat).
This contains the set of field names used in the index.
-* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
+* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
This contains, for each document, a list of attribute-value pairs, where the attributes
are field names. These are used to store auxiliary information about the document, such as
its title, url, or an identifier to access a database. The set of stored fields are what is
returned for each hit when searching. This is keyed by document number.
-* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
A dictionary containing all of the terms used in all of the
indexed fields of all of the documents. The dictionary also contains the number
of documents which contain the term, and pointers to the term's frequency and
proximity data.
-* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the numbers of all the
documents that contain that term, and the frequency of the term in that
document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
-* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the positions that the
term occurs in each document. Note that this will not exist if all fields in
all documents omit position data.
-* [Normalization factors](xref:Lucene.Net.Codecs.Lucene40.Lucene40NormsFormat).
+* [Normalization factors](xref:Lucene.Net.Codecs.Lucene40.Lucene40NormsFormat).
For each field in each document, a value is stored
that is multiplied into the score for hits on that field.
-* [Term Vectors](xref:Lucene.Net.Codecs.Lucene40.Lucene40TermVectorsFormat).
+* [Term Vectors](xref:Lucene.Net.Codecs.Lucene40.Lucene40TermVectorsFormat).
For each field in each document, the term vector (sometimes
called document vector) may be stored. A term vector consists of term text and
term frequency. To add Term Vectors to your index see the
[Field](xref:Lucene.Net.Documents.Field) constructors
-* [Per-document values](xref:Lucene.Net.Codecs.Lucene40.Lucene40DocValuesFormat).
+* [Per-document values](xref:Lucene.Net.Codecs.Lucene40.Lucene40DocValuesFormat).
Like stored values, these are also keyed by document
number, but are generally intended to be loaded into main memory for fast
access. Whereas stored values are generally intended for summary results from
searches, per-document values are useful for things like scoring factors.
-* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
+* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
An optional file indicating which documents are deleted.
Details on each of these are provided in their linked pages.
-</div>
-
## File Naming
-<div>
-
All files belonging to a segment have the same name with varying extensions. The extensions correspond to the different file formats described below. When using the Compound File format (default in 1.4 and greater) these files (except for the Segment info file, the Lock file, and Deleted documents file) are collapsed into a single .cfs file (see below for details)
Typically, all segments in an index are stored in a single directory, although this is not required.
-As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is segments_1, then segments_2, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
-
-</div>
+As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is `segments_1`, then `segments_2`, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
## Summary of File Extensions
-<div>
-
The following table summarizes the names and extensions of the files in Lucene:
<table cellspacing="1" cellpadding="4">
@@ -293,13 +269,12 @@ systems that frequently run out of file handles.</td>
<td>Info about what files are deleted</td>
</tr>
</table>
-</div>
## Lock File
The write lock, which is stored in the index directory by default, is named
-"write.lock". If the lock directory is different from the index directory then
-the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix
+`write.lock`. If the lock directory is different from the index directory then
+the write lock will be named `XXXX-write.lock` where XXXX is a unique prefix
derived from the full path to the index directory. When this file is present, a
writer is currently modifying the index (adding or removing documents). This
lock file ensures that only one writer is modifying the index at a time.
@@ -324,48 +299,44 @@ the same way as the lock-less commits change in 2.1).
Java's modified UTF-8. See [
LUCENE-510](http://issues.apache.org/jira/browse/LUCENE-510) for details.
-* In version 2.9, an optional opaque Map<String,String> CommitUserData
+* In version 2.9, an optional opaque `IDictionary<string, string>` CommitUserData
may be passed to IndexWriter's commit methods (and later retrieved), which is
-recorded in the segments_N file. See [
-LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
+recorded in the `segments_N` file. See
+[LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
diagnostics were added to each segment written recording details about why it
was written (due to flush, merge; which OS/JRE was used; etc.). See issue
[LUCENE-1654](http://issues.apache.org/jira/browse/LUCENE-1654) for details.
-* In version 3.0, compressed fields are no longer written to the index (they
+* In version 3.0, compressed fields are no longer written to the index (they
can still be read, but on merge the new segment will write them, uncompressed).
See issue [LUCENE-1960](http://issues.apache.org/jira/browse/LUCENE-1960)
for details.
-* In version 3.1, segments records the code version that created them. See
+* In version 3.1, segments records the code version that created them. See
[LUCENE-2720](http://issues.apache.org/jira/browse/LUCENE-2720) for details.
Additionally segments track explicitly whether or not they have term vectors.
See [LUCENE-2811](http://issues.apache.org/jira/browse/LUCENE-2811)
for details.
-* In version 3.2, numeric fields are written as natively to stored fields
+* In version 3.2, numeric fields are written as natively to stored fields
file, previously they were stored in text format only.
-* In version 3.4, fields can omit position data while still indexing term
+* In version 3.4, fields can omit position data while still indexing term
frequencies.
-* In version 4.0, the format of the inverted index became extensible via
+* In version 4.0, the format of the inverted index became extensible via
the [Codec](xref:Lucene.Net.Codecs.Codec) api. Fast per-document storage
-({@code DocValues}) was introduced. Normalization factors need no longer be a
+(`DocValues`) was introduced. Normalization factors need no longer be a
single byte, they can be any [NumericDocValues](xref:Lucene.Net.Index.NumericDocValues).
Terms need not be unicode strings, they can be any byte sequence. Term offsets
can optionally be indexed into the postings lists. Payloads can be stored in the
term vectors.
-* In version 4.1, the format of the postings list changed to use either
+* In version 4.1, the format of the postings list changed to use either
of FOR compression or variable-byte encoding, depending upon the frequency
of the term. Terms appearing only once were changed to inline directly into
the term dictionary. Stored fields are compressed by default.
## Limitations
-<div>
-
-Lucene uses a Java `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#methods) values which have no limit.
-
-</div>
\ No newline at end of file
+Lucene uses a .NET `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#Lucene_Net_Store_DataOutput_WriteVInt32_System_Int32_) values which have no limit.
\ No newline at end of file
diff --git a/src/Lucene.Net/Codecs/Lucene42/package.md b/src/Lucene.Net/Codecs/Lucene42/package.md
index 3c79cf3..db4d873 100644
--- a/src/Lucene.Net/Codecs/Lucene42/package.md
+++ b/src/Lucene.Net/Codecs/Lucene42/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Codecs.Lucene42
summary: *content
---
@@ -24,59 +24,51 @@ Lucene 4.2 file format.
# Apache Lucene - Index File Formats
-<div>
-* [Introduction](#introduction)
+* [Introduction](#introduction)
-* [Definitions](#definitions)
+* [Definitions](#definitions)
- * [Inverted Indexing](#inverted-indexing)
+ * [Inverted Indexing](#inverted-indexing)
- * [Types of Fields](#types-of-fields)
+ * [Types of Fields](#types-of-fields)
- * [Segments](#segments)
+ * [Segments](#segments)
- * [Document Numbers](#document-numbers)
+ * [Document Numbers](#document-numbers)
-* [Index Structure Overview](#overview)
+* [Index Structure Overview](#index-structure-overview)
-* [File Naming](#file-naming)
+* [File Naming](#file-naming)
-* [Summary of File Extensions](#file-names)
+* [Summary of File Extensions](#summary-of-file-extensions)
- * [Lock File](#lock-file)
+ * [Lock File](#lock-file)
- * [History](#history)
+ * [History](#history)
- * [Limitations](#limitations)
+ * [Limitations](#limitations)
-</div>
## Introduction
-<div>
-
This document defines the index file formats used in this version of Lucene. If you are using a different version of Lucene, please consult the copy of `docs/` that was distributed with the version you are using.
-Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations). If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
+Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations) including this implementation in .NET. If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
As Lucene evolves, this document should evolve. Versions of Lucene in different programming languages should endeavor to agree on file formats, and generate new versions of this document.
-</div>
-
## Definitions
-<div>
-
The fundamental concepts in Lucene are index, document, field and term.
An index contains a sequence of documents.
-* A document is a sequence of fields.
+* A document is a sequence of fields.
-* A field is a named sequence of terms.
+* A field is a named sequence of terms.
-* A term is a sequence of bytes.
+* A term is a sequence of bytes.
The same sequence of bytes in two different fields is considered a different term. Thus terms are represented as a pair: the string naming the field, and the bytes within the field.
@@ -90,7 +82,7 @@ In Lucene, fields may be _stored_, in which case their text is stored in the ind
The text of a field may be _tokenized_ into terms to be indexed, or the text of a field may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for certain identifier fields to be indexed literally.
-See the [Field](xref:Lucene.Net.Documents.Field) java docs for more information on Fields.
+See the [Field](xref:Lucene.Net.Documents.Field) docs for more information on Fields.
### Segments
@@ -108,90 +100,75 @@ Internally, Lucene refers to documents by an integer _document number_. The firs
Note that a document's number may change, so caution should be taken when storing these numbers outside of Lucene. In particular, numbers may change in the following situations:
-*
+* The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the extern [...]
-The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the external [...]
+* When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
-*
-
-When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
-
-</div>
## Index Structure Overview
-<div>
-
Each segment index maintains the following:
-* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
+* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
This contains metadata about a segment, such as the number of documents,
what files it uses,
-* [Field names](xref:Lucene.Net.Codecs.Lucene42.Lucene42FieldInfosFormat).
+* [Field names](xref:Lucene.Net.Codecs.Lucene42.Lucene42FieldInfosFormat).
This contains the set of field names used in the index.
-* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
+* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
This contains, for each document, a list of attribute-value pairs, where the attributes
are field names. These are used to store auxiliary information about the document, such as
its title, url, or an identifier to access a database. The set of stored fields are what is
returned for each hit when searching. This is keyed by document number.
-* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
A dictionary containing all of the terms used in all of the
indexed fields of all of the documents. The dictionary also contains the number
of documents which contain the term, and pointers to the term's frequency and
proximity data.
-* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the numbers of all the
documents that contain that term, and the frequency of the term in that
document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
-* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the positions that the
term occurs in each document. Note that this will not exist if all fields in
all documents omit position data.
-* [Normalization factors](xref:Lucene.Net.Codecs.Lucene42.Lucene42NormsFormat).
+* [Normalization factors](xref:Lucene.Net.Codecs.Lucene42.Lucene42NormsFormat).
For each field in each document, a value is stored
that is multiplied into the score for hits on that field.
-* [Term Vectors](xref:Lucene.Net.Codecs.Lucene42.Lucene42TermVectorsFormat).
+* [Term Vectors](xref:Lucene.Net.Codecs.Lucene42.Lucene42TermVectorsFormat).
For each field in each document, the term vector (sometimes
called document vector) may be stored. A term vector consists of term text and
term frequency. To add Term Vectors to your index see the
[Field](xref:Lucene.Net.Documents.Field) constructors
-* [Per-document values](xref:Lucene.Net.Codecs.Lucene42.Lucene42DocValuesFormat).
+* [Per-document values](xref:Lucene.Net.Codecs.Lucene42.Lucene42DocValuesFormat).
Like stored values, these are also keyed by document
number, but are generally intended to be loaded into main memory for fast
access. Whereas stored values are generally intended for summary results from
searches, per-document values are useful for things like scoring factors.
-* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
+* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
An optional file indicating which documents are deleted.
Details on each of these are provided in their linked pages.
-</div>
-
## File Naming
-<div>
-
All files belonging to a segment have the same name with varying extensions. The extensions correspond to the different file formats described below. When using the Compound File format (default in 1.4 and greater) these files (except for the Segment info file, the Lock file, and Deleted documents file) are collapsed into a single .cfs file (see below for details)
Typically, all segments in an index are stored in a single directory, although this is not required.
-As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is segments_1, then segments_2, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
-
-</div>
+As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is `segments_1`, then `segments_2`, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
## Summary of File Extensions
-<div>
-
The following table summarizes the names and extensions of the files in Lucene:
<table cellspacing="1" cellpadding="4">
@@ -293,13 +270,12 @@ systems that frequently run out of file handles.</td>
<td>Info about what files are deleted</td>
</tr>
</table>
-</div>
## Lock File
The write lock, which is stored in the index directory by default, is named
-"write.lock". If the lock directory is different from the index directory then
-the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix
+`write.lock`. If the lock directory is different from the index directory then
+the write lock will be named `XXXX-write.lock` where XXXX is a unique prefix
derived from the full path to the index directory. When this file is present, a
writer is currently modifying the index (adding or removing documents). This
lock file ensures that only one writer is modifying the index at a time.
@@ -308,68 +284,64 @@ lock file ensures that only one writer is modifying the index at a time.
Compatibility notes are provided in this document, describing how file formats have changed from prior versions:
-* In version 2.1, the file format was changed to allow lock-less commits (ie,
+* In version 2.1, the file format was changed to allow lock-less commits (ie,
no more commit lock). The change is fully backwards compatible: you can open a
pre-2.1 index for searching or adding/deleting of docs. When the new segments
file is saved (committed), it will be written in the new file format (meaning
no specific "upgrade" process is needed). But note that once a commit has
occurred, pre-2.1 Lucene will not be able to read the index.
-* In version 2.3, the file format was changed to allow segments to share a
+* In version 2.3, the file format was changed to allow segments to share a
single set of doc store (vectors & stored fields) files. This allows for
faster indexing in certain cases. The change is fully backwards compatible (in
the same way as the lock-less commits change in 2.1).
-* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
+* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
Java's modified UTF-8. See [
LUCENE-510](http://issues.apache.org/jira/browse/LUCENE-510) for details.
-* In version 2.9, an optional opaque Map<String,String> CommitUserData
+* In version 2.9, an optional opaque `IDictionary<string, string>` CommitUserData
may be passed to IndexWriter's commit methods (and later retrieved), which is
-recorded in the segments_N file. See [
-LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
+recorded in the `segments_N` file. See
+[LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
diagnostics were added to each segment written recording details about why it
was written (due to flush, merge; which OS/JRE was used; etc.). See issue
[LUCENE-1654](http://issues.apache.org/jira/browse/LUCENE-1654) for details.
-* In version 3.0, compressed fields are no longer written to the index (they
+* In version 3.0, compressed fields are no longer written to the index (they
can still be read, but on merge the new segment will write them, uncompressed).
See issue [LUCENE-1960](http://issues.apache.org/jira/browse/LUCENE-1960)
for details.
-* In version 3.1, segments records the code version that created them. See
+* In version 3.1, segments records the code version that created them. See
[LUCENE-2720](http://issues.apache.org/jira/browse/LUCENE-2720) for details.
Additionally segments track explicitly whether or not they have term vectors.
See [LUCENE-2811](http://issues.apache.org/jira/browse/LUCENE-2811)
for details.
-* In version 3.2, numeric fields are written as natively to stored fields
+* In version 3.2, numeric fields are written as natively to stored fields
file, previously they were stored in text format only.
-* In version 3.4, fields can omit position data while still indexing term
+* In version 3.4, fields can omit position data while still indexing term
frequencies.
-* In version 4.0, the format of the inverted index became extensible via
+* In version 4.0, the format of the inverted index became extensible via
the [Codec](xref:Lucene.Net.Codecs.Codec) api. Fast per-document storage
-({@code DocValues}) was introduced. Normalization factors need no longer be a
+(`DocValues`) was introduced. Normalization factors need no longer be a
single byte, they can be any [NumericDocValues](xref:Lucene.Net.Index.NumericDocValues).
Terms need not be unicode strings, they can be any byte sequence. Term offsets
can optionally be indexed into the postings lists. Payloads can be stored in the
term vectors.
-* In version 4.1, the format of the postings list changed to use either
+* In version 4.1, the format of the postings list changed to use either
of FOR compression or variable-byte encoding, depending upon the frequency
of the term. Terms appearing only once were changed to inline directly into
the term dictionary. Stored fields are compressed by default.
-* In version 4.2, term vectors are compressed by default. DocValues has
+* In version 4.2, term vectors are compressed by default. DocValues has
a new multi-valued type (SortedSet), that can be used for faceting/grouping/joining
on multi-valued fields.
## Limitations
-<div>
-
-Lucene uses a Java `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#methods) values which have no limit.
-
-</div>
\ No newline at end of file
+Lucene uses a .NET `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#Lucene_Net_Store_DataOutput_WriteVInt32_System_Int32_) values which have no limit.
\ No newline at end of file
diff --git a/src/Lucene.Net/Codecs/Lucene45/package.md b/src/Lucene.Net/Codecs/Lucene45/package.md
index 3e28189..adc6571 100644
--- a/src/Lucene.Net/Codecs/Lucene45/package.md
+++ b/src/Lucene.Net/Codecs/Lucene45/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Codecs.Lucene45
summary: *content
---
@@ -24,59 +24,51 @@ Lucene 4.5 file format.
# Apache Lucene - Index File Formats
-<div>
-* [Introduction](#introduction)
+* [Introduction](#introduction)
-* [Definitions](#definitions)
+* [Definitions](#definitions)
- * [Inverted Indexing](#inverted-indexing)
+ * [Inverted Indexing](#inverted-indexing)
- * [Types of Fields](#types-of-fields)
+ * [Types of Fields](#types-of-fields)
- * [Segments](#segments)
+ * [Segments](#segments)
- * [Document Numbers](#document-numbers)
+ * [Document Numbers](#document-numbers)
-* [Index Structure Overview](#overview)
+* [Index Structure Overview](#index-structure-overview)
-* [File Naming](#file-naming)
+* [File Naming](#file-naming)
-* [Summary of File Extensions](#file-names)
+* [Summary of File Extensions](#summary-of-file-extensions)
- * [Lock File](#lock-file)
+ * [Lock File](#lock-file)
- * [History](#history)
+ * [History](#history)
- * [Limitations](#limitations)
+ * [Limitations](#limitations)
-</div>
## Introduction
-<div>
-
This document defines the index file formats used in this version of Lucene. If you are using a different version of Lucene, please consult the copy of `docs/` that was distributed with the version you are using.
-Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations). If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
+Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations) including this implementation in .NET. If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
As Lucene evolves, this document should evolve. Versions of Lucene in different programming languages should endeavor to agree on file formats, and generate new versions of this document.
-</div>
-
## Definitions
-<div>
-
The fundamental concepts in Lucene are index, document, field and term.
An index contains a sequence of documents.
-* A document is a sequence of fields.
+* A document is a sequence of fields.
-* A field is a named sequence of terms.
+* A field is a named sequence of terms.
-* A term is a sequence of bytes.
+* A term is a sequence of bytes.
The same sequence of bytes in two different fields is considered a different term. Thus terms are represented as a pair: the string naming the field, and the bytes within the field.
@@ -90,7 +82,7 @@ In Lucene, fields may be _stored_, in which case their text is stored in the ind
The text of a field may be _tokenized_ into terms to be indexed, or the text of a field may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for certain identifier fields to be indexed literally.
-See the [Field](xref:Lucene.Net.Documents.Field) java docs for more information on Fields.
+See the [Field](xref:Lucene.Net.Documents.Field) docs for more information on Fields.
### Segments
@@ -108,90 +100,74 @@ Internally, Lucene refers to documents by an integer _document number_. The firs
Note that a document's number may change, so caution should be taken when storing these numbers outside of Lucene. In particular, numbers may change in the following situations:
-*
-
-The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the external [...]
-
-*
-
-When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
+* The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the extern [...]
-</div>
+* When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
## Index Structure Overview
-<div>
-
Each segment index maintains the following:
-* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
+* [Segment info](xref:Lucene.Net.Codecs.Lucene40.Lucene40SegmentInfoFormat).
This contains metadata about a segment, such as the number of documents,
what files it uses,
-* [Field names](xref:Lucene.Net.Codecs.Lucene42.Lucene42FieldInfosFormat).
+* [Field names](xref:Lucene.Net.Codecs.Lucene42.Lucene42FieldInfosFormat).
This contains the set of field names used in the index.
-* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
+* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
This contains, for each document, a list of attribute-value pairs, where the attributes
are field names. These are used to store auxiliary information about the document, such as
its title, url, or an identifier to access a database. The set of stored fields are what is
returned for each hit when searching. This is keyed by document number.
-* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
A dictionary containing all of the terms used in all of the
indexed fields of all of the documents. The dictionary also contains the number
of documents which contain the term, and pointers to the term's frequency and
proximity data.
-* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the numbers of all the
documents that contain that term, and the frequency of the term in that
document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
-* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the positions that the
term occurs in each document. Note that this will not exist if all fields in
all documents omit position data.
-* [Normalization factors](xref:Lucene.Net.Codecs.Lucene42.Lucene42NormsFormat).
+* [Normalization factors](xref:Lucene.Net.Codecs.Lucene42.Lucene42NormsFormat).
For each field in each document, a value is stored
that is multiplied into the score for hits on that field.
-* [Term Vectors](xref:Lucene.Net.Codecs.Lucene42.Lucene42TermVectorsFormat).
+* [Term Vectors](xref:Lucene.Net.Codecs.Lucene42.Lucene42TermVectorsFormat).
For each field in each document, the term vector (sometimes
called document vector) may be stored. A term vector consists of term text and
term frequency. To add Term Vectors to your index see the
[Field](xref:Lucene.Net.Documents.Field) constructors
-* [Per-document values](xref:Lucene.Net.Codecs.Lucene45.Lucene45DocValuesFormat).
+* [Per-document values](xref:Lucene.Net.Codecs.Lucene45.Lucene45DocValuesFormat).
Like stored values, these are also keyed by document
number, but are generally intended to be loaded into main memory for fast
access. Whereas stored values are generally intended for summary results from
searches, per-document values are useful for things like scoring factors.
-* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
+* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
An optional file indicating which documents are deleted.
Details on each of these are provided in their linked pages.
-</div>
-
## File Naming
-<div>
-
All files belonging to a segment have the same name with varying extensions. The extensions correspond to the different file formats described below. When using the Compound File format (default in 1.4 and greater) these files (except for the Segment info file, the Lock file, and Deleted documents file) are collapsed into a single .cfs file (see below for details)
Typically, all segments in an index are stored in a single directory, although this is not required.
-As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is segments_1, then segments_2, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
-
-</div>
+As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is `segments_1`, then `segments_2`, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
## Summary of File Extensions
-<div>
-
The following table summarizes the names and extensions of the files in Lucene:
<table cellspacing="1" cellpadding="4">
@@ -293,13 +269,12 @@ systems that frequently run out of file handles.</td>
<td>Info about what files are deleted</td>
</tr>
</table>
-</div>
## Lock File
The write lock, which is stored in the index directory by default, is named
-"write.lock". If the lock directory is different from the index directory then
-the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix
+`write.lock`. If the lock directory is different from the index directory then
+the write lock will be named `XXXX-write.lock` where XXXX is a unique prefix
derived from the full path to the index directory. When this file is present, a
writer is currently modifying the index (adding or removing documents). This
lock file ensures that only one writer is modifying the index at a time.
@@ -308,70 +283,66 @@ lock file ensures that only one writer is modifying the index at a time.
Compatibility notes are provided in this document, describing how file formats have changed from prior versions:
-* In version 2.1, the file format was changed to allow lock-less commits (ie,
+* In version 2.1, the file format was changed to allow lock-less commits (ie,
no more commit lock). The change is fully backwards compatible: you can open a
pre-2.1 index for searching or adding/deleting of docs. When the new segments
file is saved (committed), it will be written in the new file format (meaning
no specific "upgrade" process is needed). But note that once a commit has
occurred, pre-2.1 Lucene will not be able to read the index.
-* In version 2.3, the file format was changed to allow segments to share a
+* In version 2.3, the file format was changed to allow segments to share a
single set of doc store (vectors & stored fields) files. This allows for
faster indexing in certain cases. The change is fully backwards compatible (in
the same way as the lock-less commits change in 2.1).
-* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
+* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
Java's modified UTF-8. See [
LUCENE-510](http://issues.apache.org/jira/browse/LUCENE-510) for details.
-* In version 2.9, an optional opaque Map<String,String> CommitUserData
+* In version 2.9, an optional opaque `IDictionary<string, string>` CommitUserData
may be passed to IndexWriter's commit methods (and later retrieved), which is
-recorded in the segments_N file. See [
-LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
+recorded in the `segments_N` file. See
+[LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
diagnostics were added to each segment written recording details about why it
was written (due to flush, merge; which OS/JRE was used; etc.). See issue
[LUCENE-1654](http://issues.apache.org/jira/browse/LUCENE-1654) for details.
-* In version 3.0, compressed fields are no longer written to the index (they
+* In version 3.0, compressed fields are no longer written to the index (they
can still be read, but on merge the new segment will write them, uncompressed).
See issue [LUCENE-1960](http://issues.apache.org/jira/browse/LUCENE-1960)
for details.
-* In version 3.1, segments records the code version that created them. See
+* In version 3.1, segments records the code version that created them. See
[LUCENE-2720](http://issues.apache.org/jira/browse/LUCENE-2720) for details.
Additionally segments track explicitly whether or not they have term vectors.
See [LUCENE-2811](http://issues.apache.org/jira/browse/LUCENE-2811)
for details.
-* In version 3.2, numeric fields are written as natively to stored fields
+* In version 3.2, numeric fields are written as natively to stored fields
file, previously they were stored in text format only.
-* In version 3.4, fields can omit position data while still indexing term
+* In version 3.4, fields can omit position data while still indexing term
frequencies.
-* In version 4.0, the format of the inverted index became extensible via
+* In version 4.0, the format of the inverted index became extensible via
the [Codec](xref:Lucene.Net.Codecs.Codec) api. Fast per-document storage
-({@code DocValues}) was introduced. Normalization factors need no longer be a
+(`DocValues`) was introduced. Normalization factors need no longer be a
single byte, they can be any [NumericDocValues](xref:Lucene.Net.Index.NumericDocValues).
Terms need not be unicode strings, they can be any byte sequence. Term offsets
can optionally be indexed into the postings lists. Payloads can be stored in the
term vectors.
-* In version 4.1, the format of the postings list changed to use either
+* In version 4.1, the format of the postings list changed to use either
of FOR compression or variable-byte encoding, depending upon the frequency
of the term. Terms appearing only once were changed to inline directly into
the term dictionary. Stored fields are compressed by default.
-* In version 4.2, term vectors are compressed by default. DocValues has
+* In version 4.2, term vectors are compressed by default. DocValues has
a new multi-valued type (SortedSet), that can be used for faceting/grouping/joining
on multi-valued fields.
-* In version 4.5, DocValues were extended to explicitly represent missing values.
+* In version 4.5, DocValues were extended to explicitly represent missing values.
## Limitations
-<div>
-
-Lucene uses a Java `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#methods) values which have no limit.
-
-</div>
\ No newline at end of file
+Lucene uses a .NET `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#Lucene_Net_Store_DataOutput_WriteVInt32_System_Int32_) values which have no limit.
\ No newline at end of file
diff --git a/src/Lucene.Net/Codecs/Lucene46/package.md b/src/Lucene.Net/Codecs/Lucene46/package.md
index 937c10f..d2ba9d3 100644
--- a/src/Lucene.Net/Codecs/Lucene46/package.md
+++ b/src/Lucene.Net/Codecs/Lucene46/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Codecs.Lucene46
summary: *content
---
@@ -24,59 +24,51 @@ Lucene 4.6 file format.
# Apache Lucene - Index File Formats
-<div>
-* [Introduction](#introduction)
+* [Introduction](#introduction)
-* [Definitions](#definitions)
+* [Definitions](#definitions)
- * [Inverted Indexing](#inverted-indexing)
+ * [Inverted Indexing](#inverted-indexing)
- * [Types of Fields](#types-of-fields)
+ * [Types of Fields](#types-of-fields)
- * [Segments](#segments)
+ * [Segments](#segments)
- * [Document Numbers](#document-numbers)
+ * [Document Numbers](#document-numbers)
-* [Index Structure Overview](#overview)
+* [Index Structure Overview](#index-structure-overview)
-* [File Naming](#file-naming)
+* [File Naming](#file-naming)
-* [Summary of File Extensions](#file-names)
+* [Summary of File Extensions](#summary-of-file-extensions)
- * [Lock File](#lock-file)
+ * [Lock File](#lock-file)
- * [History](#history)
+ * [History](#history)
- * [Limitations](#limitations)
+ * [Limitations](#limitations)
-</div>
## Introduction
-<div>
-
This document defines the index file formats used in this version of Lucene. If you are using a different version of Lucene, please consult the copy of `docs/` that was distributed with the version you are using.
-Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations). If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
+Apache Lucene is written in Java, but several efforts are underway to write [versions of Lucene in other programming languages](http://wiki.apache.org/lucene-java/LuceneImplementations) including this implementation in .NET. If these versions are to remain compatible with Apache Lucene, then a language-independent definition of the Lucene index format is required. This document thus attempts to provide a complete and independent definition of the Apache Lucene file formats.
As Lucene evolves, this document should evolve. Versions of Lucene in different programming languages should endeavor to agree on file formats, and generate new versions of this document.
-</div>
-
## Definitions
-<div>
-
The fundamental concepts in Lucene are index, document, field and term.
An index contains a sequence of documents.
-* A document is a sequence of fields.
+* A document is a sequence of fields.
-* A field is a named sequence of terms.
+* A field is a named sequence of terms.
-* A term is a sequence of bytes.
+* A term is a sequence of bytes.
The same sequence of bytes in two different fields is considered a different term. Thus terms are represented as a pair: the string naming the field, and the bytes within the field.
@@ -90,7 +82,7 @@ In Lucene, fields may be _stored_, in which case their text is stored in the ind
The text of a field may be _tokenized_ into terms to be indexed, or the text of a field may be used literally as a term to be indexed. Most fields are tokenized, but sometimes it is useful for certain identifier fields to be indexed literally.
-See the [Field](xref:Lucene.Net.Documents.Field) java docs for more information on Fields.
+See the [Field](xref:Lucene.Net.Documents.Field) docs for more information on Fields.
### Segments
@@ -108,90 +100,75 @@ Internally, Lucene refers to documents by an integer _document number_. The firs
Note that a document's number may change, so caution should be taken when storing these numbers outside of Lucene. In particular, numbers may change in the following situations:
-*
+* The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the extern [...]
-The numbers stored in each segment are unique only within the segment, and must be converted before they can be used in a larger context. The standard technique is to allocate each segment a range of values, based on the range of numbers used in that segment. To convert a document number from a segment to an external value, the segment's _base_ document number is added. To convert an external value back to a segment-specific value, the segment is identified by the range that the external [...]
+* When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
-*
-
-When documents are deleted, gaps are created in the numbering. These are eventually removed as the index evolves through merging. Deleted documents are dropped when segments are merged. A freshly-merged segment thus has no gaps in its numbering.
-
-</div>
## Index Structure Overview
-<div>
-
Each segment index maintains the following:
-* [Segment info](xref:Lucene.Net.Codecs.Lucene46.Lucene46SegmentInfoFormat).
+* [Segment info](xref:Lucene.Net.Codecs.Lucene46.Lucene46SegmentInfoFormat).
This contains metadata about a segment, such as the number of documents,
what files it uses,
-* [Field names](xref:Lucene.Net.Codecs.Lucene46.Lucene46FieldInfosFormat).
+* [Field names](xref:Lucene.Net.Codecs.Lucene46.Lucene46FieldInfosFormat).
This contains the set of field names used in the index.
-* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
+* [Stored Field values](xref:Lucene.Net.Codecs.Lucene41.Lucene41StoredFieldsFormat).
This contains, for each document, a list of attribute-value pairs, where the attributes
are field names. These are used to store auxiliary information about the document, such as
its title, url, or an identifier to access a database. The set of stored fields are what is
returned for each hit when searching. This is keyed by document number.
-* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term dictionary](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
A dictionary containing all of the terms used in all of the
indexed fields of all of the documents. The dictionary also contains the number
of documents which contain the term, and pointers to the term's frequency and
proximity data.
-* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Frequency data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the numbers of all the
documents that contain that term, and the frequency of the term in that
document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
-* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
+* [Term Proximity data](xref:Lucene.Net.Codecs.Lucene41.Lucene41PostingsFormat).
For each term in the dictionary, the positions that the
term occurs in each document. Note that this will not exist if all fields in
all documents omit position data.
-* [Normalization factors](xref:Lucene.Net.Codecs.Lucene42.Lucene42NormsFormat).
+* [Normalization factors](xref:Lucene.Net.Codecs.Lucene42.Lucene42NormsFormat).
For each field in each document, a value is stored
that is multiplied into the score for hits on that field.
-* [Term Vectors](xref:Lucene.Net.Codecs.Lucene42.Lucene42TermVectorsFormat).
+* [Term Vectors](xref:Lucene.Net.Codecs.Lucene42.Lucene42TermVectorsFormat).
For each field in each document, the term vector (sometimes
called document vector) may be stored. A term vector consists of term text and
term frequency. To add Term Vectors to your index see the
[Field](xref:Lucene.Net.Documents.Field) constructors
-* [Per-document values](xref:Lucene.Net.Codecs.Lucene42.Lucene42DocValuesFormat).
+* [Per-document values](xref:Lucene.Net.Codecs.Lucene42.Lucene42DocValuesFormat).
Like stored values, these are also keyed by document
number, but are generally intended to be loaded into main memory for fast
access. Whereas stored values are generally intended for summary results from
searches, per-document values are useful for things like scoring factors.
-* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
+* [Deleted documents](xref:Lucene.Net.Codecs.Lucene40.Lucene40LiveDocsFormat).
An optional file indicating which documents are deleted.
Details on each of these are provided in their linked pages.
-</div>
-
## File Naming
-<div>
-
All files belonging to a segment have the same name with varying extensions. The extensions correspond to the different file formats described below. When using the Compound File format (default in 1.4 and greater) these files (except for the Segment info file, the Lock file, and Deleted documents file) are collapsed into a single .cfs file (see below for details)
Typically, all segments in an index are stored in a single directory, although this is not required.
-As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is segments_1, then segments_2, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
-
-</div>
+As of version 2.1 (lock-less commits), file names are never re-used (there is one exception, "segments.gen", see below). That is, when any file is saved to the Directory it is given a never before used filename. This is achieved using a simple generations approach. For example, the first segments file is `segments_1`, then `segments_2`, etc. The generation is a sequential long integer represented in alpha-numeric (base 36) form.
## Summary of File Extensions
-<div>
-
The following table summarizes the names and extensions of the files in Lucene:
<table cellspacing="1" cellpadding="4">
@@ -293,13 +270,12 @@ systems that frequently run out of file handles.</td>
<td>Info about what files are deleted</td>
</tr>
</table>
-</div>
## Lock File
The write lock, which is stored in the index directory by default, is named
-"write.lock". If the lock directory is different from the index directory then
-the write lock will be named "XXXX-write.lock" where XXXX is a unique prefix
+`write.lock`. If the lock directory is different from the index directory then
+the write lock will be named `XXXX-write.lock` where XXXX is a unique prefix
derived from the full path to the index directory. When this file is present, a
writer is currently modifying the index (adding or removing documents). This
lock file ensures that only one writer is modifying the index at a time.
@@ -308,77 +284,73 @@ lock file ensures that only one writer is modifying the index at a time.
Compatibility notes are provided in this document, describing how file formats have changed from prior versions:
-* In version 2.1, the file format was changed to allow lock-less commits (ie,
+* In version 2.1, the file format was changed to allow lock-less commits (ie,
no more commit lock). The change is fully backwards compatible: you can open a
pre-2.1 index for searching or adding/deleting of docs. When the new segments
file is saved (committed), it will be written in the new file format (meaning
no specific "upgrade" process is needed). But note that once a commit has
occurred, pre-2.1 Lucene will not be able to read the index.
-* In version 2.3, the file format was changed to allow segments to share a
+* In version 2.3, the file format was changed to allow segments to share a
single set of doc store (vectors & stored fields) files. This allows for
faster indexing in certain cases. The change is fully backwards compatible (in
the same way as the lock-less commits change in 2.1).
-* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
+* In version 2.4, Strings are now written as true UTF-8 byte sequence, not
Java's modified UTF-8. See [
LUCENE-510](http://issues.apache.org/jira/browse/LUCENE-510) for details.
-* In version 2.9, an optional opaque Map<String,String> CommitUserData
+* In version 2.9, an optional opaque `IDictionary<string, string>` CommitUserData
may be passed to IndexWriter's commit methods (and later retrieved), which is
-recorded in the segments_N file. See [
-LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
+recorded in the `segments_N` file. See
+[LUCENE-1382](http://issues.apache.org/jira/browse/LUCENE-1382) for details. Also,
diagnostics were added to each segment written recording details about why it
was written (due to flush, merge; which OS/JRE was used; etc.). See issue
[LUCENE-1654](http://issues.apache.org/jira/browse/LUCENE-1654) for details.
-* In version 3.0, compressed fields are no longer written to the index (they
+* In version 3.0, compressed fields are no longer written to the index (they
can still be read, but on merge the new segment will write them, uncompressed).
See issue [LUCENE-1960](http://issues.apache.org/jira/browse/LUCENE-1960)
for details.
-* In version 3.1, segments records the code version that created them. See
+* In version 3.1, segments records the code version that created them. See
[LUCENE-2720](http://issues.apache.org/jira/browse/LUCENE-2720) for details.
Additionally segments track explicitly whether or not they have term vectors.
See [LUCENE-2811](http://issues.apache.org/jira/browse/LUCENE-2811)
for details.
-* In version 3.2, numeric fields are written as natively to stored fields
+* In version 3.2, numeric fields are written as natively to stored fields
file, previously they were stored in text format only.
-* In version 3.4, fields can omit position data while still indexing term
+* In version 3.4, fields can omit position data while still indexing term
frequencies.
-* In version 4.0, the format of the inverted index became extensible via
+* In version 4.0, the format of the inverted index became extensible via
the [Codec](xref:Lucene.Net.Codecs.Codec) api. Fast per-document storage
-({@code DocValues}) was introduced. Normalization factors need no longer be a
+(`DocValues`) was introduced. Normalization factors need no longer be a
single byte, they can be any [NumericDocValues](xref:Lucene.Net.Index.NumericDocValues).
Terms need not be unicode strings, they can be any byte sequence. Term offsets
can optionally be indexed into the postings lists. Payloads can be stored in the
term vectors.
-* In version 4.1, the format of the postings list changed to use either
+* In version 4.1, the format of the postings list changed to use either
of FOR compression or variable-byte encoding, depending upon the frequency
of the term. Terms appearing only once were changed to inline directly into
the term dictionary. Stored fields are compressed by default.
-* In version 4.2, term vectors are compressed by default. DocValues has
+* In version 4.2, term vectors are compressed by default. DocValues has
a new multi-valued type (SortedSet), that can be used for faceting/grouping/joining
on multi-valued fields.
-* In version 4.5, DocValues were extended to explicitly represent missing values.
+* In version 4.5, DocValues were extended to explicitly represent missing values.
-* In version 4.6, FieldInfos were extended to support per-field DocValues generation, to
+* In version 4.6, FieldInfos were extended to support per-field DocValues generation, to
allow updating NumericDocValues fields.
-* In version 4.8, checksum footers were added to the end of each index file
+* In version 4.8, checksum footers were added to the end of each index file
for improved data integrity. Specifically, the last 8 bytes of every index file
contain the zlib-crc32 checksum of the file.
## Limitations
-<div>
-
-Lucene uses a Java `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#methods) values which have no limit.
-
-</div>
\ No newline at end of file
+Lucene uses a .NET `int` to refer to document numbers, and the index file format uses an `Int32` on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either `UInt64` values, or better yet, [VInt](xref:Lucene.Net.Store.DataOutput#Lucene_Net_Store_DataOutput_WriteVInt32_System_Int32_) values which have no limit.
\ No newline at end of file
diff --git a/src/Lucene.Net/Search/Payloads/package.md b/src/Lucene.Net/Search/Payloads/package.md
index e8afdff..70bf0fe 100644
--- a/src/Lucene.Net/Search/Payloads/package.md
+++ b/src/Lucene.Net/Search/Payloads/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Search.Payloads
summary: *content
---
@@ -22,4 +22,7 @@ summary: *content
The payloads package provides Query mechanisms for finding and using payloads.
- The following Query implementations are provided: 1. [PayloadTermQuery](xref:Lucene.Net.Search.Payloads.PayloadTermQuery) -- Boost a term's score based on the value of the payload located at that term. 2. [PayloadNearQuery](xref:Lucene.Net.Search.Payloads.PayloadNearQuery) -- A [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) that factors in the value of the payloads located at each of the positions where the spans occur.
\ No newline at end of file
+The following Query implementations are provided:
+
+1. [PayloadTermQuery](xref:Lucene.Net.Search.Payloads.PayloadTermQuery) -- Boost a term's score based on the value of the payload located at that term.
+2. [PayloadNearQuery](xref:Lucene.Net.Search.Payloads.PayloadNearQuery) -- A [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) that factors in the value of the payloads located at each of the positions where the spans occur.
\ No newline at end of file
diff --git a/src/Lucene.Net/Search/Similarities/package.md b/src/Lucene.Net/Search/Similarities/package.md
index d441963..34bb4ef 100644
--- a/src/Lucene.Net/Search/Similarities/package.md
+++ b/src/Lucene.Net/Search/Similarities/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Search.Similarities
summary: *content
---
@@ -28,7 +28,8 @@ package.
## Table Of Contents
- 1. [Summary of the Ranking Methods](#sims) 2. [Changing the Similarity](#changingsimilarity)
+1. [Summary of the Ranking Methods](#summary-of-the-ranking-methods)
+2. [Changing the Similarity](#changing-similarity)
## Summary of the Ranking Methods
@@ -36,24 +37,38 @@ package.
<xref:Lucene.Net.Search.Similarities.BM25Similarity> is an optimized implementation of the successful Okapi BM25 model.
-<xref:Lucene.Net.Search.Similarities.SimilarityBase> provides a basic implementation of the Similarity contract and exposes a highly simplified interface, which makes it an ideal starting point for new ranking functions. Lucene ships the following methods built on <xref:Lucene.Net.Search.Similarities.SimilarityBase>: * Amati and Rijsbergen's {@linkplain org.apache.lucene.search.similarities.DFRSimilarity DFR} framework; * Clinchant and Gaussier's {@linkplain org.apache.lucene.search.simi [...]
+<xref:Lucene.Net.Search.Similarities.SimilarityBase> provides a basic implementation of the Similarity contract and exposes a highly simplified interface, which makes it an ideal starting point for new ranking functions. Lucene ships the following methods built on <xref:Lucene.Net.Search.Similarities.SimilarityBase>:
+
+<a id="framework"></a>
+
+* Amati and Rijsbergen's [DFR](xref:Lucene.Net.Search.Similarities.DFRSimilarity) framework;
+* Clinchant and Gaussier's [Information-based models](xref:Lucene.Net.Search.Similarities.IBSimilarity) for IR;
+* The implementation of two [language models](xref:Lucene.Net.Search.Similarities.LMSimilarity) from Zhai and Lafferty's paper.
+
+Since <xref:Lucene.Net.Search.Similarities.SimilarityBase> is not optimized to the same extent as <xref:Lucene.Net.Search.Similarities.DefaultSimilarity> and <xref:Lucene.Net.Search.Similarities.BM25Similarity>, a difference in performance is to be expected when using the methods listed above. However, optimizations can always be implemented in subclasses; see [below](#changing-similarity).
## Changing Similarity
-Chances are the available Similarities are sufficient for all your searching needs. However, in some applications it may be necessary to customize your [Similarity](Similarity.html) implementation. For instance, some applications do not need to distinguish between shorter and longer documents (see [a "fair" similarity](http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967)).
+Chances are the available Similarities are sufficient for all your searching needs. However, in some applications it may be necessary to customize your <xref:Lucene.Net.Search.Similarities.Similarity> implementation. For instance, some applications do not need to distinguish between shorter and longer documents (see [a "fair" similarity](http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967)).
To change <xref:Lucene.Net.Search.Similarities.Similarity>, one must do so for both indexing and searching, and the changes must happen before either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it just isn't well-defined what is going to happen.
-To make this change, implement your own <xref:Lucene.Net.Search.Similarities.Similarity> (likely you'll want to simply subclass an existing method, be it <xref:Lucene.Net.Search.Similarities.DefaultSimilarity> or a descendant of <xref:Lucene.Net.Search.Similarities.SimilarityBase>), and then register the new class by calling [#setSimilarity(Similarity)](xref:Lucene.Net.Index.IndexWriterConfig) before indexing and [#setSimilarity(Similarity)](xref:Lucene.Net.Search.IndexSearcher) before s [...]
+To make this change, implement your own <xref:Lucene.Net.Search.Similarities.Similarity> (likely you'll want to simply subclass an existing method, be it <xref:Lucene.Net.Search.Similarities.DefaultSimilarity> or a descendant of <xref:Lucene.Net.Search.Similarities.SimilarityBase>), and then register the new class by setting [IndexWriterConfig.Similarity](xref:Lucene.Net.Index.IndexWriterConfig#Lucene_Net_Index_IndexWriterConfig_Similarity) before indexing and [IndexSearcher.Similarity]( [...]
-### Extending {@linkplain org.apache.lucene.search.similarities.SimilarityBase}
+### Extending [SimilarityBase](xref:Lucene.Net.Search.Similarities.SimilarityBase)
- The easiest way to quickly implement a new ranking method is to extend <xref:Lucene.Net.Search.Similarities.SimilarityBase>, which provides basic implementations for the low level . Subclasses are only required to implement the [Float)](xref:Lucene.Net.Search.Similarities.SimilarityBase#methods) and [#toString()](xref:Lucene.Net.Search.Similarities.SimilarityBase) methods.
+The easiest way to quickly implement a new ranking method is to extend <xref:Lucene.Net.Search.Similarities.SimilarityBase>, which provides basic implementations for the low level . Subclasses are only required to implement the [Float)](xref:Lucene.Net.Search.Similarities.SimilarityBase#methods) and [#toString()](xref:Lucene.Net.Search.Similarities.SimilarityBase) methods.
Another option is to extend one of the [frameworks](#framework) based on <xref:Lucene.Net.Search.Similarities.SimilarityBase>. These Similarities are implemented modularly, e.g. <xref:Lucene.Net.Search.Similarities.DFRSimilarity> delegates computation of the three parts of its formula to the classes <xref:Lucene.Net.Search.Similarities.BasicModel>, <xref:Lucene.Net.Search.Similarities.AfterEffect> and <xref:Lucene.Net.Search.Similarities.Normalization>. Instead of subclassing the Similar [...]
-### Changing {@linkplain org.apache.lucene.search.similarities.DefaultSimilarity}
+### Changing [DefaultSimilarity](xref:Lucene.Net.Search.Similarities.DefaultSimilarity)
+
+If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at [Overriding Similarity](http://www.gossamer-threads.com/lists/lucene/java-user/39125). In summary, here are a few use cases:
+
+1. The [SweetSpotSimilarity](../../misc/Lucene.Net.Misc.SweetSpotSimilarity.html) gives small increases as the frequency increases a small amount and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is more significant.
+2. Overriding Tf — In some applications, it doesn't matter what the score of a document is as long as a matching term occurs. In these cases people have overridden Similarity to return 1 from the Tf() method.
+3. Changing Length Normalization — By overriding [State)](xref:Lucene.Net.Search.Similarities.Similarity#methods), it is possible to discount how the length of a field contributes to a score. In <xref:Lucene.Net.Search.Similarities.DefaultSimilarity>, lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be 1 / (numTerms in field), all fields will be treated ["fairly"](http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967).
- If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at [Overriding Similarity](http://www.gossamer-threads.com/lists/lucene/java-user/39125). In summary, here are a few use cases: 1. <p>The `SweetSpotSimilarity` in `org.apache.lucene.misc` gives small increases as the frequency increases a small amount and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is more significant.</p> 2. <p>Overr [...]
+In general, Chris Hostetter sums it up best in saying (from [the Lucene users's mailing list](http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125)):
> [One would override the Similarity in] ... any situation where you know more about your data then just that it's "text" is a situation where it *might* make sense to to override your Similarity method.
\ No newline at end of file
diff --git a/src/Lucene.Net/Search/Spans/package.md b/src/Lucene.Net/Search/Spans/package.md
index e9eb946..c24945e 100644
--- a/src/Lucene.Net/Search/Spans/package.md
+++ b/src/Lucene.Net/Search/Spans/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Search.Spans
summary: *content
---
@@ -24,33 +24,46 @@ The calculus of spans.
A span is a `<doc,startPosition,endPosition>` tuple.
-The following span query operators are implemented: * A [SpanTermQuery](xref:Lucene.Net.Search.Spans.SpanTermQuery) matches all spans containing a particular [Term](xref:Lucene.Net.Index.Term). * A [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) matches spans which occur near one another, and can be used to implement things like phrase search (when constructed from <xref:Lucene.Net.Search.Spans.SpanTermQuery>s) and inter-phrase proximity (when constructed from other <xref:Luc [...]
+The following span query operators are implemented:
+
+* A [SpanTermQuery](xref:Lucene.Net.Search.Spans.SpanTermQuery) matches all spans containing a particular [Term](xref:Lucene.Net.Index.Term).
+* A [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) matches spans which occur near one another, and can be used to implement things like phrase search (when constructed from <xref:Lucene.Net.Search.Spans.SpanTermQuery>s) and inter-phrase proximity (when constructed from other <xref:Lucene.Net.Search.Spans.SpanNearQuery>s).
+* A [SpanOrQuery](xref:Lucene.Net.Search.Spans.SpanOrQuery) merges spans from a number of other <xref:Lucene.Net.Search.Spans.SpanQuery>s.
+* A [SpanNotQuery](xref:Lucene.Net.Search.Spans.SpanNotQuery) removes spans matching one [SpanQuery](xref:Lucene.Net.Search.Spans.SpanQuery) which overlap (or comes near) another. This can be used, e.g., to implement within-paragraph search.
+* A [SpanFirstQuery](xref:Lucene.Net.Search.Spans.SpanFirstQuery) matches spans matching `q` whose end position is less than `n`. This can be used to constrain matches to the first part of the document.
+* A [SpanPositionRangeQuery](xref:Lucene.Net.Search.Spans.SpanPositionRangeQuery) is a more general form of SpanFirstQuery that can constrain matches to arbitrary portions of the document. In all cases, output spans are minimally inclusive. In other words, a span formed by matching a span in x and y starts at the lesser of the two starts and ends at the greater of the two ends.
For example, a span query which matches "John Kerry" within ten
words of "George Bush" within the first 100 words of the document
could be constructed with:
- SpanQuery john = new SpanTermQuery(new Term("content", "john"));
- SpanQuery kerry = new SpanTermQuery(new Term("content", "kerry"));
- SpanQuery george = new SpanTermQuery(new Term("content", "george"));
- SpanQuery bush = new SpanTermQuery(new Term("content", "bush"));
+```cs
+SpanQuery john = new SpanTermQuery(new Term("content", "john"));
+SpanQuery kerry = new SpanTermQuery(new Term("content", "kerry"));
+SpanQuery george = new SpanTermQuery(new Term("content", "george"));
+SpanQuery bush = new SpanTermQuery(new Term("content", "bush"));
SpanQuery johnKerry =
- new SpanNearQuery(new SpanQuery[] {john, kerry}, 0, true);
+ new SpanNearQuery(new SpanQuery[] { john, kerry }, 0, true);
SpanQuery georgeBush =
- new SpanNearQuery(new SpanQuery[] {george, bush}, 0, true);
+ new SpanNearQuery(new SpanQuery[] { george, bush }, 0, true);
SpanQuery johnKerryNearGeorgeBush =
- new SpanNearQuery(new SpanQuery[] {johnKerry, georgeBush}, 10, false);
+ new SpanNearQuery(new SpanQuery[] { johnKerry, georgeBush }, 10, false);
SpanQuery johnKerryNearGeorgeBushAtStart =
- new SpanFirstQuery(johnKerryNearGeorgeBush, 100);
+ new SpanFirstQuery(johnKerryNearGeorgeBush, 100);
+```
Span queries may be freely intermixed with other Lucene queries.
So, for example, the above query can be restricted to documents which
also use the word "iraq" with:
- Query query = new BooleanQuery();
- query.add(johnKerryNearGeorgeBushAtStart, true, false);
- query.add(new TermQuery("content", "iraq"), true, false);
\ No newline at end of file
+```cs
+Query query = new BooleanQuery
+{
+ johnKerryNearGeorgeBushAtStart,
+ new TermQuery("content", "iraq")
+};
+```
\ No newline at end of file
diff --git a/src/Lucene.Net/Search/package.md b/src/Lucene.Net/Search/package.md
index 32ea227..c5f3b9d 100644
--- a/src/Lucene.Net/Search/package.md
+++ b/src/Lucene.Net/Search/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Search
summary: *content
---
@@ -24,99 +24,111 @@ Code to search indices.
## Table Of Contents
- 1. [Search Basics](#search) 2. [The Query Classes](#query) 3. [Scoring: Introduction](#scoring) 4. [Scoring: Basics](#scoringBasics) 5. [Changing the Scoring](#changingscoring) 6. [Appendix: Search Algorithm](#algorithm)
+ 1. [Search Basics](#search-basics)
+ 2. [The Query Classes](#query-classes)
+ 3. [Scoring: Introduction](#scoring--introduction)
+ 4. [Scoring: Basics](#scoring--basics)
+ 5. [Changing the Scoring](#changing-scoring--similarity)
+ 6. [Appendix: Search Algorithm](#appendix-search-algorithm)
## Search Basics
- Lucene offers a wide variety of <xref:Lucene.Net.Search.Query> implementations, most of which are in this package, its subpackages ([spans](xref:Lucene.Net.Search.Spans), [payloads](xref:Lucene.Net.Search.Payloads)), or the [queries module]({@docRoot}/../queries/overview-summary.html). These implementations can be combined in a wide variety of ways to provide complex querying capabilities along with information about where matches took place in the document collection. The [Query Classe [...]
-
- To perform a search, applications usually call [#search(Query,int)](xref:Lucene.Net.Search.IndexSearcher) or [#search(Query,Filter,int)](xref:Lucene.Net.Search.IndexSearcher).
+ Lucene offers a wide variety of <xref:Lucene.Net.Search.Query> implementations, most of which are in this package, its subpackages ([Lucene.Net.Spans](xref:Lucene.Net.Search.Spans), [Lucene.Net.Payloads](xref:Lucene.Net.Search.Payloads)), or the [Lucene.Net.Queries module](../queries/Lucene.Net.Queries.html). These implementations can be combined in a wide variety of ways to provide complex querying capabilities along with information about where matches took place in the document colle [...]
- Once a Query has been created and submitted to the [IndexSearcher](xref:Lucene.Net.Search.IndexSearcher), the scoring process begins. After some infrastructure setup, control finally passes to the [Weight](xref:Lucene.Net.Search.Weight) implementation and its [Scorer](xref:Lucene.Net.Search.Scorer) or [BulkScore](xref:Lucene.Net.Search.BulkScorer) instances. See the [Algorithm](#algorithm) section for more notes on the process.
+ To perform a search, applications usually call [Search(Query, int)](xref:Lucene.Net.Search.IndexSearcher#Lucene_Net_Search_IndexSearcher_Search_Lucene_Net_Search_Query_System_Int32_) or [Search(Query, Filter, int)](xref:Lucene.Net.Search.IndexSearcher#Lucene_Net_Search_IndexSearcher_Search_Lucene_Net_Search_Query_Lucene_Net_Search_Filter_System_Int32_).
- <!-- TODO: this page over-links the same things too many times -->
+ Once a Query has been created and submitted to the [IndexSearcher](xref:Lucene.Net.Search.IndexSearcher), the scoring process begins. After some infrastructure setup, control finally passes to the [Weight](xref:Lucene.Net.Search.Weight) implementation and its [Scorer](xref:Lucene.Net.Search.Scorer) or [BulkScorer](xref:Lucene.Net.Search.BulkScorer) instances. See the [Algorithm](#appendix-search-algorithm) section for more notes on the process.
## Query Classes
-####
- [TermQuery](xref:Lucene.Net.Search.TermQuery)
+#### [TermQuery](xref:Lucene.Net.Search.TermQuery)
-Of the various implementations of [Query](xref:Lucene.Net.Search.Query), the [TermQuery](xref:Lucene.Net.Search.TermQuery) is the easiest to understand and the most often used in applications. A [TermQuery](xref:Lucene.Net.Search.TermQuery) matches all the documents that contain the specified [Term](xref:Lucene.Net.Index.Term), which is a word that occurs in a certain [Field](xref:Lucene.Net.Documents.Field). Thus, a [TermQuery](xref:Lucene.Net.Search.TermQuery) identifies and scores all [...]
+Of the various implementations of [Query](xref:Lucene.Net.Search.Query), the [TermQuery](xref:Lucene.Net.Search.TermQuery) is the easiest to understand and the most often used in applications. A [TermQuery](xref:Lucene.Net.Search.TermQuery) matches all the documents that contain the specified [Term](xref:Lucene.Net.Index.Term), which is a word that occurs in a certain [Field](xref:Lucene.Net.Documents.Field). Thus, a [TermQuery](xref:Lucene.Net.Search.TermQuery) identifies and scores all [...]
-####
- [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery)
+```cs
+TermQuery tq = new TermQuery(new Term("fieldName", "term"));
+```
-Things start to get interesting when one combines multiple [TermQuery](xref:Lucene.Net.Search.TermQuery) instances into a [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery). A [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery) contains multiple [BooleanClause](xref:Lucene.Net.Search.BooleanClause)s, where each clause contains a sub-query ([Query](xref:Lucene.Net.Search.Query) instance) and an operator (from [BooleanClause.Occur](xref:Lucene.Net.Search.BooleanClause.Occur)) describing h [...]
+In this example, the [Query](xref:Lucene.Net.Search.Query) identifies all [Document](xref:Lucene.Net.Documents.Document)s that have the [Field](xref:Lucene.Net.Documents.Field) named `"fieldName"` containing the word `"term"`.
-#### Phrases
+#### [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery)
+
+Things start to get interesting when one combines multiple [TermQuery](xref:Lucene.Net.Search.TermQuery) instances into a [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery). A [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery) contains multiple [BooleanClause](xref:Lucene.Net.Search.BooleanClause)s, where each clause contains a sub-query ([Query](xref:Lucene.Net.Search.Query) instance) and an operator (from [BooleanClause.Occur](xref:Lucene.Net.Search.Occur)) describing how that sub-qu [...]
-Another common search is to find documents containing certain phrases. This
- is handled three different ways:
+1. <p>[Occur.SHOULD](xref:Lucene.Net.Search.Occur#Lucene_Net_Search_Occur_SHOULD) — Use this operator when a clause can occur in the result set, but is not required. If a query is made up of all SHOULD clauses, then every document in the result set matches at least one of these clauses.</p>
+2. <p>[Occur.MUST](xref:Lucene.Net.Search.Occur#Lucene_Net_Search_Occur_MUST) — Use this operator when a clause is required to occur in the result set. Every document in the result set will match all such clauses.</p>
+3. <p>[Occur.MUST_NOT](xref:Lucene.Net.Search.Occur#Lucene_Net_Search_Occur_MUST_NOT) — Use this operator when a clause must not occur in the result set. No document in the result set will match any such clauses.</p>
-1.
+Boolean queries are constructed by adding two or more [BooleanClause](xref:Lucene.Net.Search.BooleanClause) instances. If too many clauses are added, a [TooManyClauses](xref:Lucene.Net.Search.BooleanQuery.TooManyClauses) exception will be thrown during searching. This most often occurs when a [Query](xref:Lucene.Net.Search.Query) is rewritten into a [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery) with many [TermQuery](xref:Lucene.Net.Search.TermQuery) clauses, for example by [Wildcar [...]
-[PhraseQuery](xref:Lucene.Net.Search.PhraseQuery) — Matches a sequence of [Term](xref:Lucene.Net.Index.Term)s. [PhraseQuery](xref:Lucene.Net.Search.PhraseQuery) uses a slop factor to determine how many positions may occur between any two terms in the phrase and still be considered a match. The slop is 0 by default, meaning the phrase must match exactly.
+#### Phrases
+
+Another common search is to find documents containing certain phrases. This is handled three different ways:
-2.
+1. [PhraseQuery](xref:Lucene.Net.Search.PhraseQuery) — Matches a sequence of [Term](xref:Lucene.Net.Index.Term)s. [PhraseQuery](xref:Lucene.Net.Search.PhraseQuery) uses a slop factor to determine how many positions may occur between any two terms in the phrase and still be considered a match. The slop is 0 by default, meaning the phrase must match exactly.
-[MultiPhraseQuery](xref:Lucene.Net.Search.MultiPhraseQuery) — A more general form of PhraseQuery that accepts multiple Terms for a position in the phrase. For example, this can be used to perform phrase queries that also incorporate synonyms. 3. <p>[SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) — Matches a sequence of other [SpanQuery](xref:Lucene.Net.Search.Spans.SpanQuery) instances. [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) allows for much more complicat [...]
+2. [MultiPhraseQuery](xref:Lucene.Net.Search.MultiPhraseQuery) — A more general form of PhraseQuery that accepts multiple Terms for a position in the phrase. For example, this can be used to perform phrase queries that also incorporate synonyms.
-####
- [TermRangeQuery](xref:Lucene.Net.Search.TermRangeQuery)
+3. [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) — Matches a sequence of other [SpanQuery](xref:Lucene.Net.Search.Spans.SpanQuery) instances. [SpanNearQuery](xref:Lucene.Net.Search.Spans.SpanNearQuery) allows for much more complicated phrase queries since it is constructed from other [SpanQuery](xref:Lucene.Net.Search.Spans.SpanQuery) instances, instead of only [TermQuery](xref:Lucene.Net.Search.TermQuery) instances.
-The [TermRangeQuery](xref:Lucene.Net.Search.TermRangeQuery) matches all documents that occur in the exclusive range of a lower [Term](xref:Lucene.Net.Index.Term) and an upper [Term](xref:Lucene.Net.Index.Term) according to [TermsEnum.getComparator](xref:Lucene.Net.Index.TermsEnum#methods). It is not intended for numerical ranges; use [NumericRangeQuery](xref:Lucene.Net.Search.NumericRangeQuery) instead. For example, one could find all documents that have terms beginning with the letters [...]
+#### [TermRangeQuery](xref:Lucene.Net.Search.TermRangeQuery)
-####
- [NumericRangeQuery](xref:Lucene.Net.Search.NumericRangeQuery)
+The [TermRangeQuery](xref:Lucene.Net.Search.TermRangeQuery) matches all documents that occur in the exclusive range of a lower [Term](xref:Lucene.Net.Index.Term) and an upper [Term](xref:Lucene.Net.Index.Term) according to [TermsEnum.Comparer](xref:Lucene.Net.Index.TermsEnum#Lucene_Net_Index_TermsEnum_Comparer). It is not intended for numerical ranges; use [NumericRangeQuery](xref:Lucene.Net.Search.NumericRangeQuery) instead. For example, one could find all documents that have terms begi [...]
-The [NumericRangeQuery](xref:Lucene.Net.Search.NumericRangeQuery) matches all documents that occur in a numeric range. For NumericRangeQuery to work, you must index the values using a one of the numeric fields ([IntField](xref:Lucene.Net.Documents.IntField), [LongField](xref:Lucene.Net.Documents.LongField), [FloatField](xref:Lucene.Net.Documents.FloatField), or [DoubleField](xref:Lucene.Net.Documents.DoubleField)).
+#### [NumericRangeQuery](xref:Lucene.Net.Search.NumericRangeQuery)
-####
- [PrefixQuery](xref:Lucene.Net.Search.PrefixQuery),
- [WildcardQuery](xref:Lucene.Net.Search.WildcardQuery),
- [RegexpQuery](xref:Lucene.Net.Search.RegexpQuery)
+The [NumericRangeQuery](xref:Lucene.Net.Search.NumericRangeQuery) matches all documents that occur in a numeric range. For NumericRangeQuery to work, you must index the values using a one of the numeric fields ([Int32Field](xref:Lucene.Net.Documents.Int32Field), [Int64Field](xref:Lucene.Net.Documents.Int64Field), [SingleField](xref:Lucene.Net.Documents.SingleField), or [DoubleField](xref:Lucene.Net.Documents.DoubleField)).
-While the [PrefixQuery](xref:Lucene.Net.Search.PrefixQuery) has a different implementation, it is essentially a special case of the [WildcardQuery](xref:Lucene.Net.Search.WildcardQuery). The [PrefixQuery](xref:Lucene.Net.Search.PrefixQuery) allows an application to identify all documents with terms that begin with a certain string. The [WildcardQuery](xref:Lucene.Net.Search.WildcardQuery) generalizes this by allowing for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> [...]
+#### [PrefixQuery](xref:Lucene.Net.Search.PrefixQuery), [WildcardQuery](xref:Lucene.Net.Search.WildcardQuery), [RegexpQuery](xref:Lucene.Net.Search.RegexpQuery)
-####
- [FuzzyQuery](xref:Lucene.Net.Search.FuzzyQuery)
+While the [PrefixQuery](xref:Lucene.Net.Search.PrefixQuery) has a different implementation, it is essentially a special case of the [WildcardQuery](xref:Lucene.Net.Search.WildcardQuery). The [PrefixQuery](xref:Lucene.Net.Search.PrefixQuery) allows an application to identify all documents with terms that begin with a certain string. The [WildcardQuery](xref:Lucene.Net.Search.WildcardQuery) generalizes this by allowing for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> [...]
+
+#### [FuzzyQuery](xref:Lucene.Net.Search.FuzzyQuery)
A [FuzzyQuery](xref:Lucene.Net.Search.FuzzyQuery) matches documents that contain terms similar to the specified term. Similarity is determined using [Levenshtein (edit) distance](http://en.wikipedia.org/wiki/Levenshtein). This type of query can be useful when accounting for spelling variations in the collection.
## Scoring — Introduction
-Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on [java-user@lucene.apache.org](mailto:java-user@lucene.apache.org) to figure out why a document with five of our query terms scores lower than a different document with only on [...]
+Lucene scoring is the heart of why we all love Lucene. It is blazingly fast and it hides almost all of the complexity from the user. In a nutshell, it works. At least, that is, until it doesn't work, or doesn't work as one would expect it to work. Then we are left digging into Lucene internals or asking for help on [user@lucenenet.apache.org](mailto:user@lucenenet.apache.org) to figure out why a document with five of our query terms scores lower than a different document with only one of [...]
While this document won't answer your specific scoring issues, it will, hopefully, point you to the places that can help you figure out the _what_ and _why_ of Lucene scoring.
-Lucene scoring supports a number of pluggable information retrieval [models](http://en.wikipedia.org/wiki/Information_retrieval#Model_types), including: * [Vector Space Model (VSM)](http://en.wikipedia.org/wiki/Vector_Space_Model) * [Probablistic Models](http://en.wikipedia.org/wiki/Probabilistic_relevance_model) such as [Okapi BM25](http://en.wikipedia.org/wiki/Probabilistic_relevance_model_(BM25)) and [DFR](http://en.wikipedia.org/wiki/Divergence-from-randomness_model) * [Language mode [...]
+Lucene scoring supports a number of pluggable information retrieval [models](http://en.wikipedia.org/wiki/Information_retrieval#Model_types), including:
+
+* [Vector Space Model (VSM)](http://en.wikipedia.org/wiki/Vector_Space_Model)
+* [Probablistic Models](http://en.wikipedia.org/wiki/Probabilistic_relevance_model) such as [Okapi BM25](http://en.wikipedia.org/wiki/Probabilistic_relevance_model_(BM25)) and [DFR](http://en.wikipedia.org/wiki/Divergence-from-randomness_model)
+* [Language models](http://en.wikipedia.org/wiki/Language_model)
+
+These models can be plugged in via the [Similarity API](xref:Lucene.Net.Search.Similarities), and offer extension hooks and parameters for tuning. In general, Lucene first finds the documents that need to be scored based on boolean logic in the Query specification, and then ranks this subset of matching documents via the retrieval model. For some valuable references on VSM and IR in general refer to [Lucene Wiki IR references](http://wiki.apache.org/lucene-java/InformationRetrieval).
-The rest of this document will cover [Scoring basics](#scoringbasics) and explain how to change your [Similarity](xref:Lucene.Net.Search.Similarities.Similarity). Next, it will cover ways you can customize the lucene internals in [Custom Queries -- Expert Level](#customQueriesExpert), which gives details on implementing your own [Query](xref:Lucene.Net.Search.Query) class and related functionality. Finally, we will finish up with some reference material in the [Appendix](#algorithm).
+The rest of this document will cover [Scoring basics](#scoring--basics) and explain how to change your [Similarity](xref:Lucene.Net.Search.Similarities.Similarity). Next, it will cover ways you can customize the Lucene internals in [Custom Queries -- Expert Level](#customQueriesExpert), which gives details on implementing your own [Query](xref:Lucene.Net.Search.Query) class and related functionality. Finally, we will finish up with some reference material in the [Appendix](#appendix-sear [...]
## Scoring — Basics
-Scoring is very much dependent on the way documents are indexed, so it is important to understand
- indexing. (see [Lucene overview]({@docRoot}/overview-summary.html#overview_description)
- before continuing on with this section) Be sure to use the useful
- [Doc)](xref:Lucene.Net.Search.IndexSearcher#methods)
- to understand how the score for a certain matching document was
- computed.
+Scoring is very much dependent on the way documents are indexed, so it is important to understand indexing. (see [Lucene overview](xref:Lucene.Net) before continuing on with this section) Be sure to use the useful [IndexSearcher.Explain(Query, int)](xref:Lucene.Net.Search.IndexSearcher#Lucene_Net_Search_IndexSearcher_Explain_Lucene_Net_Search_Query_System_Int32_) to understand how the score for a certain matching document was computed.
Generally, the Query determines which documents match (a binary decision), while the Similarity determines how to assign scores to the matching documents.
#### Fields and Documents
-In Lucene, the objects we are scoring are [Document](xref:Lucene.Net.Documents.Document)s. A Document is a collection of [Field](xref:Lucene.Net.Documents.Field)s. Each Field has [semantics](xref:Lucene.Net.Documents.FieldType) about how it is created and stored ([Tokenized](xref:Lucene.Net.Documents.FieldType#methods), [Stored](xref:Lucene.Net.Documents.FieldType#methods), etc). It is important to note that Lucene scoring works on Fields and then combines the results to return Documents [...]
+In Lucene, the objects we are scoring are [Document](xref:Lucene.Net.Documents.Document)s. A Document is a collection of [Field](xref:Lucene.Net.Documents.Field)s. Each Field has [semantics](xref:Lucene.Net.Documents.FieldType) about how it is created and stored ([Tokenized](xref:Lucene.Net.Documents.FieldType#Lucene_Net_Documents_FieldType_IsTokenized), [Stored](xref:Lucene.Net.Documents.FieldType#Lucene_Net_Documents_FieldType_IsStored), etc). It is important to note that Lucene scorin [...]
#### Score Boosting
-Lucene allows influencing search results by "boosting" at different times: * __Index-time boost__ by calling [Field.setBoost](xref:Lucene.Net.Documents.Field#methods) before a document is added to the index. * __Query-time boost__ by setting a boost on a query clause, calling [Query.setBoost](xref:Lucene.Net.Search.Query#methods).
+Lucene allows influencing search results by "boosting" at different times:
-Indexing time boosts are pre-processed for storage efficiency and written to storage for a field as follows: * All boosts of that field (i.e. all boosts under the same field name in that doc) are multiplied. * The boost is then encoded into a normalization value by the Similarity object at index-time: [ComputeNorm](xref:Lucene.Net.Search.Similarities.Similarity#methods). The actual encoding depends upon the Similarity implementation, but note that most use a lossy encoding (such as multi [...]
+* __Index-time boost__ by setting [Field.Boost](xref:Lucene.Net.Documents.Field#Lucene_Net_Documents_Field_Boost) before a document is added to the index.
+* __Query-time boost__ by setting a boost on a query clause, setting [Query.Boost](xref:Lucene.Net.Search.Query#Lucene_Net_Search_Query_Boost).
+
+Indexing time boosts are pre-processed for storage efficiency and written to storage for a field as follows:
+
+* All boosts of that field (i.e. all boosts under the same field name in that doc) are multiplied.
+* The boost is then encoded into a normalization value by the Similarity object at index-time: [ComputeNorm](xref:Lucene.Net.Search.Similarities.Similarity#Lucene_Net_Search_Similarities_Similarity_ComputeNorm_Lucene_Net_Index_FieldInvertState_). The actual encoding depends upon the Similarity implementation, but note that most use a lossy encoding (such as multiplying the boost with document length or similar, packed into a single byte!).
+* Decoding of any index-time normalization values and integration into the document's score is also performed at search time by the Similarity.
## Changing Scoring — Similarity
- Changing [Similarity](xref:Lucene.Net.Search.Similarities.Similarity) is an easy way to influence scoring, this is done at index-time with [IndexWriterConfig.setSimilarity](xref:Lucene.Net.Index.IndexWriterConfig#methods) and at query-time with [IndexSearcher.setSimilarity](xref:Lucene.Net.Search.IndexSearcher#methods). Be sure to use the same Similarity at query-time as at index-time (so that norms are encoded/decoded correctly); Lucene makes no effort to verify this.
+ Changing [Similarity](xref:Lucene.Net.Search.Similarities.Similarity) is an easy way to influence scoring, this is done at index-time with [IndexWriterConfig.setSimilarity](xref:Lucene.Net.Index.IndexWriterConfig#methods) and at query-time with [IndexSearcher.Similarity](xref:Lucene.Net.Search.IndexSearcher#Lucene_Net_Index_IndexWriterConfig_Similarity). Be sure to use the same Similarity at query-time as at index-time (so that norms are encoded/decoded correctly); Lucene makes no effor [...]
You can influence scoring by configuring a different built-in Similarity implementation, or by tweaking its parameters, subclassing it to override behavior. Some implementations also offer a modular API which you can extend by plugging in a different component (e.g. term frequency normalizer).
@@ -128,23 +140,49 @@ Indexing time boosts are pre-processed for storage efficiency and written to sto
Custom queries are an expert level task, so tread carefully and be prepared to share your code if you want help.
-With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by <span>three main classes</span>: 1. [Query](xref:Lucene.Net.Search.Query) — The abstract object representation of the user's information need. 2. [Weight](xref:Lucene.Net.Search.Weight) — The internal interface representation of the user's Query, so that Query objects may be reused. This i [...]
+With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to matching and scoring in Lucene. Lucene's search is a complex mechanism that is grounded by <span>three main classes</span>:
+
+1. [Query](xref:Lucene.Net.Search.Query) — The abstract object representation of the user's information need.
+2. [Weight](xref:Lucene.Net.Search.Weight) — The internal interface representation of the user's Query, so that Query objects may be reused. This is global (across all segments of the index) and generally will require global statistics (such as DocFreq for a given term across all segments).
+3. [Scorer](xref:Lucene.Net.Search.Scorer) — An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities. This is created per-segment.
+4. [BulkScorer](xref:Lucene.Net.Search.BulkScorer) — An abstract class that scores a range of documents. A default implementation simply iterates through the hits from [Scorer](xref:Lucene.Net.Search.Scorer), but some queries such as [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery) have more efficient implementations. Details on each of these classes, and their children, can be found in the subsections below.
#### The Query Class
-In some sense, the [Query](xref:Lucene.Net.Search.Query) class is where it all begins. Without a Query, there would be nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it is often responsible for creating them or coordinating the functionality between them. The [Query](xref:Lucene.Net.Search.Query) class has several methods that are important for derived classes: 1. [Searcher)](xref:Lucene.Net.Search.Query#methods) — A [Weight](xref:Lucene.N [...]
+In some sense, the [Query](xref:Lucene.Net.Search.Query) class is where it all begins. Without a Query, there would be nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it is often responsible for creating them or coordinating the functionality between them. The [Query](xref:Lucene.Net.Search.Query) class has several methods that are important for derived classes:
+
+1. [CreateWeight(IndexSearcher searcher)](xref:Lucene.Net.Search.Query#Lucene_Net_Search_Query_CreateWeight_Lucene_Net_Search_IndexSearcher_) — A [Weight](xref:Lucene.Net.Search.Weight) is the internal representation of the Query, so each Query implementation must provide an implementation of Weight. See the subsection on [The Weight Interface](#weightclass) below for details on implementing the Weight interface.
+2. [Rewrite(IndexReader reader)](xref:Lucene.Net.Search.Query#Lucene_Net_Search_Query_Rewrite_Lucene_Net_Index_IndexReader_) — Rewrites queries into primitive queries. Primitive queries are: [TermQuery](xref:Lucene.Net.Search.TermQuery), [BooleanQuery](xref:Lucene.Net.Search.BooleanQuery), <span>and other queries that implement [CreateWeight(IndexSearcher searcher)](xref:Lucene.Net.Search.Query#Lucene_Net_Search_Query_CreateWeight_Lucene_Net_Search_IndexSearcher_)</span>
#### The Weight Interface
-The [Weight](xref:Lucene.Net.Search.Weight) interface provides an internal representation of the Query so that it can be reused. Any [IndexSearcher](xref:Lucene.Net.Search.IndexSearcher) dependent state should be stored in the Weight implementation, not in the Query class. The interface defines five methods that must be implemented: 1. [GetQuery](xref:Lucene.Net.Search.Weight#methods) — Pointer to the Query that this Weight represents. 2. [GetValueForNormalization](xref:Lucene.Net.Search [...]
+The [Weight](xref:Lucene.Net.Search.Weight) interface provides an internal representation of the Query so that it can be reused. Any [IndexSearcher](xref:Lucene.Net.Search.IndexSearcher) dependent state should be stored in the Weight implementation, not in the Query class. The interface defines five members that must be implemented:
+
+1. [Query](xref:Lucene.Net.Search.Weight#Lucene_Net_Search_Weight_Query) — Pointer to the Query that this Weight represents.
+2. [GetValueForNormalization()](xref:Lucene.Net.Search.Weight#Lucene_Net_Search_Weight_GetValueForNormalization) — A weight can return a floating point value to indicate its magnitude for query normalization. Typically a weight such as TermWeight that scores via a [Similarity](xref:Lucene.Net.Search.Similarities.Similarity) will just defer to the Similarity's implementation: [SimWeight.GetValueForNormalization()](xref:Lucene.Net.Search.Similarities.Similarity.SimWeight#Lucene_Net_Search_ [...]
+3. [Normalize(float norm, float topLevelBoost)](xref:Lucene.Net.Search.Weight#Lucene_Net_Search_Weight_Normalize_System_Single_System_Single_) — Performs query normalization:
+ * `topLevelBoost`: A query-boost factor from any wrapping queries that should be multiplied into every document's score. For example, a TermQuery that is wrapped within a BooleanQuery with a boost of `5` would receive this value at this time. This allows the TermQuery (the leaf node in this case) to compute this up-front a single time (e.g. by multiplying into the IDF), rather than for every document.
+ * `norm`: Passes in a a normalization factor which may allow for comparing scores between queries. Typically a weight such as TermWeight that scores via a [Similarity](xref:Lucene.Net.Search.Similarities.Similarity) will just defer to the Similarity's implementation: [SimWeight.Normalize(float, float)](xref:Lucene.Net.Search.Similarities.Similarity.SimWeight#Lucene_Net_Search_Similarities_Similarity_SimWeight_Normalize_System_Single_System_Single_).
+4. [GetScorer(AtomicReaderContext context, IBits acceptDocs)](xref:Lucene.Net.Search.Weight#Lucene_Net_Search_Weight_GetScorer_Lucene_Net_Index_AtomicReaderContext_Lucene_Net_Util_IBits_) — Construct a new [Scorer](xref:Lucene.Net.Search.Scorer) for this Weight. See [The Scorer Class](#the-scorer-class) below for help defining a Scorer. As the name implies, the Scorer is responsible for doing the actual scoring of documents given the Query.
+5. [GetScorer(AtomicReaderContext, bool scoreDocsInOrder, IBits acceptDocs)](xref:Lucene.Net.Search.Weight#Lucene_Net_Search_Weight_GetBulkScorer_Lucene_Net_Index_AtomicReaderContext_System_Boolean_Lucene_Net_Util_IBits_) — Construct a new [BulkScorer](xref:Lucene.Net.Search.BulkScorer) for this Weight. See [The BulkScorer Class](#the-bulkscorer-class) below for help defining a BulkScorer. This is an optional method, and most queries do not implement it.
+6. [Explain(AtomicReaderContext context, int doc)](xref:Lucene.Net.Search.Weight#Lucene_Net_Search_Weight_Explain_Lucene_Net_Index_AtomicReaderContext_System_Int32_) — Provide a means for explaining why a given document was scored the way it was. Typically a weight such as TermWeight that scores via a [Similarity](xref:Lucene.Net.Search.Similarities.Similarity) will make use of the Similarity's implementation: [SimScorer.Explain(int doc, Explanation freq)](xref:Lucene.Net.Search.Similari [...]
#### The Scorer Class
-The [Scorer](xref:Lucene.Net.Search.Scorer) abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not yet abstract, but will be in future versions and should be considered as such now) methods which must be implemented (some of them inherited from [DocIdSetIterator](xref:Lucene.Net.Search.DocIdSetIterator)): 1. [NextDoc](xref:Lucene.Net.Search.Scorer#m [...]
+The [Scorer](xref:Lucene.Net.Search.Scorer) abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not yet abstract, but will be in future versions and should be considered as such now) methods which must be implemented (some of them inherited from [DocIdSetIterator](xref:Lucene.Net.Search.DocIdSetIterator)):
+
+1. [NextDoc()](xref:Lucene.Net.Search.DocIdSetIterator#Lucene_Net_Search_DocIdSetIterator_NextDoc) — Advances to the next document that matches this Query, returning true if and only if there is another document that matches.
+2. [DocID](xref:Lucene.Net.Search.DocIdSetIterator#Lucene_Net_Search_DocIdSetIterator_DocID) — Returns the id of the [Document](xref:Lucene.Net.Documents.Document) that contains the match.
+3. [GetScore()](xref:Lucene.Net.Search.Scorer#Lucene_Net_Search_Scorer_GetScore) — Return the score of the current document. This value can be determined in any appropriate way for an application. For instance, the [TermScorer](xref:Lucene.Net.Search.TermScorer) simply defers to the configured Similarity: [SimScorer.Score(int doc, float freq)](xref:Lucene.Net.Search.Similarities.Similarity.SimScorer#Lucene_Net_Search_Similarities_Similarity_SimScorer_Score_System_Int32_System_Single_).
+4. [Freq](xref:Lucene.Net.Index.DocsEnum#Lucene_Net_Index_DocsEnum_Freq) — Returns the number of matches for the current document. This value can be determined in any appropriate way for an application. For instance, the [TermScorer](xref:Lucene.Net.Search.TermScorer) simply defers to the term frequency from the inverted index: [DocsEnum.Freq](xref:Lucene.Net.Index.DocsEnum#Lucene_Net_Index_DocsEnum_Freq).
+5. [Advance()](xref:Lucene.Net.Search.Scorer#Lucene_Net_Search_DocIdSetIterator_Advance_System_Int32_) — Skip ahead in the document matches to the document whose id is greater than or equal to the passed in value. In many instances, advance can be implemented more efficiently than simply looping through all the matching documents until the target document is identified.
+6. [GetChildren()](xref:Lucene.Net.Search.Scorer#Lucene_Net_Search_Scorer_GetChildren) — Returns any child subscorers underneath this scorer. This allows for users to navigate the scorer hierarchy and receive more fine-grained details on the scoring process.
#### The BulkScorer Class
-The [BulkScorer](xref:Lucene.Net.Search.BulkScorer) scores a range of documents. There is only one abstract method: 1. [Score](xref:Lucene.Net.Search.BulkScorer#methods) — Score all documents up to but not including the specified max document.
+The [BulkScorer](xref:Lucene.Net.Search.BulkScorer) scores a range of documents. There is only one abstract method:
+
+1. [Score(ICollector, int)](xref:Lucene.Net.Search.BulkScorer#Lucene_Net_Search_BulkScorer_Score_Lucene_Net_Search_ICollector_System_Int32_) — Score all documents up to but not including the specified max document.
#### Why would I want to add my own Query?
@@ -156,12 +194,17 @@ This section is mostly notes on stepping through the Scoring process and serves
In the typical search application, a [Query](xref:Lucene.Net.Search.Query) is passed to the [IndexSearcher](xref:Lucene.Net.Search.IndexSearcher), beginning the scoring process.
-Once inside the IndexSearcher, a [Collector](xref:Lucene.Net.Search.Collector) is used for the scoring and sorting of the search results. These important objects are involved in a search: 1. The [Weight](xref:Lucene.Net.Search.Weight) object of the Query. The Weight object is an internal representation of the Query that allows the Query to be reused by the IndexSearcher. 2. The IndexSearcher that initiated the call. 3. A [Filter](xref:Lucene.Net.Search.Filter) for limiting the result set [...]
+Once inside the IndexSearcher, a [ICollector](xref:Lucene.Net.Search.ICollector) is used for the scoring and sorting of the search results. These important objects are involved in a search:
+
+1. The [Weight](xref:Lucene.Net.Search.Weight) object of the Query. The Weight object is an internal representation of the Query that allows the Query to be reused by the IndexSearcher.
+2. The IndexSearcher that initiated the call.
+3. A [Filter](xref:Lucene.Net.Search.Filter) for limiting the result set. Note, the Filter may be `null`.
+4. A [Sort](xref:Lucene.Net.Search.Sort) object for specifying how to sort the results if the standard score-based sort method is not desired.
-Assuming we are not sorting (since sorting doesn't affect the raw Lucene score), we call one of the search methods of the IndexSearcher, passing in the [Weight](xref:Lucene.Net.Search.Weight) object created by [IndexSearcher.createNormalizedWeight](xref:Lucene.Net.Search.IndexSearcher#methods), [Filter](xref:Lucene.Net.Search.Filter) and the number of results we want. This method returns a [TopDocs](xref:Lucene.Net.Search.TopDocs) object, which is an internal collection of search results [...]
+Assuming we are not sorting (since sorting doesn't affect the raw Lucene score), we call one of the search methods of the IndexSearcher, passing in the [Weight](xref:Lucene.Net.Search.Weight) object created by [IndexSearcher.CreateNormalizedWeight(Query)](xref:Lucene.Net.Search.IndexSearcher#Lucene_Net_Search_IndexSearcher_CreateNormalizedWeight_Lucene_Net_Search_Query_), [Filter](xref:Lucene.Net.Search.Filter) and the number of results we want. This method returns a [TopDocs](xref:Lucen [...]
-If a Filter is being used, some initial setup is done to determine which docs to include. Otherwise, we ask the Weight for a [Scorer](xref:Lucene.Net.Search.Scorer) for each [IndexReader](xref:Lucene.Net.Index.IndexReader) segment and proceed by calling [BulkScorer.score](xref:Lucene.Net.Search.BulkScorer#methods).
+If a Filter is being used, some initial setup is done to determine which docs to include. Otherwise, we ask the Weight for a [Scorer](xref:Lucene.Net.Search.Scorer) for each [IndexReader](xref:Lucene.Net.Index.IndexReader) segment and proceed by calling [BulkScorer.Score(ICollector)](xref:Lucene.Net.Search.BulkScorer#Lucene_Net_Search_BulkScorer_Score_Lucene_Net_Search_ICollector_).
-At last, we are actually going to score some documents. The score method takes in the Collector (most likely the TopScoreDocCollector or TopFieldCollector) and does its business.Of course, here is where things get involved. The [Scorer](xref:Lucene.Net.Search.Scorer) that is returned by the [Weight](xref:Lucene.Net.Search.Weight) object depends on what type of Query was submitted. In most real world applications with multiple query terms, the [Scorer](xref:Lucene.Net.Search.Scorer) is go [...]
+At last, we are actually going to score some documents. The score method takes in the ICollector (most likely the TopScoreDocCollector or TopFieldCollector) and does its business. Of course, here is where things get involved. The [Scorer](xref:Lucene.Net.Search.Scorer) that is returned by the [Weight](xref:Lucene.Net.Search.Weight) object depends on what type of Query was submitted. In most real world applications with multiple query terms, the [Scorer](xref:Lucene.Net.Search.Scorer) is [...]
-Assuming a BooleanScorer2, we first initialize the Coordinator, which is used to apply the coord() factor. We then get a internal Scorer based on the required, optional and prohibited parts of the query. Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the [Scorer.nextDoc](xref:Lucene.Net.Search.Scorer#methods) method. The nextDoc() method advances to the next document matching the query. This is an abstract method in the Scorer class and is thus ov [...]
\ No newline at end of file
+Assuming a BooleanScorer2, we first initialize the Coordinator, which is used to apply the Coord() factor. We then get a internal Scorer based on the required, optional and prohibited parts of the query. Using this internal Scorer, the BooleanScorer2 then proceeds into a while loop based on the [Scorer.NextDoc()](xref:Lucene.Net.Search.DocIdSetIterator#Lucene_Net_Search_DocIdSetIterator_NextDoc) method. The NextDoc() method advances to the next document matching the query. This is an abs [...]
\ No newline at end of file
diff --git a/src/Lucene.Net/Util/Fst/package.md b/src/Lucene.Net/Util/Fst/package.md
index aa0f391..d75f4d0 100644
--- a/src/Lucene.Net/Util/Fst/package.md
+++ b/src/Lucene.Net/Util/Fst/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Util.Fst
summary: *content
---
@@ -30,65 +30,87 @@ Finite State Transducers](http://en.wikipedia.org/wiki/Finite_state_transducer)
* Low object overhead and quick deserialization (byte[] representation)
-* Optional two-pass compression: [FST.pack](xref:Lucene.Net.Util.Fst.FST#methods)
+<!-- LUCENENET NOTE: This method is marked internal in Lucene and their link doesn't work -->
+* Optional two-pass compression: [FST.Pack()](xref:Lucene.Net.Util.Fst.FST#methods)
-* [Lookup-by-output](xref:Lucene.Net.Util.Fst.Util#methods) when the
+* [Lookup-by-output](xref:Lucene.Net.Util.Fst.Util#Lucene_Net_Util_Fst_Util_GetByOutput_Lucene_Net_Util_Fst_FST_System_Nullable_System_Int64___System_Int64_) when the
outputs are in sorted order (e.g., ordinals or file pointers)
* Pluggable [Outputs](xref:Lucene.Net.Util.Fst.Outputs) representation
-* [N-shortest-paths](xref:Lucene.Net.Util.Fst.Util#methods) search by
+* [N-shortest-paths](xref:Lucene.Net.Util.Fst.Util#Lucene_Net_Util_Fst_Util_ShortestPaths__1_Lucene_Net_Util_Fst_FST___0__Lucene_Net_Util_Fst_FST_Arc___0____0_System_Collections_Generic_IComparer___0__System_Int32_System_Boolean_) search by
weight
-* Enumerators ([IntsRef](xref:Lucene.Net.Util.Fst.IntsRefFSTEnum) and [BytesRef](xref:Lucene.Net.Util.Fst.BytesRefFSTEnum)) that behave like {@link java.util.SortedMap SortedMap} iterators
+* Enumerators ([Int32sRef](xref:Lucene.Net.Util.Fst.Int32sRefFSTEnum) and [BytesRef](xref:Lucene.Net.Util.Fst.BytesRefFSTEnum)) that behave like [SortedDictionary<TKey, TValue>](https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.sorteddictionary-2) enumerators
FST Construction example:
- // Input values (keys). These must be provided to Builder in Unicode sorted order!
- String inputValues[] = {"cat", "dog", "dogs"};
- long outputValues[] = {5, 7, 12};
-
- PositiveIntOutputs outputs = PositiveIntOutputs.getSingleton();
- Builder<Long> builder = new Builder<Long>(INPUT_TYPE.BYTE1, outputs);
- BytesRef scratchBytes = new BytesRef();
- IntsRef scratchInts = new IntsRef();
- for (int i = 0; i < inputValues.length; i++) {
- scratchBytes.copyChars(inputValues[i]);
- builder.add(Util.toIntsRef(scratchBytes, scratchInts), outputValues[i]);
- }
- FST<Long> fst = builder.finish();
+```cs
+// Input values (keys). These must be provided to Builder in Unicode sorted order!
+string[] inputValues = new string[] { "cat", "dog", "dogs" };
+long[] outputValues = new long[] { 5, 7, 12 };
+
+PositiveInt32Outputs outputs = PositiveInt32Outputs.Singleton;
+Builder<long?> builder = new Builder<long?>(FST.INPUT_TYPE.BYTE1, outputs);
+BytesRef scratchBytes = new BytesRef();
+Int32sRef scratchInts = new Int32sRef();
+for (int i = 0; i < inputValues.Length; i++)
+{
+ scratchBytes.CopyChars(inputValues[i]);
+ builder.Add(Util.ToInt32sRef(scratchBytes, scratchInts), outputValues[i]);
+}
+FST<long?> fst = builder.Finish();
+```
Retrieval by key:
- Long value = Util.get(fst, new BytesRef("dog"));
- System.out.println(value); // 7
+```cs
+long? value = Util.Get(fst, new BytesRef("dog"));
+Console.WriteLine(value); // 7
+```
Retrieval by value:
- // Only works because outputs are also in sorted order
- IntsRef key = Util.getByOutput(fst, 12);
- System.out.println(Util.toBytesRef(key, scratchBytes).utf8ToString()); // dogs
-
-Iterate over key-value pairs in sorted order:
-
- // Like TermsEnum, this also supports seeking (advance)
- BytesRefFSTEnum<Long> iterator = new BytesRefFSTEnum<Long>(fst);
- while (iterator.next() != null) {
- InputOutput<Long> mapEntry = iterator.current();
- System.out.println(mapEntry.input.utf8ToString());
- System.out.println(mapEntry.output);
- }
+```cs
+// Only works because outputs are also in sorted order
+Int32sRef key = Util.GetByOutput(fst, 12);
+Console.WriteLine(Util.ToBytesRef(key, scratchBytes).Utf8ToString()); // dogs
+```
+
+Iterate over key - value pairs in sorted order:
+
+```cs
+// Like TermsEnum, this also supports seeking (advance)
+BytesRefFSTEnum<long?> enumerator = new BytesRefFSTEnum<long?>(fst);
+while (enumerator.MoveNext())
+{
+ BytesRefFSTEnum.InputOutput<long?> mapEntry = enumerator.Current;
+ Console.WriteLine(mapEntry.Input.Utf8ToString());
+ Console.WriteLine(mapEntry.Output);
+}
+```
N-shortest paths by weight:
- Comparator<Long> comparator = new Comparator<Long>() {
- public int compare(Long left, Long right) {
- return left.compareTo(right);
- }
- };
- Arc<Long> firstArc = fst.getFirstArc(new Arc<Long>());
- MinResult<Long> paths[] = Util.shortestPaths(fst, firstArc, comparator, 2);
- System.out.println(Util.toBytesRef(paths[0].input, scratchBytes).utf8ToString()); // cat
- System.out.println(paths[0].output); // 5
- System.out.println(Util.toBytesRef(paths[1].input, scratchBytes).utf8ToString()); // dog
- System.out.println(paths[1].output); // 7
\ No newline at end of file
+```cs
+var comparer = Comparer<long?>.Create((left, right) =>
+{
+ return left.GetValueOrDefault().CompareTo(right);
+});
+FST.Arc<long?> firstArc = fst.GetFirstArc(new FST.Arc<long?>());
+Util.TopResults<long?> paths = Util.ShortestPaths(
+ fst, firstArc, startOutput: 0, comparer, topN: 2, allowEmptyString: false);
+
+foreach (Util.Result<long?> path in paths)
+{
+ Console.WriteLine(Util.ToBytesRef(path.Input, scratchBytes).Utf8ToString());
+ Console.WriteLine(path.Output);
+}
+
+// Results:
+//
+// cat
+// 5
+// dog
+// 7
+```
\ No newline at end of file
diff --git a/src/Lucene.Net/Util/Packed/package.md b/src/Lucene.Net/Util/Packed/package.md
index ef153c9..18d003b 100644
--- a/src/Lucene.Net/Util/Packed/package.md
+++ b/src/Lucene.Net/Util/Packed/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Util.Packed
summary: *content
---
@@ -22,13 +22,18 @@ summary: *content
## Packed integer arrays and streams.
- The packed package provides * sequential and random access capable arrays of positive longs, * routines for efficient serialization and deserialization of streams of packed integers. The implementations provide different trade-offs between memory usage and access speed. The standard usage scenario is replacing large int or long arrays in order to reduce the memory footprint.
+ The packed package provides
+
+ * sequential and random access capable arrays of positive longs,
+ * routines for efficient serialization and deserialization of streams of packed integers.
+
+ The implementations provide different trade-offs between memory usage and access speed. The standard usage scenario is replacing large int or long arrays in order to reduce the memory footprint.
- The main access point is the <xref:Lucene.Net.Util.Packed.PackedInts> factory.
+ The main access point is the <xref:Lucene.Net.Util.Packed.PackedInt32s> factory.
### In-memory structures
-* __<xref:Lucene.Net.Util.Packed.PackedInts.Mutable>__
+* __<xref:Lucene.Net.Util.Packed.PackedInt32s.Mutable>__
* Only supports positive longs.
@@ -50,7 +55,7 @@ summary: *content
* You should use Appending(Delta)PackedLongBuffer instead if you don't need random write access.
-* __<xref:Lucene.Net.Util.Packed.AppendingDeltaPackedLongBuffer>__
+* __<xref:Lucene.Net.Util.Packed.AppendingDeltaPackedInt64Buffer>__
* Can store any sequence of longs.
@@ -60,17 +65,17 @@ summary: *content
* Can address up to 2^42 values.
-* __<xref:Lucene.Net.Util.Packed.AppendingPackedLongBuffer>__
+* __<xref:Lucene.Net.Util.Packed.AppendingPackedInt64Buffer>__
- * Same as AppendingDeltaPackedLongBuffer but assumes values are 0-based.
+ * Same as AppendingDeltaPackedInt64Buffer but assumes values are 0-based.
-* __<xref:Lucene.Net.Util.Packed.MonotonicAppendingLongBuffer>__
+* __<xref:Lucene.Net.Util.Packed.MonotonicAppendingInt64Buffer>__
- * Same as AppendingDeltaPackedLongBuffer except that compression is good when the stream is a succession of affine functions.
+ * Same as AppendingDeltaPackedInt64Buffer except that compression is good when the stream is a succession of affine functions.
### Disk-based structures
-* __<xref:Lucene.Net.Util.Packed.PackedInts.Writer>, <xref:Lucene.Net.Util.Packed.PackedInts.Reader>, <xref:Lucene.Net.Util.Packed.PackedInts.ReaderIterator>__
+* __<xref:Lucene.Net.Util.Packed.PackedInt32s.Writer>, <xref:Lucene.Net.Util.Packed.PackedInt32s.Reader>, <xref:Lucene.Net.Util.Packed.PackedInt32s.IReaderIterator>__
* Only supports positive longs.
[lucenenet] 01/06: docs: Lucene.Net/Index/package.md: Fixed broken
formatting and links (see #284, #300)
Posted by ni...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
commit 4be60a7b28c4b63c94649628d809236959d4b171
Author: Shad Storhaug <sh...@shadstorhaug.com>
AuthorDate: Sun Mar 28 06:56:33 2021 +0700
docs: Lucene.Net/Index/package.md: Fixed broken formatting and links (see #284, #300)
---
src/Lucene.Net/Index/package.md | 154 +++++++++++++++++++++++++++++++---------
1 file changed, 119 insertions(+), 35 deletions(-)
diff --git a/src/Lucene.Net/Index/package.md b/src/Lucene.Net/Index/package.md
index 36aa8de..87c4896 100644
--- a/src/Lucene.Net/Index/package.md
+++ b/src/Lucene.Net/Index/package.md
@@ -1,4 +1,4 @@
----
+---
uid: Lucene.Net.Index
summary: *content
---
@@ -24,52 +24,136 @@ Code to maintain and access indices.
## Table Of Contents
- 1. [Postings APIs](#postings) * [Fields](#fields) * [Terms](#terms) * [Documents](#documents) * [Positions](#positions) 2. [Index Statistics](#stats) * [Term-level](#termstats) * [Field-level](#fieldstats) * [Segment-level](#segmentstats) * [Document-level](#documentstats)
+ 1. [Postings APIs](#postings)
+ * [Fields](#fields)
+ * [Terms](#terms)
+ * [Documents](#documents)
+ * [Positions](#positions)
+ 2. [Index Statistics](#index-statistics)
+ * [Term-level](#term-statistics)
+ * [Field-level](#field-statistics)
+ * [Segment-level](#segment-statistics)
+ * [Document-level](#document-statistics)
## Postings APIs
-####
- Fields
-
- <xref:Lucene.Net.Index.Fields> is the initial entry point into the postings APIs, this can be obtained in several ways: // access indexed fields for an index segment Fields fields = reader.fields(); // access term vector fields for a specified document Fields fields = reader.getTermVectors(docid); Fields implements Java's Iterable interface, so its easy to enumerate the list of fields: // enumerate list of fields for (String field : fields) { // access the terms for this field Terms ter [...]
-
-####
- Terms
-
- <xref:Lucene.Net.Index.Terms> represents the collection of terms within a field, exposes some metadata and [statistics](#fieldstats), and an API for enumeration. // metadata about the field System.out.println("positions? " + terms.hasPositions()); System.out.println("offsets? " + terms.hasOffsets()); System.out.println("payloads? " + terms.hasPayloads()); // iterate through terms TermsEnum termsEnum = terms.iterator(null); BytesRef term = null; while ((term = termsEnum.next()) != null) [...]
-
-####
- Documents
-
- <xref:Lucene.Net.Index.DocsEnum> is an extension of <xref:Lucene.Net.Search.DocIdSetIterator>that iterates over the list of documents for a term, along with the term frequency within that document. int docid; while ((docid = docsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) { System.out.println(docid); System.out.println(docsEnum.freq()); }
-
-####
- Positions
-
- <xref:Lucene.Net.Index.DocsAndPositionsEnum> is an extension of <xref:Lucene.Net.Index.DocsEnum> that additionally allows iteration of the positions a term occurred within the document, and any additional per-position information (offsets and payload) int docid; while ((docid = docsAndPositionsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS) { System.out.println(docid); int freq = docsAndPositionsEnum.freq(); for (int i = 0; i < freq;="" i++)="" {="" system.out.println(docsandpositions [...]
+#### Fields
+
+ <xref:Lucene.Net.Index.Fields> is the initial entry point into the postings APIs, this can be obtained in several ways:
+
+```cs
+// access indexed fields for an index segment
+Fields fields = reader.Fields; // access term vector fields for a specified document
+Fields fields = reader.GetTermVectors(docid);
+```
+
+Fields implements .NET's `IEnumerable<T>` interface, so its easy to enumerate the list of fields:
+
+```cs
+// enumerate list of fields
+foreach (string field in fields) // access the terms for this field
+{
+ Terms terms = fields.GetTerms(field);
+}
+```
+
+#### Terms
+
+ <xref:Lucene.Net.Index.Terms> represents the collection of terms within a field, exposes some metadata and [statistics](#field-statistics), and an API for enumeration.
+
+```cs
+// metadata about the field
+Console.WriteLine("positions? " + terms.HasPositions);
+Console.WriteLine("offsets? " + terms.HasOffsets);
+Console.WriteLine("payloads? " + terms.HasPayloads);
+// iterate through terms
+TermsEnum termsEnum = terms.GetEnumerator();
+while (termsEnum.MoveNext())
+{
+ DoSomethingWith(termsEnum.Term); // Term is a BytesRef
+}
+```
+
+<xref:Lucene.Net.Index.TermsEnum> provides an enumerator over the list of terms within a field, some [statistics](#term-statistics) about the term, and methods to access the term's [documents](#documents) and [positions](#positions).
+
+```cs
+// seek to a specific term
+bool found = termsEnum.SeekExact(new BytesRef("foobar"));
+if (found)
+{
+ // get the document frequency
+ Console.WriteLine(termsEnum.DocFreq);
+ // enumerate through documents
+ DocsEnum docs = termsEnum.Docs(null, null);
+ // enumerate through documents and positions
+ DocsAndPositionsEnum docsAndPositions = termsEnum.DocsAndPositions(null, null);
+}
+```
+
+#### Documents
+
+ <xref:Lucene.Net.Index.DocsEnum> is an extension of <xref:Lucene.Net.Search.DocIdSetIterator> that iterates over the list of documents for a term, along with the term frequency within that document.
+
+```cs
+int docid;
+while ((docid = docsEnum.NextDoc()) != DocIdSetIterator.NO_MORE_DOCS)
+{
+ Console.WriteLine(docid);
+ Console.WriteLine(docsEnum.Freq);
+}
+```
+
+#### Positions
+
+ <xref:Lucene.Net.Index.DocsAndPositionsEnum> is an extension of <xref:Lucene.Net.Index.DocsEnum> that additionally allows iteration of the positions a term occurred within the document, and any additional per-position information (offsets and payload)
+
+```cs
+int docid;
+while ((docid = docsAndPositionsEnum.NextDoc()) != DocIdSetIterator.NO_MORE_DOCS)
+{
+ Console.WriteLine(docid);
+ int freq = docsAndPositionsEnum.Freq;
+ for (int i = 0; i < freq; i++)
+ {
+ Console.WriteLine(docsAndPositionsEnum.NextPosition());
+ Console.WriteLine(docsAndPositionsEnum.StartOffset);
+ Console.WriteLine(docsAndPositionsEnum.EndOffset);
+ Console.WriteLine(docsAndPositionsEnum.GetPayload());
+ }
+}
+```
## Index Statistics
-####
- Term statistics
+#### Term statistics
- * [#docFreq](xref:Lucene.Net.Index.TermsEnum): Returns the number of documents that contain at least one occurrence of the term. This statistic is always available for an indexed term. Note that it will also count deleted documents, when segments are merged the statistic is updated as those deleted documents are merged away. * [#totalTermFreq](xref:Lucene.Net.Index.TermsEnum): Returns the number of occurrences of this term across all documents. Note that this statistic is unavailable (r [...]
+ * [DocFreq](xref:Lucene.Net.Index.TermsEnum#Lucene_Net_Index_TermsEnum_DocFreq): Returns the number of documents that contain at least one occurrence of the term. This statistic is always available for an indexed term. Note that it will also count deleted documents, when segments are merged the statistic is updated as those deleted documents are merged away.
+ * [TotalTermFreq](xref:Lucene.Net.Index.TermsEnum#Lucene_Net_Index_TermsEnum_TotalTermFreq): Returns the number of occurrences of this term across all documents. Note that this statistic is unavailable (returns `-1`) if term frequencies were omitted from the index ([DOCS_ONLY](xref:Lucene.Net.Index.FieldInfo.IndexOptions#Lucene_Net_Index_IndexOptions_DOCS_ONLY)) for the field. Like `DocFreq`, it will also count occurrences that appear in deleted documents.
-####
- Field statistics
+#### Field statistics
- * [#size](xref:Lucene.Net.Index.Terms): Returns the number of unique terms in the field. This statistic may be unavailable (returns `-1`) for some Terms implementations such as <xref:Lucene.Net.Index.MultiTerms>, where it cannot be efficiently computed. Note that this count also includes terms that appear only in deleted documents: when segments are merged such terms are also merged away and the statistic is then updated. * [#getDocCount](xref:Lucene.Net.Index.Terms): Returns the number [...]
+ * [Count](xref:Lucene.Net.Index.Terms#Lucene_Net_Index_Terms_Count): Returns the number of unique terms in the field. This statistic may be unavailable (returns `-1`) for some Terms implementations such as <xref:Lucene.Net.Index.MultiTerms>, where it cannot be efficiently computed. Note that this count also includes terms that appear only in deleted documents: when segments are merged such terms are also merged away and the statistic is then updated.
+ * [DocCount](xref:Lucene.Net.Index.Terms#Lucene_Net_Index_Terms_DocCount): Returns the number of documents that contain at least one occurrence of any term for this field. This can be thought of as a Field-level `DocFreq`. Like `DocFreq` it will also count deleted documents.
+ * [SumDocFreq](xref:Lucene.Net.Index.Terms#Lucene_Net_Index_Terms_SumDocFreq): Returns the number of postings (term-document mappings in the inverted index) for the field. This can be thought of as the sum of [TermsEnum.DocFreq](xref:Lucene.Net.Index.TermsEnum#Lucene_Net_Index_TermsEnum_DocFreq) across all terms in the field, and like `DocFreq` it will also count postings that appear in deleted documents.
+ * [SumTotalTermFreq](xref:Lucene.Net.Index.Terms#Lucene_Net_Index_Terms_SumTotalTermFreq): Returns the number of tokens for the field. This can be thought of as the sum of [TermsEnum.TotalTermFreq](xref:Lucene.Net.Index.TermsEnum#Lucene_Net_Index_TermsEnum_TotalTermFreq) across all terms in the field, and like `TotalTermFreq` it will also count occurrences that appear in deleted documents, and will be unavailable (returns `-1`) if term frequencies were omitted from the index ([DOCS_ONLY [...]
-####
- Segment statistics
+#### Segment statistics
- * [#maxDoc](xref:Lucene.Net.Index.IndexReader): Returns the number of documents (including deleted documents) in the index. * [#numDocs](xref:Lucene.Net.Index.IndexReader): Returns the number of live documents (excluding deleted documents) in the index. * [#numDeletedDocs](xref:Lucene.Net.Index.IndexReader): Returns the number of deleted documents in the index. * [#size](xref:Lucene.Net.Index.Fields): Returns the number of indexed fields. * [#getUniqueTermCount](xref:Lucene.Net.Index.Fi [...]
+ * [MaxDoc](xref:Lucene.Net.Index.IndexReader#Lucene_Net_Index_IndexReader_MaxDoc): Returns the number of documents (including deleted documents) in the index.
+ * [NumDocs](xref:Lucene.Net.Index.IndexReader#Lucene_Net_Index_IndexReader_NumDocs): Returns the number of live documents (excluding deleted documents) in the index.
+ * [NumDeletedDocs](xref:Lucene.Net.Index.IndexReader#Lucene_Net_Index_IndexReader_NumDeletedDocs): Returns the number of deleted documents in the index.
+ * [Count](xref:Lucene.Net.Index.Fields#Lucene_Net_Index_Fields_Count): Returns the number of indexed fields.
+ * [UniqueTermCount](xref:Lucene.Net.Index.Fields#Lucene_Net_Index_Fields_UniqueTermCount): Returns the number of indexed terms, the sum of [Count](xref:Lucene.Net.Index.Terms#Lucene_Net_Index_Terms_Count) across all fields.
-####
- Document statistics
+#### Document statistics
- Document statistics are available during the indexing process for an indexed field: typically a <xref:Lucene.Net.Search.Similarities.Similarity> implementation will store some of these values (possibly in a lossy way), into the normalization value for the document in its [#computeNorm](xref:Lucene.Net.Search.Similarities.Similarity) method.
+ Document statistics are available during the indexing process for an indexed field: typically a <xref:Lucene.Net.Search.Similarities.Similarity> implementation will store some of these values (possibly in a lossy way), into the normalization value for the document in its [Similarity.ComputeNorm(FieldInvertState)](xref:Lucene.Net.Search.Similarities.Similarity#Lucene_Net_Search_Similarities_Similarity_ComputeNorm_Lucene_Net_Index_FieldInvertState_) method.
- * [#getLength](xref:Lucene.Net.Index.FieldInvertState): Returns the number of tokens for this field in the document. Note that this is just the number of times that [#incrementToken](xref:Lucene.Net.Analysis.TokenStream) returned true, and is unrelated to the values in <xref:Lucene.Net.Analysis.TokenAttributes.PositionIncrementAttribute>. * [#getNumOverlap](xref:Lucene.Net.Index.FieldInvertState): Returns the number of tokens for this field in the document that had a position increment [...]
+ * [Length](xref:Lucene.Net.Index.FieldInvertState#Lucene_Net_Index_FieldInvertState_Length): Returns the number of tokens for this field in the document. Note that this is just the number of times that [IncrementToken()](xref:Lucene.Net.Analysis.TokenStream#Lucene_Net_Analysis_TokenStream_IncrementToken) returned `true`, and is unrelated to the values in <xref:Lucene.Net.Analysis.TokenAttributes.PositionIncrementAttribute>.
+ * [NumOverlap](xref:Lucene.Net.Index.FieldInvertState#Lucene_Net_Index_FieldInvertState_NumOverlap): Returns the number of tokens for this field in the document that had a position increment of zero. This can be used to compute a document length that discounts artificial tokens such as synonyms.
+ * [Position](xref:Lucene.Net.Index.FieldInvertState#Lucene_Net_Index_FieldInvertState_Position): Returns the accumulated position value for this field in the document: computed from the values of <xref:Lucene.Net.Analysis.TokenAttributes.PositionIncrementAttribute> and including [GetPositionIncrementGap(String)](xref:Lucene.Net.Analysis.Analyzer#Lucene_Net_Analysis_Analyzer_GetPositionIncrementGap_System_String_)s across multivalued fields.
+ * [Offset](xref:Lucene.Net.Index.FieldInvertState#Lucene_Net_Index_FieldInvertState_Offset): Returns the total character offset value for this field in the document: computed from the values of <xref:Lucene.Net.Analysis.TokenAttributes.OffsetAttribute> returned by [End()](xref:Lucene.Net.Analysis.TokenStream#Lucene_Net_Analysis_TokenStream_End), and including [GetOffsetGap(String)](xref:Lucene.Net.Analysis.Analyzer#Lucene_Net_Analysis_Analyzer_GetOffsetGap_System_String)s across multiva [...]
+ * [UniqueTermCount](xref:Lucene.Net.Index.FieldInvertState#Lucene_Net_Index_FieldInvertState_UniqueTermCount): Returns the number of unique terms encountered for this field in the document.
+ * [MaxTermFrequency](xref:Lucene.Net.Index.FieldInvertState#Lucene_Net_Index_FieldInvertState_MaxTermFrequency): Returns the maximum frequency across all unique terms encountered for this field in the document.
- Additional user-supplied statistics can be added to the document as DocValues fields and accessed via [#getNumericDocValues](xref:Lucene.Net.Index.AtomicReader).
\ No newline at end of file
+ Additional user-supplied statistics can be added to the document as DocValues fields and accessed via [GetNumericDocValues(String)](xref:Lucene.Net.Index.AtomicReader#Lucene_Net_Index_AtomicReader_GetNumericDocValues_System_String_).
\ No newline at end of file