You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@jena.apache.org by an...@apache.org on 2022/11/04 14:43:12 UTC

[jena-site] branch lucene-9 created (now cd7efe318)

This is an automated email from the ASF dual-hosted git repository.

andy pushed a change to branch lucene-9
in repository https://gitbox.apache.org/repos/asf/jena-site.git


      at cd7efe318 Note change of StandardAnalyzer stopwords at Lucene 9

This branch includes the following new commits:

     new cd7efe318 Note change of StandardAnalyzer stopwords at Lucene 9

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



[jena-site] 01/01: Note change of StandardAnalyzer stopwords at Lucene 9

Posted by an...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

andy pushed a commit to branch lucene-9
in repository https://gitbox.apache.org/repos/asf/jena-site.git

commit cd7efe3183e37e8d2d4d5abafda9a7e83067a9a5
Author: Andy Seaborne <an...@apache.org>
AuthorDate: Fri Nov 4 14:43:00 2022 +0000

    Note change of StandardAnalyzer stopwords at Lucene 9
---
 source/documentation/query/text-query.md | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/source/documentation/query/text-query.md b/source/documentation/query/text-query.md
index 8141244ed..bb8df34f1 100644
--- a/source/documentation/query/text-query.md
+++ b/source/documentation/query/text-query.md
@@ -13,7 +13,11 @@ searches within SPARQL queries. Here is a version compatibility table:
 | 3.3.0 - 3.9.0    | 6.4.x              | not supported     | 5.2.2 - 5.2.13 |
 | 3.10.0           | 7.4.0              | not supported     | 6.4.2          |
 | 3.15.0 - 3.17.0  | 7.7.x              | not supported     | 6.8.6          |
-| 4.0.0 - current  | 8.8.x              | not supported     | not supported  |
+| 4.0.0 - 4.6.1    | 8.8.x              | not supported     | not supported  |
+| 4.7.0 - current  | 9.4.x              | not supported     | not supported  |
+
+Note: In Lucene 9, the default setup of the `StandardAnalyzer` changed to having
+no stop words. For more details, see [analyzer specifications](#configuring-an-analyzer) below.
 
 SPARQL allows the use of 
 [regex](https://www.w3.org/TR/2013/REC-sparql11-query-20130321/#func-regex) 
@@ -988,6 +992,16 @@ Text to be indexed is passed through a text analyzer that divides it into tokens
 and may perform other transformations such as eliminating stop words. If a Lucene
 or Elasticsearch text index is used, then by default the Lucene `StandardAnalyzer` is used.
 
+As of Jena 4.7.x / Lucene 9.x onwards, the `StandardAnalyzer` does not default to having
+English stopwords if no stop words are provided. The setting up until
+Apache Lucene 8 had the stopwords:
+
+<pre>
+      "a"  "an"  "and"  "are"  "as"  "at"  "be"  "but"  "by"  "for"  "if"  "in"  "into"  "is" 
+      "it"  "no"  "not"  "of"  "on"  "or"  "such"  "that"  "the"  "their"  "then"  "there" 
+      "these"  "they"  "this"  "to"  "was"  "will"  "with"
+</pre>
+
 In case of a `TextIndexLucene` the default analyzer can be replaced by another analyzer with 
 the `text:analyzer` property on the `text:TextIndexLucene` resource in the 
 [text dataset assembler](#text-dataset-assembler),  for example with a `SimpleAnalyzer`: