You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "AndreyBozhko (via GitHub)" <gi...@apache.org> on 2023/09/26 22:00:30 UTC
[GitHub] [solr] AndreyBozhko opened a new pull request, #1961: Update indexing docs in Solr ref guide
AndreyBozhko opened a new pull request, #1961:
URL: https://github.com/apache/solr/pull/1961
https://issues.apache.org/jira/browse/SOLR-XXXXX
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Solr:
* https://issues.apache.org/jira/projects/SOLR
For something minor (i.e. that wouldn't be worth putting in release notes), you can skip JIRA.
To create a Jira issue, you will need to create an account there first.
The title of the PR should reference the Jira issue number in the form:
* SOLR-####: <short description of problem or changes>
SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
This PR contains several updates for the Solr ref guide:
- use consistent formatting for descriptions of tokenizer/filter parameters
- rename `CharFilterFactories` page to `CharFilters` (no other page has "Factories" in its title)
- update descriptions of parameters for tokenizers/filters that had outdated docs; the up-to-date parameter names and descriptions are taken from the corresponding Lucene javadocs
- fix a few typos
# Solution
Please provide a short description of the approach taken to implement your solution.
# Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `main` branch.
- [x] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Update indexing docs in Solr ref guide [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh merged PR #1961:
URL: https://github.com/apache/solr/pull/1961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] AndreyBozhko commented on a diff in pull request #1961: Update indexing docs in Solr ref guide
Posted by "AndreyBozhko (via GitHub)" <gi...@apache.org>.
AndreyBozhko commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1341570013
##########
solr/solr-ref-guide/modules/indexing-guide/pages/field-type-definitions-and-properties.adoc:
##########
@@ -201,7 +201,7 @@ The table below includes the default value for most `FieldType` implementations
|`sortMissingFirst`, `sortMissingLast` |Control the placement of documents when a sort field is not present. |`false`
|`multiValued` |If `true`, indicates that a single document might contain multiple values for this field type. |`false`
|`uninvertible` |If `true`, indicates that an `indexed="true" docValues="false"` field can be "un-inverted" at query time to build up large in memory data structure to serve in place of xref:docvalues.adoc[]. *Defaults to `true` for historical reasons, but users are strongly encouraged to set this to `false` for stability and use `docValues="true"` as needed.* |`true`
-|`omitNorms` |If `true`, omits the norms associated with this field (this disables length normalization for the field, and saves some memory). *Defaults to true for all primitive (non-analyzed) field types, such as int, float, data, bool, and string.* Only full-text fields or fields need norms. |*
+|`omitNorms` |If `true`, omits the norms associated with this field (this disables length normalization for the field, and saves some memory). *Defaults to true for all primitive (non-analyzed) field types, such as int, float, data, bool, and string.* Only full-text fields or ??? fields need norms. |*
Review Comment:
I believe I found the original phrase in other places in the codebase - e.g., in the techproducts schema https://github.com/apache/solr/blob/f0fcd300c896b858ae83235ecdb0a109eaea5cea/solr/server/solr/configsets/sample_techproducts_configs/conf/managed-schema.xml#L86-L87
##########
solr/solr-ref-guide/modules/indexing-guide/pages/schema-elements.adoc:
##########
@@ -95,18 +95,23 @@ However, `uniqueKey` will continue to work, as long as the field is properly use
Similarity is a Lucene class used to score a document in searching.
Each collection has one "global" Similarity.
-By default Solr uses an implicit {solr-javadocs}/core/org/apache/solr/search/similarities/SchemaSimilarityFactory.html[`SchemaSimilarityFactory`] which allows individual field types to be configured with a "per-type" specific Similarity and implicitly uses `BM25Similarity` for any field type which does not have an explicit Similarity.
+By default, Solr uses an implicit {solr-javadocs}/core/org/apache/solr/search/similarities/SchemaSimilarityFactory.html[`SchemaSimilarityFactory`] which allows individual field types to be configured with a "per-type" specific Similarity and implicitly uses `BM25Similarity` for any field type which does not have an explicit Similarity.
This default behavior can be overridden by declaring a top level `<similarity/>` element in your schema, outside of any single field type.
This similarity declaration can either refer directly to the name of a class with a no-argument constructor, such as in this example showing `BM25Similarity`:
[source,xml]
----
-<similarity class="solr.BM25SimilarityFactory"/>
+<similarity class="org.apache.lucene.search.similarities.BM25Similarity"/>
Review Comment:
Here, I just updated the xml snippets - so if the text says `in this example showing BM25Similarity`, the example indeed uses the similarity directly and not the factory. I tried writing this snippet as
```xml
<similarity class="solr.BM25Similarity">
```
at first - but it seems like if the similarity class is from Lucene, the `solr.` shorthand doesn't work.
And the original xml snippet (that was using the similarity factory) now appears after the text that talks about similarity factories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a diff in pull request #1961: Update indexing docs in Solr ref guide
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1340899863
##########
solr/solr-ref-guide/modules/indexing-guide/pages/charfilterfactories.adoc:
##########
Review Comment:
I went ahead and did the page rename...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a diff in pull request #1961: Update indexing docs in Solr ref guide
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1340878722
##########
solr/solr-ref-guide/modules/indexing-guide/pages/charfilterfactories.adoc:
##########
Review Comment:
I wonder if we should go ahead and rename the the from charfilteractories.adoc to charfilters.adoc?
##########
solr/solr-ref-guide/modules/indexing-guide/pages/schema-elements.adoc:
##########
@@ -95,18 +95,23 @@ However, `uniqueKey` will continue to work, as long as the field is properly use
Similarity is a Lucene class used to score a document in searching.
Each collection has one "global" Similarity.
-By default Solr uses an implicit {solr-javadocs}/core/org/apache/solr/search/similarities/SchemaSimilarityFactory.html[`SchemaSimilarityFactory`] which allows individual field types to be configured with a "per-type" specific Similarity and implicitly uses `BM25Similarity` for any field type which does not have an explicit Similarity.
+By default, Solr uses an implicit {solr-javadocs}/core/org/apache/solr/search/similarities/SchemaSimilarityFactory.html[`SchemaSimilarityFactory`] which allows individual field types to be configured with a "per-type" specific Similarity and implicitly uses `BM25Similarity` for any field type which does not have an explicit Similarity.
This default behavior can be overridden by declaring a top level `<similarity/>` element in your schema, outside of any single field type.
This similarity declaration can either refer directly to the name of a class with a no-argument constructor, such as in this example showing `BM25Similarity`:
[source,xml]
----
-<similarity class="solr.BM25SimilarityFactory"/>
+<similarity class="org.apache.lucene.search.similarities.BM25Similarity"/>
Review Comment:
Oh interesting... is this because lucene chnaged? Odd that it used to be in solr and now is in Lucene??
##########
solr/solr-ref-guide/modules/indexing-guide/pages/field-type-definitions-and-properties.adoc:
##########
@@ -201,7 +201,7 @@ The table below includes the default value for most `FieldType` implementations
|`sortMissingFirst`, `sortMissingLast` |Control the placement of documents when a sort field is not present. |`false`
|`multiValued` |If `true`, indicates that a single document might contain multiple values for this field type. |`false`
|`uninvertible` |If `true`, indicates that an `indexed="true" docValues="false"` field can be "un-inverted" at query time to build up large in memory data structure to serve in place of xref:docvalues.adoc[]. *Defaults to `true` for historical reasons, but users are strongly encouraged to set this to `false` for stability and use `docValues="true"` as needed.* |`true`
-|`omitNorms` |If `true`, omits the norms associated with this field (this disables length normalization for the field, and saves some memory). *Defaults to true for all primitive (non-analyzed) field types, such as int, float, data, bool, and string.* Only full-text fields or fields need norms. |*
+|`omitNorms` |If `true`, omits the norms associated with this field (this disables length normalization for the field, and saves some memory). *Defaults to true for all primitive (non-analyzed) field types, such as int, float, data, bool, and string.* Only full-text fields or ??? fields need norms. |*
Review Comment:
I wish I knew what went here too.. if nothing comes ot mind, lets just edit it to `full-text fields need norms.` ;-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] AndreyBozhko commented on a diff in pull request #1961: Update indexing docs in Solr ref guide
Posted by "AndreyBozhko (via GitHub)" <gi...@apache.org>.
AndreyBozhko commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1337824984
##########
solr/solr-ref-guide/modules/indexing-guide/pages/field-type-definitions-and-properties.adoc:
##########
@@ -201,7 +201,7 @@ The table below includes the default value for most `FieldType` implementations
|`sortMissingFirst`, `sortMissingLast` |Control the placement of documents when a sort field is not present. |`false`
|`multiValued` |If `true`, indicates that a single document might contain multiple values for this field type. |`false`
|`uninvertible` |If `true`, indicates that an `indexed="true" docValues="false"` field can be "un-inverted" at query time to build up large in memory data structure to serve in place of xref:docvalues.adoc[]. *Defaults to `true` for historical reasons, but users are strongly encouraged to set this to `false` for stability and use `docValues="true"` as needed.* |`true`
-|`omitNorms` |If `true`, omits the norms associated with this field (this disables length normalization for the field, and saves some memory). *Defaults to true for all primitive (non-analyzed) field types, such as int, float, data, bool, and string.* Only full-text fields or fields need norms. |*
+|`omitNorms` |If `true`, omits the norms associated with this field (this disables length normalization for the field, and saves some memory). *Defaults to true for all primitive (non-analyzed) field types, such as int, float, data, bool, and string.* Only full-text fields or ??? fields need norms. |*
Review Comment:
It seemed to me that a word was missing here, but I'm not sure what that word needs to be - appreciate the help here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] AndreyBozhko commented on a diff in pull request #1961: Update indexing docs in Solr ref guide
Posted by "AndreyBozhko (via GitHub)" <gi...@apache.org>.
AndreyBozhko commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1341570910
##########
solr/solr-ref-guide/modules/indexing-guide/pages/schema-elements.adoc:
##########
@@ -95,18 +95,23 @@ However, `uniqueKey` will continue to work, as long as the field is properly use
Similarity is a Lucene class used to score a document in searching.
Each collection has one "global" Similarity.
-By default Solr uses an implicit {solr-javadocs}/core/org/apache/solr/search/similarities/SchemaSimilarityFactory.html[`SchemaSimilarityFactory`] which allows individual field types to be configured with a "per-type" specific Similarity and implicitly uses `BM25Similarity` for any field type which does not have an explicit Similarity.
+By default, Solr uses an implicit {solr-javadocs}/core/org/apache/solr/search/similarities/SchemaSimilarityFactory.html[`SchemaSimilarityFactory`] which allows individual field types to be configured with a "per-type" specific Similarity and implicitly uses `BM25Similarity` for any field type which does not have an explicit Similarity.
This default behavior can be overridden by declaring a top level `<similarity/>` element in your schema, outside of any single field type.
This similarity declaration can either refer directly to the name of a class with a no-argument constructor, such as in this example showing `BM25Similarity`:
[source,xml]
----
-<similarity class="solr.BM25SimilarityFactory"/>
+<similarity class="org.apache.lucene.search.similarities.BM25Similarity"/>
Review Comment:
Here, I just updated the xml snippets - so if the text says `in this example showing BM25Similarity`, the example indeed uses the similarity directly and not the factory. I tried writing this snippet as
```xml
<similarity class="solr.BM25Similarity"/>
```
at first - but it seems like if the similarity class is from Lucene, the `solr.` shorthand doesn't work.
And the original xml snippet (that was using the similarity factory) now appears after the text that talks about similarity factories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Update indexing docs in Solr ref guide [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1344649962
##########
solr/solr-ref-guide/modules/indexing-guide/pages/tokenizers.adoc:
##########
@@ -772,11 +964,25 @@ The syntax is more limited than `PatternTokenizerFactory`, but the tokenization
*Arguments:*
-`pattern`: (Required) The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters that should split tokens.
+`pattern`::
++
+[%autowidth,frame=none]
+|===
+s|Required |Default: none
+|===
++
+The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters that should split tokens.
The matching is greedy such that the longest token separator matching at a given point is matched.
Empty tokens are never created.
-`maxDeterminizedStates`: (Optional, default 10000) the limit on total state count for the determined automaton computed from the regexp.
+`determinizeWorkLimit`::
Review Comment:
Just to confirm, did this variable change names? I don't see either in the source code of the Solr project, maybe because they are in Lucene?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] epugh commented on a diff in pull request #1961: Update indexing docs in Solr ref guide
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1340877351
##########
solr/solr-ref-guide/modules/indexing-guide/pages/analyzers.adoc:
##########
Review Comment:
I didn't actually really know about the shorter way of naming things... We should also look at the techproducts and other sample schema.xml and see how we name things...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Update indexing docs in Solr ref guide [solr]
Posted by "AndreyBozhko (via GitHub)" <gi...@apache.org>.
AndreyBozhko commented on code in PR #1961:
URL: https://github.com/apache/solr/pull/1961#discussion_r1344744097
##########
solr/solr-ref-guide/modules/indexing-guide/pages/tokenizers.adoc:
##########
@@ -772,11 +964,25 @@ The syntax is more limited than `PatternTokenizerFactory`, but the tokenization
*Arguments:*
-`pattern`: (Required) The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters that should split tokens.
+`pattern`::
++
+[%autowidth,frame=none]
+|===
+s|Required |Default: none
+|===
++
+The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters that should split tokens.
The matching is greedy such that the longest token separator matching at a given point is matched.
Empty tokens are never created.
-`maxDeterminizedStates`: (Optional, default 10000) the limit on total state count for the determined automaton computed from the regexp.
+`determinizeWorkLimit`::
Review Comment:
Correct, it seems that the name has changed to `determinizeWorkLimit` in Lucene 9.0 as part of [LUCENE-9981](https://issues.apache.org/jira/browse/LUCENE-9981).
https://github.com/apache/lucene/blob/75da33836b16e78e6c1cfcf76a049ba8a6600f1e/lucene/analysis/common/src/java/org/apache/lucene/analysis/pattern/SimplePatternTokenizerFactory.java#L37-L39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Update indexing docs in Solr ref guide [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #1961:
URL: https://github.com/apache/solr/pull/1961#issuecomment-1745849061
Thanks for this housekeeping!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Update indexing docs in Solr ref guide [solr]
Posted by "AndreyBozhko (via GitHub)" <gi...@apache.org>.
AndreyBozhko commented on PR #1961:
URL: https://github.com/apache/solr/pull/1961#issuecomment-1743605262
@epugh This should be ready now for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Update indexing docs in Solr ref guide [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #1961:
URL: https://github.com/apache/solr/pull/1961#issuecomment-1745623646
@AndreyBozhko I added one last question about parameter name, and it looks good. Nice sleuthing on the missing words ;-). Just check that one, and then we are ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] AndreyBozhko commented on pull request #1961: Update indexing docs in Solr ref guide
Posted by "AndreyBozhko (via GitHub)" <gi...@apache.org>.
AndreyBozhko commented on PR #1961:
URL: https://github.com/apache/solr/pull/1961#issuecomment-1736360372
cc @anshumg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org