You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/01/18 18:24:23 UTC
[GitHub] [lucene-solr] donnerpeter opened a new pull request #2217: LUCENE-9676: Hunspell: improve stemming of all-caps words
donnerpeter opened a new pull request #2217:
URL: https://github.com/apache/lucene-solr/pull/2217
<!--
_(If you are a project committer then you may remove some/all of the following template.)_
Before creating a pull request, please file an issue in the ASF Jira system for Lucene or Solr:
* https://issues.apache.org/jira/projects/LUCENE
* https://issues.apache.org/jira/projects/SOLR
You will need to create an account in Jira in order to create an issue.
The title of the PR should reference the Jira issue number in the form:
* LUCENE-####: <short description of problem or changes>
* SOLR-####: <short description of problem or changes>
LUCENE and SOLR must be fully capitalized. A short description helps people scanning pull requests for items they can work on.
Properly referencing the issue in the title ensures that Jira is correctly updated with code review comments and commits. -->
# Description
Currently words like "OPENOFFICE.ORG" result in no stems even if the dictionary contains "OpenOffice.org"
# Solution
Repeat Hunspell's logic:
* when encountering a mixed- or (inflectable) all-case dictionary entry, add its title-case analog as a hidden entry
* use that hidden entry for stemming case variants for title- and uppercase words, but don't consider it a valid word itself
* ...unless there's another explicit dictionary entry of that title case
# Tests
Adapted `allcaps` from Hunspell C++ repository, corrected existing `TestEscaped` to match Hunspell's behavior.
# Checklist
Please review the following and check all that apply:
- [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request title.
- [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended)
- [ ] I have developed this patch against the `master` branch.
- [ ] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] donnerpeter commented on pull request #2217: LUCENE-9676: Hunspell: improve stemming of all-caps words
Posted by GitBox <gi...@apache.org>.
donnerpeter commented on pull request #2217:
URL: https://github.com/apache/lucene-solr/pull/2217#issuecomment-762432969
It might be easier to review commits separately
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss commented on a change in pull request #2217: LUCENE-9676: Hunspell: improve stemming of all-caps words
Posted by GitBox <gi...@apache.org>.
dweiss commented on a change in pull request #2217:
URL: https://github.com/apache/lucene-solr/pull/2217#discussion_r559806181
##########
File path: lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/Dictionary.java
##########
@@ -74,6 +74,8 @@
static final char[] NOFLAGS = new char[0];
+ private static final char HIDDEN_FLAG = (char) 65511; // called 'ONLYUPCASEFLAG' in Hunspell
Review comment:
I think you could use an explicit char here? '\uFFE7'? Not sure though because this isn't valid unicode so some validation tools may complain later on... Let's leave it.
##########
File path: lucene/analysis/common/src/java/org/apache/lucene/analysis/hunspell/WordCase.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.analysis.hunspell;
+
+enum WordCase {
+ UPPER,
+ TITLE,
+ LOWER,
+ MIXED;
+
+ static WordCase caseOf(char[] word, int length) {
+ boolean capitalized = Character.isUpperCase(word[0]);
+
+ boolean seenUpper = false;
+ boolean seenLower = false;
+ for (int i = 1; i < length; i++) {
+ char ch = word[i];
+ seenUpper = seenUpper || Character.isUpperCase(ch);
+ seenLower = seenLower || Character.isLowerCase(ch);
+ }
+
+ return get(capitalized, seenUpper, seenLower);
+ }
+
+ static WordCase caseOf(CharSequence word, int length) {
+ boolean capitalized = Character.isUpperCase(word.charAt(0));
+
+ boolean seenUpper = false;
+ boolean seenLower = false;
+ for (int i = 1; i < length; i++) {
Review comment:
don't know if this makes much sense to optimize but you could break the loop too if (seenLower || seenUpper) as checking further on doesn't make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] dweiss merged pull request #2217: LUCENE-9676: Hunspell: improve stemming of all-caps words
Posted by GitBox <gi...@apache.org>.
dweiss merged pull request #2217:
URL: https://github.com/apache/lucene-solr/pull/2217
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org