You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "eusousu (via GitHub)" <gi...@apache.org> on 2024/12/17 13:29:59 UTC
[PR] Update stopwords.txt [lucene]
eusousu opened a new pull request, #14075:
URL: https://github.com/apache/lucene/pull/14075
### Description
In brazillian portuguese the conjuntion "em(preposition)+<a|o|as|os>(article)" take the form "na, nas, no, nos" being commom stop words.
For some reason the "nas" conjunction appear twice and the "no" is nowhere to be found, I think it was probably a mistake.
This pull request add the word to the list and remove the duplication.
For reference the same word can be found on the Portuguese [stopwords.txt](https://github.com/apache/lucene/blob/0203815/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/portuguese_stop.txt) on line 34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549147744
Yes, that's it. the `REP 3619` should be changed to `REP 3621`. I guess we could send the PR to libreoffice, since the "upstream" dictionary looks totally different here: https://github.com/bataak/dict-mn/blob/main/mn_MN/mn_MN.aff#L49
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549192152
https://github.com/LibreOffice/dictionaries/pull/46
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2548911336
> I am unable to understand how my change could impact the analysis on the Mongolian language.
I don't understand the connection to your change either, but it looks to me as if Mongolian is coming in via the randomized test framework, which sets the active Java Locale to something random. If you were to capture the full "reproduce with" command line that is output by the test framework when there is a failure, we could tell if that's what's going on. Are you able to reproduce the failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2548996704
I reran that check to see what happens, if it reproduces.
This "extra regressions" check is also doing unpinned shallow `git clone` of external dictionaries repositories, so they could make a commit to a dictionary in those repos that causes a failure (in theory).
I guess it is also possible it could fail (e.g. network issue) in such a way that the clone is unsuccessful? It is unclear to me if it is really cloning with the git binary, or if it is doing some pure-java (jgit etc) clone that might be less robust.
And yeah, it could also be a randomization issue such as locale/charset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549010607
And yeah i see the commit date, but that's not the push date. So I suspect this issue has nothing to do with your PR and may fail all PRs until we address it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2548624976
While its a simple change, it does change the analysis chain. I wonder if it should stick to Lucene 11 (admittedly, that will not be shipped for a LONG time).
I wonder what others think on if we should backport to 10.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "eusousu (via GitHub)" <gi...@apache.org>.
eusousu commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549438980
> https://github.com/LibreOffice/dictionaries/pull/46
Their contribution process seem to be elsewhere, but I could not understand it fully 😅
I tried sending the issue to their IRC channel, hope someone sees it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2551213164
See https://github.com/LibreOffice/dictionaries/commit/c3ff53711dcac4bdec24f23a2c1f9712a0833b67
Sorry for all the unrelated noise here on your PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549163081
I will send them a one-liner PR explaining the situation, we can take it from there.
We may want to separately try to be more lenient about this part of the parsing. Have not looked at the C implementation, but I'm guessing the numbers might be used as more of a "hint" to size hashtables or something, and not really a hard rule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549123863
I'm happy to try to debug this but it might be a few days. Issue may be with REP rules in the referenced commit.
the way these rules work are:
```
REP 3619
REP a а
REP c с
... (3617 more times)
```
if the count (3619) is incorrect, then parser might try to parse a rule (such as `REP ээв$ ээ_вэ`) as a another count, leading to the NumberFormatException. maybe the C hunspell parser is lenient, so nobody has yet noticed. In my experiences the dictionaries often have such problems, hence the testing here.
It could also be some other bug in the code, this is just what it looks like to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549005107
It fails again. maybe problem comes from https://github.com/LibreOffice/dictionaries/commit/d1696029d8923ae697cb2d6d4d7d69791b1943f2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "eusousu (via GitHub)" <gi...@apache.org>.
eusousu commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2549064565
> If you were to capture the full "reproduce with" command line that is output by the test framework
You mean this?
```
Reproduce with: gradlew :lucene:analysis:common:test --tests "org.apache.lucene.analysis.hunspell.TestAllDictionaries.testDictionariesLoadSuccessfully" -Ptests.jvms=2 -Ptests.jvmargs= -Ptests.seed=18B87D355DA5CF0F -Ptests.gui=true -Ptests.file.encoding=ISO-8859-1 -Ptests.vectorsize=default -Ptests.forceintegervectors=false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2550380846
I took a stab at hacking around this on our side as well: https://github.com/apache/lucene/pull/14079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "eusousu (via GitHub)" <gi...@apache.org>.
eusousu commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2552417520
@rmuir sorry I was unaware of this requirement. Added the entry, I really hope I got the pattern right 😅
I really appreciate your work and patience, thank you 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2550335515
Thank you @eusousu
I tried to make some progress, for now at least I have an open bug report: https://bugs.documentfoundation.org/show_bug.cgi?id=164366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2551434229
@eusousu I sent the email. I will also try to chase down the issue in the upstream bataak repo later, but it should not be an issue anymore. Thanks for the help and patience!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "eusousu (via GitHub)" <gi...@apache.org>.
eusousu commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2551221786
> I have an open bug report
@rmuir they asked me about a license statement on the IRC chat that I think is necessary for them to accept your contribution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2552131795
@eusousu if you want, add entry to lucene/CHANGES.txt for your fix. Otherwise I'm happy to do it for you, you've done plenty to get this fix in, I want to merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "eusousu (via GitHub)" <gi...@apache.org>.
eusousu commented on PR #14075:
URL: https://github.com/apache/lucene/pull/14075#issuecomment-2548665777
Is there something I can investigate further? I tried parsing the error and it seems to relate to a mn_MN dictionary that refers to Mongolian 🤔
```
While checking /home/runner/work/lucene/lucene/lucene/analysis/common/build/hunspell-regressions/libreoffice/mn_MN/mn_MN.aff:
2> java.lang.NumberFormatException: For input string: "ээв$"
2> at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
2> at java.base/java.lang.Integer.parseInt(Integer.java:662)
2> at java.base/java.lang.Integer.parseInt(Integer.java:778)
2> at org.apache.lucene.analysis.hunspell.Dictionary.parseNum(Dictionary.java:594)
2> at org.apache.lucene.analysis.hunspell.Dictionary.readAffixFile(Dictionary.java:408)
2> at org.apache.lucene.analysis.hunspell.Dictionary.<init>(Dictionary.java:264)
2> at org.apache.lucene.analysis.hunspell.TestAllDictionaries$1.<init>(TestAllDictionaries.java:75)
2> at org.apache.lucene.analysis.hunspell.TestAllDictionaries.loadDictionary(TestAllDictionaries.java:75)
2> at org.apache.lucene.analysis.hunspell.TestAllDictionaries.lambda$testDictionariesLoadSuccessfully$7(TestAllDictionaries.java:165)
2> at org.apache.lucene.analysis.hunspell.TestAllDictionaries.lambda$testDictionariesLoadSuccessfully$8(TestAllDictionaries.java:187)
2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2> at java.base/java.lang.Thread.run(Thread.java:1583)
```
I am unable to understand how my change could impact the analysis on the Mongolian language.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Update stopwords.txt [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir merged PR #14075:
URL: https://github.com/apache/lucene/pull/14075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org