You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "rmuir (via GitHub)" <gi...@apache.org> on 2024/03/29 00:38:31 UTC
[PR] upgrade icu4j to 74.2 [lucene]
rmuir opened a new pull request, #13239:
URL: https://github.com/apache/lucene/pull/13239
Current ICU version is ancient (70.1): Upgrade to the latest 74.2 release supporting Unicode 15.1
Practically, because `testEmojiFromTheFuture()` passes, we don't behave that out-of-date to end-users.
But the algorithms and code here get maintained, improved, etc, bugs get fixed, we should upgrade.
It would also be confusing for 10.0 to support older unicode version than the JDK itself.
* Regenerate all related files, I forced this with `--rerun-tasks` just to be sure. I had to bump groovy version so that regeneration works with java 21.
* Update RBBI grammar, i just replaced with the new one, disabled dictionary break/rules, and added comment on how to do this. All other changes except this were autogenerated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] upgrade icu4j to 74.2 [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #13239:
URL: https://github.com/apache/lucene/pull/13239#discussion_r1552071936
##########
lucene/analysis/common/src/test/org/apache/lucene/analysis/email/TLDs.txt:
##########
@@ -1,6 +1,5 @@
# Generated from IANA TLD Database (gradlew generateTlds).aaa
aarp
-abarth
Review Comment:
Looks only FIAT/Opel automotive top level Domains were removed. 😜
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] upgrade icu4j to 74.2 [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on code in PR #13239:
URL: https://github.com/apache/lucene/pull/13239#discussion_r1546667270
##########
lucene/analysis/icu/src/data/uax29/Default.rbbi:
##########
@@ -17,6 +17,12 @@
# This file is from ICU (with some small modifications, to avoid CJK dictionary break,
# and status code change related to that)
#
+# To update this file: grab rule file corresponding to your ICU version, e.g.:
+# https://github.com/unicode-org/icu/blob/release-74-2/icu4c/source/data/brkitr/rules/word.txt
+#
+# * Prevent dictionary break: disable final rule that would chain han and kana
+# * For kana rules, change from 400 to 300 status (since there's no dictionary break)
Review Comment:
This grammar is not used by default. This grammar is used only when the user overrides defaults to disable ICU's dictionary-based CJK segmentation:
```
@param cjkAsWords true if cjk text should undergo dictionary-based segmentation, otherwise text
will be segmented according to UAX#29 defaults. If this is true, all Han+Hiragana+Katakana
words will be tagged as IDEOGRAPHIC.
```
https://github.com/apache/lucene/blob/main/lucene/analysis/icu/src/java/org/apache/lucene/analysis/icu/segmentation/DefaultICUTokenizerConfig.java#L94C1-L96C46
e.g. the user can configure `false` here and use `CJKBigramFilter` if they prefer an n-gram approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] upgrade icu4j to 74.2 [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on code in PR #13239:
URL: https://github.com/apache/lucene/pull/13239#discussion_r1546413938
##########
lucene/analysis/common/src/test/org/apache/lucene/analysis/email/TLDs.txt:
##########
@@ -1,6 +1,5 @@
# Generated from IANA TLD Database (gradlew generateTlds).aaa
aarp
-abarth
Review Comment:
Curious that all these TLDs are deleted...
##########
lucene/analysis/icu/src/data/uax29/Default.rbbi:
##########
@@ -17,6 +17,12 @@
# This file is from ICU (with some small modifications, to avoid CJK dictionary break,
# and status code change related to that)
#
+# To update this file: grab rule file corresponding to your ICU version, e.g.:
+# https://github.com/unicode-org/icu/blob/release-74-2/icu4c/source/data/brkitr/rules/word.txt
+#
+# * Prevent dictionary break: disable final rule that would chain han and kana
+# * For kana rules, change from 400 to 300 status (since there's no dictionary break)
Review Comment:
Thank you for adding these instructions!
Where do we use this `Default.rbbi`? Is it for users to pass to `ICUTokenizer` to run (our slightly modified) UAX #29 tokenization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] upgrade icu4j to 74.2 [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir merged PR #13239:
URL: https://github.com/apache/lucene/pull/13239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] upgrade icu4j to 74.2 [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on code in PR #13239:
URL: https://github.com/apache/lucene/pull/13239#discussion_r1546668472
##########
lucene/analysis/common/src/test/org/apache/lucene/analysis/email/TLDs.txt:
##########
@@ -1,6 +1,5 @@
# Generated from IANA TLD Database (gradlew generateTlds).aaa
aarp
-abarth
Review Comment:
I just regenerate, it has not been run since September 2022, which is one reason we see many differences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org