You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "janhoy (via GitHub)" <gi...@apache.org> on 2026/04/23 11:02:18 UTC
[PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
janhoy opened a new pull request, #4326:
URL: https://github.com/apache/solr/pull/4326
Replace com.cybozu.labs:langdetect (abandoned since 2012) with io.github.azagniotov:language-detection:12.5.2, a maintained fork with an active release history.
The new library bundles its own language profiles, so the 53 profile files previously shipped in the langid module resources are removed. The factory no longer loads profiles at startup; it creates a shared LanguageDetectionOrchestrator instead. The processor converts the field-content Reader to a String and calls orchestrator.detectAll().
`commons-io` was only used for profile loading and is also removed from the langid module dependencies.
Some tests needed reworking to pass due to different behavior of the libraries, and that this new supports more languages, which introduces some ambiguity.
https://issues.apache.org/jira/browse/SOLR-18208
Implemented entirely by Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #4326:
URL: https://github.com/apache/solr/pull/4326#discussion_r3130493457
##########
solr/modules/langid/src/test/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactoryTest.java:
##########
@@ -22,11 +22,15 @@
public class LangDetectLanguageIdentifierUpdateProcessorFactoryTest
extends LanguageIdentifierUpdateProcessorFactoryTestCase {
+
@Override
protected LanguageIdentifierUpdateProcessor createLangIdProcessor(ModifiableSolrParams parameters)
throws Exception {
return new LangDetectLanguageIdentifierUpdateProcessor(
- _parser.buildRequestFrom(h.getCore(), parameters, null), resp, null);
+ _parser.buildRequestFrom(h.getCore(), parameters, null),
Review Comment:
I always disliked the `_` pattern, maybe we can finally eradicate it here? Totally onboard if that is "out of scope".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy merged PR #4326:
URL: https://github.com/apache/solr/pull/4326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on PR #4326:
URL: https://github.com/apache/solr/pull/4326#issuecomment-4333879230
> I think my one request is taht some of the javadcos make sense in explaining to a reader teh changes you are mkaing, but once merged, are confusing.
Thanks, that was eager LLM documentation going on. Fixed it, will merge soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #4326:
URL: https://github.com/apache/solr/pull/4326#discussion_r3130348511
##########
solr/modules/langid/src/java/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactory.java:
##########
@@ -72,12 +66,9 @@ public void inform(SolrCore core) {}
*/
@Override
public void init(NamedList<?> args) {
- try {
- loadData();
- } catch (Exception e) {
- throw new RuntimeException(
- "Couldn't load profile data, will return empty languages always!", e);
- }
+ LanguageDetectionSettings settings = LanguageDetectionSettings.fromAllIsoCodes639_1().build();
+ orchestrator = LanguageDetectionOrchestrator.fromSettings(settings);
+
Review Comment:
The previous implementation loaded language profiles once per JVM (static loaded flag) because DetectorFactory was global. This change instantiates a new LanguageDetectionOrchestrator every time the factory is initialized, which can increase core startup time and memory usage when multiple cores use this processor. Consider switching to a static/lazy singleton orchestrator (or other shared cache) so profiles/settings are built once and reused across factory instances (while still honoring any per-core configuration if added later).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #4326:
URL: https://github.com/apache/solr/pull/4326#discussion_r3130503973
##########
solr/modules/langid/src/test/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactoryTest.java:
##########
@@ -37,8 +41,22 @@ protected SolrInputDocument tooShortDoc() {
return doc;
}
- /* we don't return 'un' for the super-short one (this detector things hungarian?).
- * replace this with japanese
+ /**
+ * Override the base class test with LangDetect-specific text samples.
+ *
+ * <p>Two differences from the base class:
+ *
+ * <ul>
+ * <li>The "too short" test case is replaced with Japanese (this detector returns Hungarian for
+ * short ambiguous text rather than "un").
+ * <li>The Russian text is replaced with a cleaner Cyrillic-only sample. The base class uses a
Review Comment:
Do we need all this commenting? It's absolutely wonderful while I am reading the PR, but once committed, it will be confusing. So maybe remove before merge?
Someday we can have comments in the code that live just for the branch.. and comments that live permenently ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] SOLR-18208: Replace abandoned langdetect dependency with maintained fork [solr]
Posted by "janhoy (via GitHub)" <gi...@apache.org>.
janhoy commented on code in PR #4326:
URL: https://github.com/apache/solr/pull/4326#discussion_r3134023841
##########
solr/modules/langid/src/test/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactoryTest.java:
##########
@@ -37,8 +41,22 @@ protected SolrInputDocument tooShortDoc() {
return doc;
}
- /* we don't return 'un' for the super-short one (this detector things hungarian?).
- * replace this with japanese
+ /**
+ * Override the base class test with LangDetect-specific text samples.
+ *
+ * <p>Two differences from the base class:
+ *
+ * <ul>
+ * <li>The "too short" test case is replaced with Japanese (this detector returns Hungarian for
+ * short ambiguous text rather than "un").
+ * <li>The Russian text is replaced with a cleaner Cyrillic-only sample. The base class uses a
Review Comment:
No we don’t. It belongs more in a PR comment I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org