You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "renatoh (via GitHub)" <gi...@apache.org> on 2026/09/16 18:16:10 UTC
[PR] Add type to compound word [lucene]
renatoh opened a new pull request, #16680:
URL: https://github.com/apache/lucene/pull/16680
Hi @uschindler
Is there a reason not to have a type on the compound words, so that I could inject a payload base on the type and make sure the extracted compound does get a lower boost than the original term?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5731452392
@uschindler something like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5865727303
@uschindler Thank you, since you backported it, it will be included in Lucene 10.6?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler merged PR #16680:
URL: https://github.com/apache/lucene/pull/16680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5715830475
I'm fine with `subword` as a type, really. But there is a limitation that you can't have a type of both `synonym` and `subword`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #16680:
URL: https://github.com/apache/lucene/pull/16680#discussion_r4050423818
##########
lucene/analysis/common/src/java/org/apache/lucene/analysis/compound/CompoundWordTokenFilterBase.java:
##########
@@ -25,9 +25,13 @@
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;
+import org.apache.lucene.analysis.tokenattributes.TypeAttribute;
/** Base class for decomposition token filters. */
public abstract class CompoundWordTokenFilterBase extends TokenFilter {
+ /** Type assigned to subword tokens generated by {@link #decompose()}. */
+ public static final String TOKEN_TYPE_COMPOUND = "subword";
Review Comment:
The constant should be named `TOKEN_TYPE_SUBWORD`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5712252294
Hi,
I think this is a good addition. I am not sure if the type name "compound" is correct, as this is more about the parts of the compound. The compound itsself inherits the type of the original term (e.g., from the tokenizer).
As a general question: Do we also add separate types to synonyms or other generated tokens at same position at the moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5715804091
I think both of you have good opinions. The problem of the type attribute is that it only allows one type. In current code of analyzers the original type is copied over.
As you see in the test, the tokenizer adds "word" as type (maybe different for other tokenizers, e.g. Japanese). The old code cloned all attributes, so the sub words also get "word" as type.
The good idea here is that additional (generated tokens) get a new type assigned, e.g "subword". That helps with generic filters added later like those for payloads. It is not ideal, but it helps to classify later token additions.
So in my opinion, this is fine with type attribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "mkhludnev (via GitHub)" <gi...@apache.org>.
mkhludnev commented on code in PR #16680:
URL: https://github.com/apache/lucene/pull/16680#discussion_r4053316740
##########
lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestCompoundWordTokenFilter.java:
##########
@@ -261,6 +261,34 @@ public void testDumbCompoundWordsSELongestMatch() throws Exception {
new int[] {1, 0, 0, 0, 0, 0});
}
+ public void testTypeOnExtractedWord() throws Exception {
+ CharArraySet dict = makeDictionary("Schlüssel", "Kasten");
+
+ DictionaryCompoundWordTokenFilter tf =
+ new DictionaryCompoundWordTokenFilter(
+ whitespaceMockTokenizer("Schlüsselkasten"),
+ dict,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE,
+ true);
+
+ String[] output = {"Schlüsselkasten", "Schlüssel", "kasten"};
+ assertTokenStreamContents(
+ tf,
+ output,
+ new int[] {0, 0, 0},
+ new int[] {15, 15, 15},
+ new String[] {
+ "word",
+ CompoundWordTokenFilterBase.TOKEN_TYPE_SUBWORD,
+ CompoundWordTokenFilterBase.TOKEN_TYPE_SUBWORD
Review Comment:
There's a conception to substituting constant to tests that let to detect a change in constant value which may break a client otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5865149289
@uschindler Is this good to be merged now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5715995555
> I'm fine with `subword` as a type, really. But there is a limitation that you can't have a type of both `synonym` and `subword`.
Yes, if you expand synonyms and decompound then this hits you. But your filter chain could handle that. After synonyms you can add a payload filter that would trigger on synonyms and then later add decompounding with another payload filter.
The whole token filter chain is problematic with that type of combinations. The whole type concept is feeling like Doug Cuttings original Token class. 😝
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5714248892
I think `TypeAttribute` is an antipattern for cases like this. You can only set single value for the type and so it is no better than the old Token days.
Maybe consider SubwordAttribute which wraps a boolean, or something like that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5727855515
Can you add a changes entry. I think Lucene 10.6 should be fine as it's easy to backport
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5865873886
Should be.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on code in PR #16680:
URL: https://github.com/apache/lucene/pull/16680#discussion_r4053385617
##########
lucene/analysis/common/src/test/org/apache/lucene/analysis/compound/TestCompoundWordTokenFilter.java:
##########
@@ -261,6 +261,34 @@ public void testDumbCompoundWordsSELongestMatch() throws Exception {
new int[] {1, 0, 0, 0, 0, 0});
}
+ public void testTypeOnExtractedWord() throws Exception {
+ CharArraySet dict = makeDictionary("Schlüssel", "Kasten");
+
+ DictionaryCompoundWordTokenFilter tf =
+ new DictionaryCompoundWordTokenFilter(
+ whitespaceMockTokenizer("Schlüsselkasten"),
+ dict,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_WORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MIN_SUBWORD_SIZE,
+ CompoundWordTokenFilterBase.DEFAULT_MAX_SUBWORD_SIZE,
+ true);
+
+ String[] output = {"Schlüsselkasten", "Schlüssel", "kasten"};
+ assertTokenStreamContents(
+ tf,
+ output,
+ new int[] {0, 0, 0},
+ new int[] {15, 15, 15},
+ new String[] {
+ "word",
+ CompoundWordTokenFilterBase.TOKEN_TYPE_SUBWORD,
+ CompoundWordTokenFilterBase.TOKEN_TYPE_SUBWORD
Review Comment:
good point, thanks for your input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5714839811
@rmuir you are suggesting to create a brand new boolean attribute? I see you point, but with a custom attribute we could not use a generic filter like NumericPayloadTokenFilter to inject a payload and as far as I understand, it would require custom code to do so.
At the end, I think we should have a way to boost/demote subwords, compare to the original token, same applies for synonyms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5865403975
Merged and backported! Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5734995898
@rmuir SynonymGraphFilter emits all the token, even the original one, as type:Synonym, this seems broken to me. Do you have an idea why this is like this? I have observed this in Lucene 9.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on code in PR #16680:
URL: https://github.com/apache/lucene/pull/16680#discussion_r4050417046
##########
lucene/CHANGES.txt:
##########
@@ -379,6 +379,10 @@ API Changes
New Features
---------------------
+* GITHUB#16680: Add a "subwords" type to the subword tokens generated by
Review Comment:
Not plural, otherwise fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add type to compound word [lucene]
Posted by "renatoh (via GitHub)" <gi...@apache.org>.
renatoh commented on PR #16680:
URL: https://github.com/apache/lucene/pull/16680#issuecomment-5713368846
I think you are right, compound is the entire thing, maybe something like 'subword', let me know if you come up with something better.
Yes, we do that for the synonyms, but it is kind of broken in the SynonymGraphFilter, since all the tokens its emits, even the original, have the type 'synonyms', so it is kind of useless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org