You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Johann Höchtl (JIRA)" <ji...@apache.org> on 2011/04/12 11:16:05 UTC
[jira] [Created] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
---------------------------------------------------------------------
Key: LUCENE-3022
URL: https://issues.apache.org/jira/browse/LUCENE-3022
Project: Lucene - Java
Issue Type: Bug
Components: contrib/analyzers
Affects Versions: 3.1, 2.9.4
Reporter: Johann Höchtl
Priority: Minor
When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
So I reviewed the sourcecode and found the problem:
[code]
protected void decomposeInternal(final Token token) {
// Only words longer than minWordSize get processed
if (token.length() < this.minWordSize) {
return;
}
char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
for (int i=0;i<token.length()-this.minSubwordSize;++i) {
Token longestMatchToken=null;
for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
if(i+j>token.length()) {
break;
}
if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
if (this.onlyLongestMatch) {
if (longestMatchToken!=null) {
if (longestMatchToken.length()<j) {
longestMatchToken=createToken(i,j,token);
}
} else {
longestMatchToken=createToken(i,j,token);
}
} else {
tokens.add(createToken(i,j,token));
}
}
}
if (this.onlyLongestMatch && longestMatchToken!=null) {
tokens.add(longestMatchToken);
}
}
}
[/code]
should be changed to
[code]
protected void decomposeInternal(final Token token) {
// Only words longer than minWordSize get processed
if (token.termLength() < this.minWordSize) {
return;
}
char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
Token longestMatchToken=null;
for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
if(i+j>token.termLength()) {
break;
}
if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
if (this.onlyLongestMatch) {
if (longestMatchToken!=null) {
if (longestMatchToken.termLength()<j) {
longestMatchToken=createToken(i,j,token);
}
} else {
longestMatchToken=createToken(i,j,token);
}
} else {
tokens.add(createToken(i,j,token));
}
}
}
}
if (this.onlyLongestMatch && longestMatchToken!=null) {
tokens.add(longestMatchToken);
}
}
[/code]
So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Johann Höchtl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022351#comment-13022351 ]
Johann Höchtl commented on LUCENE-3022:
---------------------------------------
Let's stay with this example:
Dict: {"soft","so","ft"}
Word: "softball"
The first option onlyLongestMatch, which behaves in the way, that only the longest matching dictionary entry should be returned. (Should this option be modified to keep all of the longest matches? length(soft) == length(ball)?)
Output: "soft" if true; "so","ft","soft" if false
The second option should be keepRemain, which makes a term out of the remain after substracting the longestMatch (makes only sense with onlyLongestMatch!?)
Output: "soft","ball" if keepRemain==onlyLongestMatch==true
With this second option you could keep the remains, which are not in your dictionary (reduces the complexity of the required dictionary and can improve the compound-splitting-logic)
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.2, 4.0
>
> Attachments: LUCENE-3022.patch, LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Johann Höchtl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johann Höchtl updated LUCENE-3022:
----------------------------------
Attachment: LUCENE-3022.patch
Patch fixing this issue
including JUnitTest
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Priority: Minor
> Attachments: LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3022:
--------------------------------
Fix Version/s: 4.0
3.2
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.2, 4.0
>
> Attachments: LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022232#comment-13022232 ]
Robert Muir commented on LUCENE-3022:
-------------------------------------
Hi, it seems we need to figure out some things about this compound filter:
What is the onlyLongestMatch option intended to do? Its description is a bit vague, is it:
# Supposed to only extract the longest-matching subword dictionary matches?
# Supposed to only extract the longest subword from a compound?
As an example, consider a dictionary containing: so, ft, soft, ball
How should "softball" be decomposed based on this option?
If onlyLongestMatch is interpreted to mean only the longest-matching subwords, then i think it should emit "softball, soft, ball". "so" and "ft" would not be emitted, because you set this option.
However, if its interpreted to mean only the longest matching subword from the compound (like this patch), then it should only emit "softball, soft" and stop.
||onlyLongestMatch||trunk||LUCENE-3022||LUCENE-3038||
|false|softball, so, soft, ft, ball|softball, so, soft, ft, ball|softball, so, soft, ft, ball|
|true|softball, soft, ft, ball|softball, soft|softball, soft, ft, ball|
Its definitely clear that the current behavior is wrong, but what should the correct behavior be? Maybe we need two separate options here?
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.2, 4.0
>
> Attachments: LUCENE-3022.patch, LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3022:
---------------------------------------
Fix Version/s: (was: 3.4)
3.5
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.5, 4.0
>
> Attachments: LUCENE-3022.patch, LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Assigned] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir reassigned LUCENE-3022:
-----------------------------------
Assignee: Robert Muir
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Attachments: LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3022:
--------------------------------
Attachment: LUCENE-3022.patch
Hi Johann, in my opinion your patch is completely correct, thanks for fixing this.
I noticed though, that a solr test failed because its factory defaults to this value being "on" (and the previous behavior was broken!!!)
Because of this, I propose we default this behavior to "off" in the Solr factory and add an upgrading note. Previously decompounding in solr defaulted to buggy behavior, but I think by default we should index all compound components (since that seems to be what the desired intended behavior was, which mostly worked, only because of the bug!)
I'll leave the issue open for a few days to see if anyone objects to this plan.
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 3.2, 4.0
>
> Attachments: LUCENE-3022.patch, LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019637#comment-13019637 ]
Robert Muir commented on LUCENE-3022:
-------------------------------------
This sounds like a bug, do you want to try your hand at contributing a patch?
See http://wiki.apache.org/lucene-java/HowToContribute for some instructions.
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Priority: Minor
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3022) DictionaryCompoundWordTokenFilter
Flag onlyLongestMatch has no affect
Posted by "Robert Muir (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-3022:
--------------------------------
Fix Version/s: (was: 3.6)
> DictionaryCompoundWordTokenFilter Flag onlyLongestMatch has no affect
> ---------------------------------------------------------------------
>
> Key: LUCENE-3022
> URL: https://issues.apache.org/jira/browse/LUCENE-3022
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 2.9.4, 3.1
> Reporter: Johann Höchtl
> Assignee: Robert Muir
> Priority: Minor
> Fix For: 4.0
>
> Attachments: LUCENE-3022.patch, LUCENE-3022.patch
>
> Original Estimate: 5m
> Remaining Estimate: 5m
>
> When using the DictionaryCompoundWordTokenFilter with a german dictionary, I got a strange behaviour:
> The german word "streifenbluse" (blouse with stripes) was decompounded to "streifen" (stripe),"reifen"(tire) which makes no sense at all.
> I thought the flag onlyLongestMatch would fix this, because "streifen" is longer than "reifen", but it had no effect.
> So I reviewed the sourcecode and found the problem:
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.length() < this.minWordSize) {
> return;
> }
>
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.buffer());
>
> for (int i=0;i<token.length()-this.minSubwordSize;++i) {
> Token longestMatchToken=null;
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.length()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.length()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> }
> [/code]
> should be changed to
> [code]
> protected void decomposeInternal(final Token token) {
> // Only words longer than minWordSize get processed
> if (token.termLength() < this.minWordSize) {
> return;
> }
> char[] lowerCaseTermBuffer=makeLowerCaseCopy(token.termBuffer());
> Token longestMatchToken=null;
> for (int i=0;i<token.termLength()-this.minSubwordSize;++i) {
> for (int j=this.minSubwordSize-1;j<this.maxSubwordSize;++j) {
> if(i+j>token.termLength()) {
> break;
> }
> if(dictionary.contains(lowerCaseTermBuffer, i, j)) {
> if (this.onlyLongestMatch) {
> if (longestMatchToken!=null) {
> if (longestMatchToken.termLength()<j) {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> longestMatchToken=createToken(i,j,token);
> }
> } else {
> tokens.add(createToken(i,j,token));
> }
> }
> }
> }
> if (this.onlyLongestMatch && longestMatchToken!=null) {
> tokens.add(longestMatchToken);
> }
> }
> [/code]
> So, that only the longest token is really indexed and the onlyLongestMatch Flag makes sense.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org