You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Han Hui Wen (Created) (JIRA)" <ji...@apache.org> on 2011/09/29 10:54:45 UTC
[jira] [Created] (SOLR-2800) RemoveDuplicatesTokenFilterFactory can
not remove the duplicated term
RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
---------------------------------------------------------------------
Key: SOLR-2800
URL: https://issues.apache.org/jira/browse/SOLR-2800
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Affects Versions: 3.4
Environment: Windows
Reporter: Han Hui Wen
Fix For: 3.5
Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
@Override
53 public boolean incrementToken() throws IOException {
54 while (input.incrementToken()) {
55 final char term[] = termAttribute.buffer();
56 final int length = termAttribute.length();
57 final int posIncrement = posIncAttribute.getPositionIncrement();
58
59 if (posIncrement > 0) {
60 previous.clear();
61 }
62
63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
64
65 // clone the term, and add to the set of seen terms.
66 char saved[] = new char[length];
67 System.arraycopy(term, 0, saved, 0, length);
68 previous.add(saved);
69
70 if (!duplicate) {
71 return true;
72 }
73 }
74 return false;
75 }
it should be like following:
@Override
public boolean incrementToken() throws IOException {
while (input.incrementToken()) {
final char term[] = termAttribute.buffer();
final int length = termAttribute.length();
final int posIncrement = posIncAttribute.getPositionIncrement();
if (posIncrement > 0) {
previous.clear();
}
boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
if(duplicate )
{
return false;
}
else
{
// clone the term, and add to the set of seen terms.
char saved[] = new char[length];
System.arraycopy(term, 0, saved, 0, length);
previous.add(saved);
}
}
return true;
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2800) RemoveDuplicatesTokenFilterFactory
can not remove the duplicated term
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117317#comment-13117317 ]
Robert Muir commented on SOLR-2800:
-----------------------------------
{quote}
Your version seems more efficient, since duplicates are not placed in the CharArraySet, but the functionality looks the same to me as the original.
{quote}
Well if you have Joe (Posinc=1), Pizza (posinc=0), Joe (posinc=0), Beer (posinc=0), Pizza (posinc=0), will you end up with 1 Joe, 1 Pizza, and 1 Beer (This is how the filter is defined)?
Thats why the set is needed I think.
> RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
> ---------------------------------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
> Fix For: 3.5
>
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2800) RemoveDuplicatesTokenFilterFactory
can not remove the duplicated term
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117321#comment-13117321 ]
Steven Rowe commented on SOLR-2800:
-----------------------------------
I agree the set is needed, but the current code clones and then adds every single term to the set, regardless of whether it's already in the set.
> RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
> ---------------------------------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
> Fix For: 3.5
>
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2800) optimize
RemoveDuplicatesTokenFilterFactory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated SOLR-2800:
------------------------------
Attachment: SOLR-2800.patch
here's the optimization. basically it saves a clone etc only for additional posinc=0 duplicates that we aren't going to return anyway.
> optimize RemoveDuplicatesTokenFilterFactory
> -------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Assignee: Robert Muir
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
> Attachments: SOLR-2800.patch
>
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2800) optimize
RemoveDuplicatesTokenFilterFactory
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated SOLR-2800:
------------------------------
Issue Type: Improvement (was: Bug)
Summary: optimize RemoveDuplicatesTokenFilterFactory (was: RemoveDuplicatesTokenFilterFactory can not remove the duplicated term)
> optimize RemoveDuplicatesTokenFilterFactory
> -------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Assignee: Robert Muir
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2800) RemoveDuplicatesTokenFilterFactory
can not remove the duplicated term
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117372#comment-13117372 ]
Steven Rowe commented on SOLR-2800:
-----------------------------------
Wen, your changes introduce a bug: {{incrementToken()}} should return false only at end-of-stream.
That is, instead of:
{noformat}
if(duplicate )
{ return false; }
{noformat}
you should use:
{noformat}
if (duplicate) { continue; }
{noformat}
> RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
> ---------------------------------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
> Fix For: 3.5
>
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2800) RemoveDuplicatesTokenFilterFactory
can not remove the duplicated term
Posted by "Steven Rowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117312#comment-13117312 ]
Steven Rowe commented on SOLR-2800:
-----------------------------------
Can you provide a test case? Your version seems more efficient, since duplicates are not placed in the CharArraySet, but the functionality looks the same to me as the original. Your test case should demonstrate what is failing without your changes, and that your changes fix the problem.
Please provide code/tests in the form of a patch - it's much easier to see what you have changed/added. Also, it's very important when you provide code that you attach a patch to the issue and click where it says "Grant license to ASF for inclusion in ASF works (as per the Apache License ยง5)" - unless you do this, we can't use your code.
> RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
> ---------------------------------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
> Fix For: 3.5
>
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (SOLR-2800) RemoveDuplicatesTokenFilterFactory
can not remove the duplicated term
Posted by "Robert Muir (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117329#comment-13117329 ]
Robert Muir commented on SOLR-2800:
-----------------------------------
Yeah thats stupid, we only need to clone the term if it *doesnt exist*... on the other hand its not too bad since this is the 'common case', but we should fix it.
> RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
> ---------------------------------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
> Fix For: 3.5
>
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (SOLR-2800) RemoveDuplicatesTokenFilterFactory can
not remove the duplicated term
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hoss Man updated SOLR-2800:
---------------------------
Fix Version/s: (was: 4.0)
Assignee: Robert Muir
removing fixVersion=4.0 since there is no evidence that anyone is currently working on this issue.
But also assigning to [~rcmuir] since if i'm understanding his comments, it seems he thinks there is an easy win here, we just need a test case.
> RemoveDuplicatesTokenFilterFactory can not remove the duplicated term
> ---------------------------------------------------------------------
>
> Key: SOLR-2800
> URL: https://issues.apache.org/jira/browse/SOLR-2800
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.4
> Environment: Windows
> Reporter: Han Hui Wen
> Assignee: Robert Muir
> Labels: RemoveDuplicatesTokenFilterFactory, Solr
>
> Using RemoveDuplicatesTokenFilterFactory can not remove the duplicated term.
> in http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_3_4/solr/core/src/java/org/apache/solr/analysis/RemoveDuplicatesTokenFilter.java?view=markup
> @Override
> 53 public boolean incrementToken() throws IOException {
> 54 while (input.incrementToken()) {
> 55 final char term[] = termAttribute.buffer();
> 56 final int length = termAttribute.length();
> 57 final int posIncrement = posIncAttribute.getPositionIncrement();
> 58
> 59 if (posIncrement > 0) {
> 60 previous.clear();
> 61 }
> 62
> 63 boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
> 64
> 65 // clone the term, and add to the set of seen terms.
> 66 char saved[] = new char[length];
> 67 System.arraycopy(term, 0, saved, 0, length);
> 68 previous.add(saved);
> 69
> 70 if (!duplicate) {
> 71 return true;
> 72 }
> 73 }
> 74 return false;
> 75 }
> it should be like following:
> @Override
> public boolean incrementToken() throws IOException {
> while (input.incrementToken()) {
> final char term[] = termAttribute.buffer();
> final int length = termAttribute.length();
> final int posIncrement = posIncAttribute.getPositionIncrement();
> if (posIncrement > 0) {
> previous.clear();
> }
> boolean duplicate = (posIncrement == 0 && previous.contains(term, 0, length));
>
> if(duplicate )
> {
> return false;
> }
> else
> {
> // clone the term, and add to the set of seen terms.
> char saved[] = new char[length];
> System.arraycopy(term, 0, saved, 0, length);
> previous.add(saved);
> }
> }
> return true;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org