You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org> on 2006/05/02 01:28:46 UTC
[jira] Created: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Add the ability to preserve the original term when using WordDelimiterFilter
----------------------------------------------------------------------------
Key: SOLR-14
URL: http://issues.apache.org/jira/browse/SOLR-14
Project: Solr
Type: Improvement
Components: search
Reporter: Richard "Trey" Hyde
Attachments: WordDelimiterFilter.patch
When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
Data: ABC-12345
WordDelimiterFitler may change this into
ABC 12345 ABC12345
A user may enter a search such as
ABC\-123*
Which will fail to find a match given the above scenario.
The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
ABC 12345 ABC12345 ABC-12345
in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Geoffrey Young (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Geoffrey Young updated SOLR-14:
-------------------------------
Attachment: SOLR-14.patch
this new patch addresses three additional cases
o words prefixed with delimiters
o words postfixed with delimiters
o words that are all delimiters
there's a special place for people who name themselves !!! and expect to be found.
the input string
{noformat}
test 404-123 $foo bar& beer !!! *foo baz's biff
{noformat}
produces
{noformat}
test 404-123 404 123 404123 foo $foo bar bar& beer !!! foo *foo baz baz's biff
{noformat}
using options
{noformat}
org.apache.solr.analysis.WordDelimiterFilterFactory {preserveOriginal=1, generateNumberParts=1, catenateWords=1, generateWordParts=1, catenateAll=1, catenateNumbers=1}
{noformat}
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Geoffrey Young (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610229#action_12610229 ]
Geoffrey Young commented on SOLR-14:
------------------------------------
looks good from a functional pov. the ordering of the tokens looks funny (to me) in analysis.jsp, but all the right ones are there.
it still takes my full load roughly twice as long to index (3.5 minutes versus 1.75 minutes for a 120MB file) but the functionality is important enough to incur the cost.
thanks muchly.
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Assignee: Yonik Seeley
> Attachments: SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Geoffrey Young (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605063#action_12605063 ]
Geoffrey Young commented on SOLR-14:
------------------------------------
this is fairly important for our ongoing implementation - can someone with appropriate karma take steps to get this into 1.3?
thanks
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Ankur Madnani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ankur Madnani updated SOLR-14:
------------------------------
Attachment: SOLR-14.patch
The junit test for preserveOriginal=1 is added to this patch.
This patch contains the new schema.xml for test and changed TestWordDelimiterFilter.
The patch given earlier by Geoff. changes the signature of the public constructors in the WordDelimiterFilter by adding preserveOriginal as parameter.
May be we should create separate constructor which accepts preserveOriginal, to
preserve Backwards Compatibility.
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-14?page=comments#action_12377440 ]
Hoss Man commented on SOLR-14:
------------------------------
It would probably be good to make sure we have some UnitTests of the existing WDF behavior prior to applying this patch, and then some tests that use this new feature just so it's clera how it works in various situations.
As for duplicates: my initial thought was that this could be handled by the proposed Filter in SOLR-11... but then i realized yonik has a point: the common case is probably going to be no intra-word delimiters, so a short circut check that doesn't crete two of every token would probably be better
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: http://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605080#action_12605080 ]
Yonik Seeley commented on SOLR-14:
----------------------------------
Someone would need to make an up-to-date patch that works (pref w/ some tests).
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605403#action_12605403 ]
Mike Klaas commented on SOLR-14:
--------------------------------
Note that it is very easy to use an external TokenFilter, so you could just c&p WDF into your own class and make the changes.
(Though I'm not saying that this _shouldn't_ make it in for 1.3)
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610232#action_12610232 ]
Yonik Seeley commented on SOLR-14:
----------------------------------
bq. the ordering of the tokens looks funny (to me) in analysis.jsp
Currently xxx-yyy is expanded to xxx yyy/xxxyyy (xxx is in the first position, yyy and xxxyyy are in the second position).
If we also want to index the original xxx-yyy, it's a bit arbitrary if it goes in the same position as the xxx or the yyy.
In this case I put it with xxx since the original token can then just be returned first (and retain everything about the original token such as it's payload, w/o having to clone it).
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Assignee: Yonik Seeley
> Attachments: SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-14?page=all ]
Richard "Trey" Hyde updated SOLR-14:
------------------------------------
Attachment: TokenizerFactory.java
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: http://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-14?page=comments#action_12377415 ]
Yonik Seeley commented on SOLR-14:
----------------------------------
Thanks for the patch Trey!
Can you give an example with the resulting token positions (or positionIncrements?)
Also, is there an easy way to prevent duplicate tokens from being produced (the preserveOriginal version will often be identical to catenateWords or catenateNumbers, right?)
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: http://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated SOLR-14:
-----------------------------
Attachment: SOLR-14.patch
Attaching new version of patch.
The previous version had position issues, and also had issues with certain flag combos.
I changed the strategy by handling "preserveOriginal" outside of the main loop (anywhere there is a "break" that falls through) and then just returning the original token first and adjusting the offset of the next token to overlap.
This should also be faster as it avoids token copying in the common case.
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Assignee: Yonik Seeley
> Attachments: SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-14?page=comments#action_12377426 ]
Richard "Trey" Hyde commented on SOLR-14:
-----------------------------------------
IMO, it does the wrong thing with position. I think it should have the position of the original term, I'll see if I can't figure out how to do that. It also doesn't work properly in the "single token" optimized case, another easy fix. As far as preventing duplicate tokens, an iteration though the token queue with equals should do the trick, no?
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: http://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley resolved SOLR-14.
------------------------------
Resolution: Fixed
Fix Version/s: 1.3
I committed after adding a performance test to verify that performance doesn't noticeably degrade for preserveOriginal=false.
Thanks everyone!
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Assignee: Yonik Seeley
> Fix For: 1.3
>
> Attachments: SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Geoffrey Young (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Geoffrey Young updated SOLR-14:
-------------------------------
Attachment: SOLR-14.patch
ok, I've given this a shot. I'm an an open-source guy, even an ASF guy, but not a java guy, so forgive my code ;)
the patch should apply cleanly to current trunk. the last patch before mine still had some issues that needed to be worked through wrt term duplication. this patch should work a bit better.
all current tests pass when adjusted to 'preserveOriginal=0' (default behavior, same as 1.2). I looked at augmenting the current tests for WDF and 'preserveOriginal=1' but it's beyond my current java abilities.
installing the patch and running the analyzer yields stuff like this:
foo => foo
foo-bar => foo-bar foo bar foobar
foo-bar baz => foo-bar foo bar foobar baz
foo! => foo foo!
which seems reasonable to me.
a little shepherding would be awesome.
thanks
--Geoff
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-14?page=all ]
Richard "Trey" Hyde updated SOLR-14:
------------------------------------
Attachment: WordDelimiterFilter.patch
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: http://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: WordDelimiterFilter.patch
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Richard \"Trey\" Hyde (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/SOLR-14?page=all ]
Richard "Trey" Hyde updated SOLR-14:
------------------------------------
Attachment: WordDelimiterFilter.patch
Ok, this one actually works in a few more cases.
There is still term duping if numotk > 1 and there are no intraword delimiters in the original string.
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: http://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (SOLR-14) Add the ability to preserve the
original term when using WordDelimiterFilter
Posted by "Mike Klaas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605410#action_12605410 ]
Mike Klaas commented on SOLR-14:
--------------------------------
Also, voting for an issue is a good way to increase its visibility
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Attachments: TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-14) Add the ability to preserve the original
term when using WordDelimiterFilter
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-14?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley reassigned SOLR-14:
--------------------------------
Assignee: Yonik Seeley
> Add the ability to preserve the original term when using WordDelimiterFilter
> ----------------------------------------------------------------------------
>
> Key: SOLR-14
> URL: https://issues.apache.org/jira/browse/SOLR-14
> Project: Solr
> Issue Type: Improvement
> Components: search
> Reporter: Richard "Trey" Hyde
> Assignee: Yonik Seeley
> Attachments: SOLR-14.patch, SOLR-14.patch, SOLR-14.patch, TokenizerFactory.java, WordDelimiterFilter.patch, WordDelimiterFilter.patch
>
>
> When doing prefix searching, you need to hang on to the original term othewise you'll miss many matches you should be making.
> Data: ABC-12345
> WordDelimiterFitler may change this into
> ABC 12345 ABC12345
> A user may enter a search such as
> ABC\-123*
> Which will fail to find a match given the above scenario.
> The attached patch will allow the use of the "preserveOriginal" option to WordDelimiterFilter and will analyse as
> ABC 12345 ABC12345 ABC-12345
> in which case we will get a postive match.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.