You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by drogon <ji...@gmail.com> on 2011/09/25 10:54:58 UTC
How to apply filters to stored data
Is it possible to apply filters to stored data like we can apply filter when
indexing. For example I use KeepWordFilter on a field during indexing. But I
don't want filtered data to be even stored ie I want the content in index
and store for this field to be same.
Also when retrieving data(querying solr) I find that the content retrieved
is the stored data. Is it possible to get the data that is indexed as
against the stored one?
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3366230.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to apply filters to stored data
Posted by Jithin <ji...@gmail.com>.
Is UpdateProcessor triggered when updating an existing document or for new
documents also?
On Tue, Sep 27, 2011 at 6:00 AM, Chris Hostetter-3 [via Lucene] <
ml-node+s472066n3371110h28@n3.nabble.com> wrote:
>
> : Hi Erick, The problem I am trying to solve is to filter invalid entities.
>
> : Users might mispell or enter a new entity name. This new/invalid entities
>
> : need to pass through a KeepWordFilter so that it won't pollute our
> : autocomplete result.
>
> how are you doing autocomplete?
>
> if you are using the Suggest feature of solr, then thta's based on the
> indexed terms anyway (last time i checked) so you don't need to manipulate
> the stored field values.
>
> In general, the only way to manipluate the stored field values is to do it
> in an update processor -- which can mutate the documents long before the
> schema is ever even consulted.
>
> -Hoss
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3371110.html
> To unsubscribe from How to apply filters to stored data, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3366230&code=aml0aGluMTk4N0BnbWFpbC5jb218MzM2NjIzMHwtMTEwMTgwMTA3Ng==>.
>
>
--
Thanks
Jithin Emmanuel
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3371200.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to apply filters to stored data
Posted by Chris Hostetter <ho...@fucit.org>.
: Hi Erick, The problem I am trying to solve is to filter invalid entities.
: Users might mispell or enter a new entity name. This new/invalid entities
: need to pass through a KeepWordFilter so that it won't pollute our
: autocomplete result.
how are you doing autocomplete?
if you are using the Suggest feature of solr, then thta's based on the
indexed terms anyway (last time i checked) so you don't need to manipulate
the stored field values.
In general, the only way to manipluate the stored field values is to do it
in an update processor -- which can mutate the documents long before the
schema is ever even consulted.
-Hoss
Re: How to apply filters to stored data
Posted by Erik Hatcher <er...@gmail.com>.
Well ... DIH can.
And update processors can.
And of course client-side indexers.
But yeah... elbow grease required.
Erik
On Sep 25, 2011, at 16:32, Erick Erickson <er...@gmail.com> wrote:
> Not that I know of...
>
> On Sun, Sep 25, 2011 at 11:15 AM, Jithin <ji...@gmail.com> wrote:
>>
>> Erick Erickson wrote:
>>>
>>> See below:
>>>
>>> On Sun, Sep 25, 2011 at 9:53 AM, Jithin <jithin1987@gmail.com>
>>> wrote:
>>>> Hi Erick, The problem I am trying to solve is to filter invalid entities.
>>>> Users might mispell or enter a new entity name. This new/invalid entities
>>>> need to pass through a KeepWordFilter so that it won't pollute our
>>>> autocomplete result.
>>>>
>>>
>>> Right. But if you have a KeepWordFilter, that implies that you have a list
>>> of known good words. Couldn't you use that file as your base for the
>>> autosuggest component?
>>>
>>
>> I think that is possible.
>> But is there any other mechanism within solr/lucene to preprocess stored
>> data.
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3367158.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Re: How to apply filters to stored data
Posted by Erick Erickson <er...@gmail.com>.
Not that I know of...
On Sun, Sep 25, 2011 at 11:15 AM, Jithin <ji...@gmail.com> wrote:
>
> Erick Erickson wrote:
>>
>> See below:
>>
>> On Sun, Sep 25, 2011 at 9:53 AM, Jithin <jithin1987@gmail.com>
>> wrote:
>>> Hi Erick, The problem I am trying to solve is to filter invalid entities.
>>> Users might mispell or enter a new entity name. This new/invalid entities
>>> need to pass through a KeepWordFilter so that it won't pollute our
>>> autocomplete result.
>>>
>>
>> Right. But if you have a KeepWordFilter, that implies that you have a list
>> of known good words. Couldn't you use that file as your base for the
>> autosuggest component?
>>
>
> I think that is possible.
> But is there any other mechanism within solr/lucene to preprocess stored
> data.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3367158.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to apply filters to stored data
Posted by Jithin <ji...@gmail.com>.
Erick Erickson wrote:
>
> See below:
>
> On Sun, Sep 25, 2011 at 9:53 AM, Jithin <jithin1987@gmail.com>
> wrote:
>> Hi Erick, The problem I am trying to solve is to filter invalid entities.
>> Users might mispell or enter a new entity name. This new/invalid entities
>> need to pass through a KeepWordFilter so that it won't pollute our
>> autocomplete result.
>>
>
> Right. But if you have a KeepWordFilter, that implies that you have a list
> of known good words. Couldn't you use that file as your base for the
> autosuggest component?
>
I think that is possible.
But is there any other mechanism within solr/lucene to preprocess stored
data.
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3367158.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to apply filters to stored data
Posted by Erick Erickson <er...@gmail.com>.
See below:
On Sun, Sep 25, 2011 at 9:53 AM, Jithin <ji...@gmail.com> wrote:
> Hi Erick, The problem I am trying to solve is to filter invalid entities.
> Users might mispell or enter a new entity name. This new/invalid entities
> need to pass through a KeepWordFilter so that it won't pollute our
> autocomplete result.
>
Right. But if you have a KeepWordFilter, that implies that you have a list
of known good words. Couldn't you use that file as your base for the
autosuggest component?
> I was looking into Luke. And it does seem to solve my use case, but is Luke
> something I can use in a production setup?
You'll find the performance unacceptably slow if you tried to do something
similar in production. The nature of an inverted index makes reconstructing
a document from the various terms costly.
> Also when does <copyField> happens? Is the data being copied a result of
> application of all filters or unmodified one?
<copyField> happens to the raw input, not the result of your
analysis chain. And you can't chain <copyField> directives,
i.e.
<copyField source="field1" dest="field2" />
<copyField source="field2" dest="field3" />
would not put the contents of "field1" into "field3"
Best
Erick
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3366987.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to apply filters to stored data
Posted by Jithin <ji...@gmail.com>.
Hi Erick, The problem I am trying to solve is to filter invalid entities.
Users might mispell or enter a new entity name. This new/invalid entities
need to pass through a KeepWordFilter so that it won't pollute our
autocomplete result.
I was looking into Luke. And it does seem to solve my use case, but is Luke
something I can use in a production setup?
Also when does <copyField> happens? Is the data being copied a result of
application of all filters or unmodified one?
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3366987.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to apply filters to stored data
Posted by Erick Erickson <er...@gmail.com>.
No and no.
Hmmm, that's a bit terse. The split between stored and indexed
happens quite early in the update process, there's no way I know
of to use the tokenized stream as the input to your stored data.
And there's no out-of-the-box way to get the indexed tokens back
For anything except very small fields, this would be quite costly.
What problem are you trying to solve? Perhaps this is an XY problem.
See: http://people.apache.org/~hossman/#xyproblem
Best
Erick
On Sun, Sep 25, 2011 at 1:54 AM, drogon <ji...@gmail.com> wrote:
> Is it possible to apply filters to stored data like we can apply filter when
> indexing. For example I use KeepWordFilter on a field during indexing. But I
> don't want filtered data to be even stored ie I want the content in index
> and store for this field to be same.
> Also when retrieving data(querying solr) I find that the content retrieved
> is the stored data. Is it possible to get the data that is indexed as
> against the stored one?
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-to-apply-filters-to-stored-data-tp3366230p3366230.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>