You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by XuQing Tan <mi...@gmail.com> on 2017/03/03 03:29:20 UTC
How to update index after document expired.
Hi folks
in our case, we have contents need to be refreshed periodically according
to the TTL of each document.
looks like DocExpirationUpdateProcessorFactory is a quite good fit except
that it does delete the document only, but no way to update the indexing
with the new document.
I don't see there's a way to hook into DocExpirationUpdateProcessorFactory
for custom logic like get the document and update index. and even
DocExpirationUpdateProcessorFactory is a final class.
so just want to confirm with you, is there any existing solution for this?
otherwise, I might have our own copy of DocExpirationUpdateProcessorFactory
with custom code.
Thanks & Best Regards!
///
(. .)
--------ooO--(_)--Ooo--------
| Nick Tan |
------------------------------------
Re: How to update index after document expired.
Posted by XuQing Tan <mi...@gmail.com>.
On Fri, Mar 3, 2017 at 9:17 AM, Erick Erickson <er...@gmail.com>
wrote:
> you'd have to copy/paste or petition to make
> DocExpirationUpdateProcessorFactory not final.
>
yes, I copied DocExpirationUpdateProcessorFactory with additional reason
is, our xml content from external source already contains expires in Date,
i don't need to bother with converting it to TTL into date math expr (with
some update processor) then converting it back to Date, it's just
inefficient. I want to use the expires field directly.
the custom DocExpirationUpdateProcessorFactory in our case is to check the
expired document periodically (within 1 min) and refresh them from external
source if any. in this way, we can achieve almost instantly refresh once
the doc expired, rather than waiting for another large time scheduled task
process.
Thanks & Best Regards!
///
(. .)
--------ooO--(_)--Ooo--------
| Nick Tan |
------------------------------------
Re: How to update index after document expired.
Posted by Erick Erickson <er...@gmail.com>.
Right, you'd have to copy/paste or petition to make
DocExpirationUpdateProcessorFactory not final.
Although it was likely made that for a good reason.
It does beg the question however, is this the right
thing to do? What you have here is some record
of when docs should expire. You had to know that
when you indexed the doc. Would an alternative be
to just "roll your own"? That is, you have some process
that checks your presumed system-of-record and updates
the docs that it finds there that are past their expiration
date? It could even delete those that have no newer
version in the system-of-record...
Not sure whether it'd be better or not, just thinking out
loud. Well, in print.
Erick
On Thu, Mar 2, 2017 at 9:53 PM, XuQing Tan <mi...@gmail.com> wrote:
> SOLR gets the updated content from external source (by calling a REST api
> which returns xml content).
> so my question is how can I plug this logic
> in DocExpirationUpdateProcessorFactory, saying poll from external source
> and update indexing?
>
> for now i'm thinking to use a custom 'autoDeleteChainName', still i'm
> experimenting with this, is it feasible?
>
> <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
> <str name="autoDeleteChainName">scheduled-delete-and-update</str>
> ...
> </processor>
>
> Thanks & Best Regards!
>
> ///
> (. .)
> --------ooO--(_)--Ooo--------
> | Nick Tan |
> ------------------------------------
>
> On Thu, Mar 2, 2017 at 7:36 PM, Alexandre Rafalovitch <ar...@gmail.com>
> wrote:
>
>> Where would Solr get the updated content? Do you mean would it poll
>> from external source to refresh? Then, no. And if it is pushed from
>> external sources to Solr, then you just replace it as normal.
>>
>> Not sure if I understand your use-case exactly.
>>
>> Regards,
>> Alex.
>> ----
>> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>>
>>
>> On 2 March 2017 at 22:29, XuQing Tan <mi...@gmail.com> wrote:
>> > Hi folks
>> >
>> > in our case, we have contents need to be refreshed periodically according
>> > to the TTL of each document.
>> >
>> > looks like DocExpirationUpdateProcessorFactory is a quite good fit
>> except
>> > that it does delete the document only, but no way to update the indexing
>> > with the new document.
>> >
>> > I don't see there's a way to hook into DocExpirationUpdateProcessorFa
>> ctory
>> > for custom logic like get the document and update index. and even
>> > DocExpirationUpdateProcessorFactory is a final class.
>> >
>> > so just want to confirm with you, is there any existing solution for
>> this?
>> >
>> > otherwise, I might have our own copy of DocExpirationUpdateProcessorFa
>> ctory
>> > with custom code.
>> >
>> >
>> > Thanks & Best Regards!
>> >
>> > ///
>> > (. .)
>> > --------ooO--(_)--Ooo--------
>> > | Nick Tan |
>> > ------------------------------------
>>
Re: How to update index after document expired.
Posted by XuQing Tan <mi...@gmail.com>.
SOLR gets the updated content from external source (by calling a REST api
which returns xml content).
so my question is how can I plug this logic
in DocExpirationUpdateProcessorFactory, saying poll from external source
and update indexing?
for now i'm thinking to use a custom 'autoDeleteChainName', still i'm
experimenting with this, is it feasible?
<processor class="solr.processor.DocExpirationUpdateProcessorFactory">
<str name="autoDeleteChainName">scheduled-delete-and-update</str>
...
</processor>
Thanks & Best Regards!
///
(. .)
--------ooO--(_)--Ooo--------
| Nick Tan |
------------------------------------
On Thu, Mar 2, 2017 at 7:36 PM, Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> Where would Solr get the updated content? Do you mean would it poll
> from external source to refresh? Then, no. And if it is pushed from
> external sources to Solr, then you just replace it as normal.
>
> Not sure if I understand your use-case exactly.
>
> Regards,
> Alex.
> ----
> http://www.solr-start.com/ - Resources for Solr users, new and experienced
>
>
> On 2 March 2017 at 22:29, XuQing Tan <mi...@gmail.com> wrote:
> > Hi folks
> >
> > in our case, we have contents need to be refreshed periodically according
> > to the TTL of each document.
> >
> > looks like DocExpirationUpdateProcessorFactory is a quite good fit
> except
> > that it does delete the document only, but no way to update the indexing
> > with the new document.
> >
> > I don't see there's a way to hook into DocExpirationUpdateProcessorFa
> ctory
> > for custom logic like get the document and update index. and even
> > DocExpirationUpdateProcessorFactory is a final class.
> >
> > so just want to confirm with you, is there any existing solution for
> this?
> >
> > otherwise, I might have our own copy of DocExpirationUpdateProcessorFa
> ctory
> > with custom code.
> >
> >
> > Thanks & Best Regards!
> >
> > ///
> > (. .)
> > --------ooO--(_)--Ooo--------
> > | Nick Tan |
> > ------------------------------------
>
Re: How to update index after document expired.
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Where would Solr get the updated content? Do you mean would it poll
from external source to refresh? Then, no. And if it is pushed from
external sources to Solr, then you just replace it as normal.
Not sure if I understand your use-case exactly.
Regards,
Alex.
----
http://www.solr-start.com/ - Resources for Solr users, new and experienced
On 2 March 2017 at 22:29, XuQing Tan <mi...@gmail.com> wrote:
> Hi folks
>
> in our case, we have contents need to be refreshed periodically according
> to the TTL of each document.
>
> looks like DocExpirationUpdateProcessorFactory is a quite good fit except
> that it does delete the document only, but no way to update the indexing
> with the new document.
>
> I don't see there's a way to hook into DocExpirationUpdateProcessorFactory
> for custom logic like get the document and update index. and even
> DocExpirationUpdateProcessorFactory is a final class.
>
> so just want to confirm with you, is there any existing solution for this?
>
> otherwise, I might have our own copy of DocExpirationUpdateProcessorFactory
> with custom code.
>
>
> Thanks & Best Regards!
>
> ///
> (. .)
> --------ooO--(_)--Ooo--------
> | Nick Tan |
> ------------------------------------