You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "S.L" <si...@gmail.com> on 2015/02/04 19:04:13 UTC
DocExpirationUpdateProcessorFactory not deleting records
I am trying to use the DocExpirationUpdateProcessorFactoryfactory in Solr
4.10.1 version.
I have included the following in my solrconfig.xml
<updateRequestProcessorChain default="true">
<processor class="solr.UUIDUpdateProcessorFactory">
<str name="fieldName">id</str>
</processor>
<processor class="solr.TimestampUpdateProcessorFactory">
<str name="fieldName">timestamp</str>
</processor>
<processor
class="solr.processor.DocExpirationUpdateProcessorFactory">
<int name="autoDeletePeriodSeconds">30</int>
<str name="ttlFieldName">ttl</str>
<str name="expirationFieldName">expire_at</str>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
And I have included the following in my schema.xml
<field name="timestamp" type="date" indexed="true" stored="true"
default="NOW" multiValued="false"/>
<field name="ttl" type="date" indexed="true" stored="true"
default="NOW+60SECONDS" multiValued="false"/>
<field name="expire_at" type="date" indexed="true" stored="true"
multiValued="false"/>
As you can see I am setting the time to live to be 60 seconds and checking
to delete every 30 seconds, when I insert a document , and check after a
minute or couple or an hour it never gets deleted.
This is what I see in the indexed document , can you please let me know
what might be the issue here ? Please note that the expire_at field is
never getting generated in the Solr document as can be seen below.
"id": "3888a8ac-fbc4-437a-8248-132384753c00",
"timestamp": "2015-02-04T04:09:21.29Z",
"_version_": 1492147724740460500,
"ttl": "2015-02-04T04:10:21.29Z"
Re: DocExpirationUpdateProcessorFactory not deleting records
Posted by "S.L" <si...@gmail.com>.
Great, this is the first example I have seen so far, I wish we could
include this in the Wiki. Thanks again!
On Wed, Feb 4, 2015 at 2:04 PM, Chris Hostetter <ho...@fucit.org>
wrote:
> :
> : Thanks for giving multiple options , I ll try them out both ,but last
> time
> : I checked, having "+60SECONDS" as the default value for ttl was giving me
> : an invalid date format exception...., I am assuming that would only be
> the
>
> that's because ttl should not be a date field -- it should be a *string*
> (as noted in my examples)
>
> "time to live" is a date math expression that the processor will evaluate
> for you -- not a date. if you want to specify an explicit date, just set
> expire_at directly.
>
> ie: do you wnat to do the match yourself (set expire_at as a date field)
> or do you want the processor to do the math itself (set ttl as a string
> field)
>
> : > ...even if you redefined your ttl field to look like this...
> : >
> : > <field name="ttl" type="string" default="+60SECONDS" />
> : >
> : > ...the expire_at still wouldn't be populated by the processor because
> : > schema field "default" values are populated *after* the processors run
> --
> : > so when the DocExpirationUpdateProcessorFactory sees the documents
> being
> : > added, it has no idea that they all have a default ttl, so it doesn't
> know
> : > that you want it to compute an expire_at for you.
> : >
> : > instead of using default="" in the schema, you can use the
> : > DefaultValueUpdateProcessorFactory to assign it *before* the
> : > DocExpirationUpdateProcessorFactory sees the doc...
> : >
> : > <processor class="solr.DefaultValueUpdateProcessorFactory">
> : > <str name="fieldName">ttl</str>
> : > <str name="value">+60SECONDS</str>
> : > </processor>
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: DocExpirationUpdateProcessorFactory not deleting records
Posted by Chris Hostetter <ho...@fucit.org>.
:
: Thanks for giving multiple options , I ll try them out both ,but last time
: I checked, having "+60SECONDS" as the default value for ttl was giving me
: an invalid date format exception...., I am assuming that would only be the
that's because ttl should not be a date field -- it should be a *string*
(as noted in my examples)
"time to live" is a date math expression that the processor will evaluate
for you -- not a date. if you want to specify an explicit date, just set
expire_at directly.
ie: do you wnat to do the match yourself (set expire_at as a date field)
or do you want the processor to do the math itself (set ttl as a string
field)
: > ...even if you redefined your ttl field to look like this...
: >
: > <field name="ttl" type="string" default="+60SECONDS" />
: >
: > ...the expire_at still wouldn't be populated by the processor because
: > schema field "default" values are populated *after* the processors run --
: > so when the DocExpirationUpdateProcessorFactory sees the documents being
: > added, it has no idea that they all have a default ttl, so it doesn't know
: > that you want it to compute an expire_at for you.
: >
: > instead of using default="" in the schema, you can use the
: > DefaultValueUpdateProcessorFactory to assign it *before* the
: > DocExpirationUpdateProcessorFactory sees the doc...
: >
: > <processor class="solr.DefaultValueUpdateProcessorFactory">
: > <str name="fieldName">ttl</str>
: > <str name="value">+60SECONDS</str>
: > </processor>
-Hoss
http://www.lucidworks.com/
Re: DocExpirationUpdateProcessorFactory not deleting records
Posted by "S.L" <si...@gmail.com>.
Thanks for giving multiple options , I ll try them out both ,but last time
I checked, having "+60SECONDS" as the default value for ttl was giving me
an invalid date format exception...., I am assuming that would only be the
case if I use it with the default mechanism in schema.xml, but not when we
use the solr.DefaultValueUpdateProcessorFactory ?
On Wed, Feb 4, 2015 at 1:56 PM, Chris Hostetter <ho...@fucit.org>
wrote:
>
> : <processor
> : class="solr.processor.DocExpirationUpdateProcessorFactory">
> : <int name="autoDeletePeriodSeconds">30</int>
> : <str name="ttlFieldName">ttl</str>
> : <str name="expirationFieldName">expire_at</str>
> : </processor>
>
> ...
>
> : And I have included the following in my schema.xml
> :
> : <field name="ttl" type="date" indexed="true" stored="true"
> : default="NOW+60SECONDS" multiValued="false"/>
>
> there are a couple of problems here...
>
> : As you can see I am setting the time to live to be 60 seconds and
> checking
> : to delete every 30 seconds, when I insert a document , and check after a
> : minute or couple or an hour it never gets deleted.
>
> first off: you aren't actaully setting the ttl to "60 seconds" you are
> setting the ttl to be a fixed moment in time which is 60 seconds from when
> the doc is written to the index -- basically you are eliminating hte need
> for having a ttl field/param at all and saying "this is *exactly* when the
> document should expire".
>
> if that's what you want to do, just elimintae the ttleFieldName everywhere
> in your schema.xml and solrconfig.xml and setup expire_at in your
> schema.xml with a default="NOW+60SECONDS" and you'll probably be good to
> go.
>
> second...
>
> : what might be the issue here ? Please note that the expire_at field is
> : never getting generated in the Solr document as can be seen below.
>
> ...even if you redefined your ttl field to look like this...
>
> <field name="ttl" type="string" default="+60SECONDS" />
>
> ...the expire_at still wouldn't be populated by the processor because
> schema field "default" values are populated *after* the processors run --
> so when the DocExpirationUpdateProcessorFactory sees the documents being
> added, it has no idea that they all have a default ttl, so it doesn't know
> that you want it to compute an expire_at for you.
>
> instead of using default="" in the schema, you can use the
> DefaultValueUpdateProcessorFactory to assign it *before* the
> DocExpirationUpdateProcessorFactory sees the doc...
>
> <processor class="solr.DefaultValueUpdateProcessorFactory">
> <str name="fieldName">ttl</str>
> <str name="value">+60SECONDS</str>
> </processor>
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: DocExpirationUpdateProcessorFactory not deleting records
Posted by Chris Hostetter <ho...@fucit.org>.
: <processor
: class="solr.processor.DocExpirationUpdateProcessorFactory">
: <int name="autoDeletePeriodSeconds">30</int>
: <str name="ttlFieldName">ttl</str>
: <str name="expirationFieldName">expire_at</str>
: </processor>
...
: And I have included the following in my schema.xml
:
: <field name="ttl" type="date" indexed="true" stored="true"
: default="NOW+60SECONDS" multiValued="false"/>
there are a couple of problems here...
: As you can see I am setting the time to live to be 60 seconds and checking
: to delete every 30 seconds, when I insert a document , and check after a
: minute or couple or an hour it never gets deleted.
first off: you aren't actaully setting the ttl to "60 seconds" you are
setting the ttl to be a fixed moment in time which is 60 seconds from when
the doc is written to the index -- basically you are eliminating hte need
for having a ttl field/param at all and saying "this is *exactly* when the
document should expire".
if that's what you want to do, just elimintae the ttleFieldName everywhere
in your schema.xml and solrconfig.xml and setup expire_at in your
schema.xml with a default="NOW+60SECONDS" and you'll probably be good to
go.
second...
: what might be the issue here ? Please note that the expire_at field is
: never getting generated in the Solr document as can be seen below.
...even if you redefined your ttl field to look like this...
<field name="ttl" type="string" default="+60SECONDS" />
...the expire_at still wouldn't be populated by the processor because
schema field "default" values are populated *after* the processors run --
so when the DocExpirationUpdateProcessorFactory sees the documents being
added, it has no idea that they all have a default ttl, so it doesn't know
that you want it to compute an expire_at for you.
instead of using default="" in the schema, you can use the
DefaultValueUpdateProcessorFactory to assign it *before* the
DocExpirationUpdateProcessorFactory sees the doc...
<processor class="solr.DefaultValueUpdateProcessorFactory">
<str name="fieldName">ttl</str>
<str name="value">+60SECONDS</str>
</processor>
-Hoss
http://www.lucidworks.com/