You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by KaktuChakarabati <ji...@gmail.com> on 2009/08/26 00:10:04 UTC
Incremental Deletes to Index
Hey,
I was wondering - is there a mechanism in lucene and/or solr to mark a
document in the index
as deleted and then have this change reflect in query serving without
performing the whole
commit/warmup cycle? this seems to me largely appealing as it allows a kind
of solution
where deletes are simply processed by marking them in a bitmap or some such
structure
and then intersecting search results with those on a per-shard basis.
Anything in that direction? Otherwise, is there any critical issue
preventing such an implementation?
Thanks
-Chak
--
View this message in context: http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25143093.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Incremental Deletes to Index
Posted by Jason Rutherglen <ja...@gmail.com>.
You'll probably want to call Solr commit, however you'll want to
call IW.flush underneath (via a new Solr commit flag?).
Yes, the Solr caches would be somewhat useless if you're calling
Solr commit/flush rapidly. See SOLR-1308 on improving caches for
NRT.
On Tue, Aug 25, 2009 at 7:22 PM, KaktuChakarabati<ji...@gmail.com> wrote:
>
> So basically the idea is to replace the underlying IndexReader currently
> associated with a searcher/solrCore following an update without calling
> commit explicitly? This will also have the effect of bringing in inserts
> btw? or is it just usable for deletes?
> In terms of cache invalidation etc there are probably some issues i.e in
> respect to documents which are cached
> as part of some result set or so and need to expunged due to a deletion?
>
>
> Jason Rutherglen-2 wrote:
>>
>> I can give an overview, IW.getReader replaces IR.reopen. So
>> you'd replace in SolrCore.getSearcher. However as per another
>> discussion IW isn't public yet, so all you'd need to do is
>> expose it from UpdateHandler. Then it should work as you want,
>> though there would need to be a new method to create a new
>> searcher from IW.getReader without calling IW.commit.
>>
>> On Tue, Aug 25, 2009 at 4:37 PM, KaktuChakarabati<ji...@gmail.com>
>> wrote:
>>>
>>> Jason,
>>> sounds like a very promising change to me - so much that I would gladly
>>> work
>>> toward creating a patch myself.
>>> Are there any specific points in the code u could point me to if I wanna
>>> look at how to start off implementing it?
>>> Lucene/Solr Classes involved etc? i'll start looking myself anyhow but
>>> any
>>> tips would be helpful.. :)
>>>
>>> Thanks,
>>> -Chak
>>>
>>>
>>> Jason Rutherglen-2 wrote:
>>>>
>>>> This will be implemented as you're stating when
>>>> IndexWriter.getReader is incorporated. This will carry over
>>>> deletes in RAM until IW.commit is called (i.e. Solr commit).
>>>> It's a fairly simple change though perhaps too late for 1.4
>>>> release?
>>>>
>>>> On Tue, Aug 25, 2009 at 3:10 PM, KaktuChakarabati<ji...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hey,
>>>>> I was wondering - is there a mechanism in lucene and/or solr to mark a
>>>>> document in the index
>>>>> as deleted and then have this change reflect in query serving without
>>>>> performing the whole
>>>>> commit/warmup cycle? this seems to me largely appealing as it allows a
>>>>> kind
>>>>> of solution
>>>>> where deletes are simply processed by marking them in a bitmap or some
>>>>> such
>>>>> structure
>>>>> and then intersecting search results with those on a per-shard basis.
>>>>>
>>>>> Anything in that direction? Otherwise, is there any critical issue
>>>>> preventing such an implementation?
>>>>>
>>>>> Thanks
>>>>> -Chak
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25143093.html
>>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25144124.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25145535.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Incremental Deletes to Index
Posted by KaktuChakarabati <ji...@gmail.com>.
So basically the idea is to replace the underlying IndexReader currently
associated with a searcher/solrCore following an update without calling
commit explicitly? This will also have the effect of bringing in inserts
btw? or is it just usable for deletes?
In terms of cache invalidation etc there are probably some issues i.e in
respect to documents which are cached
as part of some result set or so and need to expunged due to a deletion?
Jason Rutherglen-2 wrote:
>
> I can give an overview, IW.getReader replaces IR.reopen. So
> you'd replace in SolrCore.getSearcher. However as per another
> discussion IW isn't public yet, so all you'd need to do is
> expose it from UpdateHandler. Then it should work as you want,
> though there would need to be a new method to create a new
> searcher from IW.getReader without calling IW.commit.
>
> On Tue, Aug 25, 2009 at 4:37 PM, KaktuChakarabati<ji...@gmail.com>
> wrote:
>>
>> Jason,
>> sounds like a very promising change to me - so much that I would gladly
>> work
>> toward creating a patch myself.
>> Are there any specific points in the code u could point me to if I wanna
>> look at how to start off implementing it?
>> Lucene/Solr Classes involved etc? i'll start looking myself anyhow but
>> any
>> tips would be helpful.. :)
>>
>> Thanks,
>> -Chak
>>
>>
>> Jason Rutherglen-2 wrote:
>>>
>>> This will be implemented as you're stating when
>>> IndexWriter.getReader is incorporated. This will carry over
>>> deletes in RAM until IW.commit is called (i.e. Solr commit).
>>> It's a fairly simple change though perhaps too late for 1.4
>>> release?
>>>
>>> On Tue, Aug 25, 2009 at 3:10 PM, KaktuChakarabati<ji...@gmail.com>
>>> wrote:
>>>>
>>>> Hey,
>>>> I was wondering - is there a mechanism in lucene and/or solr to mark a
>>>> document in the index
>>>> as deleted and then have this change reflect in query serving without
>>>> performing the whole
>>>> commit/warmup cycle? this seems to me largely appealing as it allows a
>>>> kind
>>>> of solution
>>>> where deletes are simply processed by marking them in a bitmap or some
>>>> such
>>>> structure
>>>> and then intersecting search results with those on a per-shard basis.
>>>>
>>>> Anything in that direction? Otherwise, is there any critical issue
>>>> preventing such an implementation?
>>>>
>>>> Thanks
>>>> -Chak
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25143093.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25144124.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25145535.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Incremental Deletes to Index
Posted by Jason Rutherglen <ja...@gmail.com>.
I can give an overview, IW.getReader replaces IR.reopen. So
you'd replace in SolrCore.getSearcher. However as per another
discussion IW isn't public yet, so all you'd need to do is
expose it from UpdateHandler. Then it should work as you want,
though there would need to be a new method to create a new
searcher from IW.getReader without calling IW.commit.
On Tue, Aug 25, 2009 at 4:37 PM, KaktuChakarabati<ji...@gmail.com> wrote:
>
> Jason,
> sounds like a very promising change to me - so much that I would gladly work
> toward creating a patch myself.
> Are there any specific points in the code u could point me to if I wanna
> look at how to start off implementing it?
> Lucene/Solr Classes involved etc? i'll start looking myself anyhow but any
> tips would be helpful.. :)
>
> Thanks,
> -Chak
>
>
> Jason Rutherglen-2 wrote:
>>
>> This will be implemented as you're stating when
>> IndexWriter.getReader is incorporated. This will carry over
>> deletes in RAM until IW.commit is called (i.e. Solr commit).
>> It's a fairly simple change though perhaps too late for 1.4
>> release?
>>
>> On Tue, Aug 25, 2009 at 3:10 PM, KaktuChakarabati<ji...@gmail.com>
>> wrote:
>>>
>>> Hey,
>>> I was wondering - is there a mechanism in lucene and/or solr to mark a
>>> document in the index
>>> as deleted and then have this change reflect in query serving without
>>> performing the whole
>>> commit/warmup cycle? this seems to me largely appealing as it allows a
>>> kind
>>> of solution
>>> where deletes are simply processed by marking them in a bitmap or some
>>> such
>>> structure
>>> and then intersecting search results with those on a per-shard basis.
>>>
>>> Anything in that direction? Otherwise, is there any critical issue
>>> preventing such an implementation?
>>>
>>> Thanks
>>> -Chak
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25143093.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25144124.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Incremental Deletes to Index
Posted by KaktuChakarabati <ji...@gmail.com>.
Jason,
sounds like a very promising change to me - so much that I would gladly work
toward creating a patch myself.
Are there any specific points in the code u could point me to if I wanna
look at how to start off implementing it?
Lucene/Solr Classes involved etc? i'll start looking myself anyhow but any
tips would be helpful.. :)
Thanks,
-Chak
Jason Rutherglen-2 wrote:
>
> This will be implemented as you're stating when
> IndexWriter.getReader is incorporated. This will carry over
> deletes in RAM until IW.commit is called (i.e. Solr commit).
> It's a fairly simple change though perhaps too late for 1.4
> release?
>
> On Tue, Aug 25, 2009 at 3:10 PM, KaktuChakarabati<ji...@gmail.com>
> wrote:
>>
>> Hey,
>> I was wondering - is there a mechanism in lucene and/or solr to mark a
>> document in the index
>> as deleted and then have this change reflect in query serving without
>> performing the whole
>> commit/warmup cycle? this seems to me largely appealing as it allows a
>> kind
>> of solution
>> where deletes are simply processed by marking them in a bitmap or some
>> such
>> structure
>> and then intersecting search results with those on a per-shard basis.
>>
>> Anything in that direction? Otherwise, is there any critical issue
>> preventing such an implementation?
>>
>> Thanks
>> -Chak
>> --
>> View this message in context:
>> http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25143093.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25144124.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Incremental Deletes to Index
Posted by Jason Rutherglen <ja...@gmail.com>.
This will be implemented as you're stating when
IndexWriter.getReader is incorporated. This will carry over
deletes in RAM until IW.commit is called (i.e. Solr commit).
It's a fairly simple change though perhaps too late for 1.4
release?
On Tue, Aug 25, 2009 at 3:10 PM, KaktuChakarabati<ji...@gmail.com> wrote:
>
> Hey,
> I was wondering - is there a mechanism in lucene and/or solr to mark a
> document in the index
> as deleted and then have this change reflect in query serving without
> performing the whole
> commit/warmup cycle? this seems to me largely appealing as it allows a kind
> of solution
> where deletes are simply processed by marking them in a bitmap or some such
> structure
> and then intersecting search results with those on a per-shard basis.
>
> Anything in that direction? Otherwise, is there any critical issue
> preventing such an implementation?
>
> Thanks
> -Chak
> --
> View this message in context: http://www.nabble.com/Incremental-Deletes-to-Index-tp25143093p25143093.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>