Instant deletes without committing

[BrendanD <br...@gmail.com>]

Hi,

Is it possible to send a command to have Solr flush the deletesPending
documents without doing a commit? I know there's a setting in solrconfig.xml
for setting a threshold value, but I'd like to somehow kick it off on
demand. We need this to be able to remove merchants from our product
listings if their accounts run out of funds.

Thanks,

Brendan
-- 
View this message in context: http://www.nabble.com/Instant-deletes-without-committing-tf4610169.html#a13165413
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Instant deletes without committing

[Yonik Seeley <yo...@apache.org>]

On 10/11/07, BrendanD <br...@gmail.com> wrote:
> Yes, we have some huge performance issues with non-cached queries. So doing a
> commit is very expensive for us. We have our autowarm count for our
> filterCache and queryResultCache both set to 4096. But I don't think that's
> near high enough. We did have it as high as 16384 before, but it took over
> an hour to warm.

Look in the logs... what took an hour to warm?   there are separate
autowarm log messages for the query and filter caches.

> Some of our queries take 30-60 seconds to complete if
> they're not cached.

1) Configure static warming requesst for any faceting that's common
2) Configure static warming requests for any filters (fq) that are common
3) size the filter cache larger than what's needed to hold all the
facets (if that's too much memory, try the minDf param... see the
wiki)
4) if indexing performance isn't an issue, lower mergeFactor to lower
the average number of segments in the index (or optimize if you can)

-Yonik

Re: Instant deletes without committing

[BrendanD <br...@gmail.com>]

Yes, we have some huge performance issues with non-cached queries. So doing a
commit is very expensive for us. We have our autowarm count for our
filterCache and queryResultCache both set to 4096. But I don't think that's
near high enough. We did have it as high as 16384 before, but it took over
an hour to warm. Some of our queries take 30-60 seconds to complete if
they're not cached.

For example:
Oct 11, 2007 4:51:50 PM org.apache.solr.core.SolrCore execute
INFO: /select/
rows=20&start=40&facet.query=attribute_id:1000013&facet.query=attribute_id:1005336&facet.query=attribute_id:1005697&facet.query=attribute_id:1006192&sort=merchant_count+desc&facet=true&facet.field=min_price_cad_rounded_to_tens&facet.field=manufacturer_id&facet.field=merchant_id&facet.field=has_coupon&facet.field=has_bundle&facet.field=has_sale_price&facet.field=has_promo&fq=product_is_active:true&fq=product_status_code:complete&fq=category_id:"1001294"&fq=attribute_id_value_en_pair:"1005697\:CCM"&qt=sti_dismax_en&f.min_price_cad_rounded_to_tens.facet.limit=-1
0 48061

As you can see, that query took 48 seconds. Usually after a commit and the
caches are flushed, we get a lot of queries that take a long time. It
usually brings our site to its knees for about 15 minutes until it catches
up.

If I could commit more often, that would be ideal, but unfortunately that's
not possible at the moment.

Does anyone know how DieselPoint can do near real-time updates to their
index? They must be using a different technique obviously. Still. We did
consider using DieselPoint before finding out about Solr. However,
DieselPoint is way too expensive for us at this point in time.

I'm sure we're doing lots wrong with our schema and queries to make them so
slow. We do have over 3 million documents in our index.

Thanks,

Brendan


Mike Klaas wrote:
> 
> On 11-Oct-07, at 2:47 PM, BrendanD wrote:
> 
>>
>> Hi,
>>
>> Is it possible to send a command to have Solr flush the deletesPending
>> documents without doing a commit? I know there's a setting in  
>> solrconfig.xml
>> for setting a threshold value, but I'd like to somehow kick it off on
>> demand. We need this to be able to remove merchants from our product
>> listings if their accounts run out of funds.
> 
> <commit/> means "make my edits visible now".  Can you explain why  
> this isn't exactly what you need?  Are there performance problems?
> 
> The flushing deletes parameter is a performance knob only: it doesn't  
> actually make the deletes visible until <commit/>
> 
> -Mike
> 
> 

-- 
View this message in context: http://www.nabble.com/Instant-deletes-without-committing-tf4610169.html#a13166522
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Instant deletes without committing

[Mike Klaas <mi...@gmail.com>]

On 11-Oct-07, at 2:47 PM, BrendanD wrote:

>
> Hi,
>
> Is it possible to send a command to have Solr flush the deletesPending
> documents without doing a commit? I know there's a setting in  
> solrconfig.xml
> for setting a threshold value, but I'd like to somehow kick it off on
> demand. We need this to be able to remove merchants from our product
> listings if their accounts run out of funds.

<commit/> means "make my edits visible now".  Can you explain why  
this isn't exactly what you need?  Are there performance problems?

The flushing deletes parameter is a performance knob only: it doesn't  
actually make the deletes visible until <commit/>

-Mike