You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by pavelhladik <pa...@profimedia.cz> on 2015/03/25 16:08:16 UTC

Optimize SolrCloud without downtime

Hi,

I didn't find the answer yet, please help. We have standalone Solr 5.0.0
with a few cores yet. One of those cores contains:

numDocs:120M
deletedDocs:110M

Our data are changing frequently so that's why so many deletedDocs.
Optimized core takes around 50GB on disk, we are now almost on 100GB and I'm
looking for best solution howto optimize this huge core without downtime. I
know optimization working in background, but anyway when the optimization is
running our search system is slow and sometimes I receive errors - this
behavior is like a downtime for us.

I would like to switch to SolrCloud, the performance is not a issue, so I
don't need the sharding feature at this time. I'm more interested with
replication and distribute requests by some Nginx proxy. Idea is:

1) proxy forward requests to node1 and optimize cores on node2
2) proxy forward requests to node2 and optimize cores on node1

But when I do optimize on node2, the node1 is doing optimization as well,
even if I use the "distrib=false" with curl.

Can you please recommend architecture for optimizing without downtime? Many
thanks.

Pavel



--
View this message in context: http://lucene.472066.n3.nabble.com/Optimize-SolrCloud-without-downtime-tp4195170.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Optimize SolrCloud without downtime

Posted by Erick Erickson <er...@gmail.com>.
bq:  It does NOT optimize multiple replicas or shards in parallel.

This behavior was changed in 4.10 though, see:
https://issues.apache.org/jira/browse/SOLR-6264

So with 5.0 Pavel is seeing the result of that JIRA I bet.

I have to agree with Shawn, the optimization step should proceed
invisibly in the background, I suspect you have something else
going on here.

FWIW,
Erick

On Wed, Mar 25, 2015 at 9:54 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 3/25/2015 9:08 AM, pavelhladik wrote:
>> Our data are changing frequently so that's why so many deletedDocs.
>> Optimized core takes around 50GB on disk, we are now almost on 100GB and I'm
>> looking for best solution howto optimize this huge core without downtime. I
>> know optimization working in background, but anyway when the optimization is
>> running our search system is slow and sometimes I receive errors - this
>> behavior is like a downtime for us.
>>
>> I would like to switch to SolrCloud, the performance is not a issue, so I
>> don't need the sharding feature at this time. I'm more interested with
>> replication and distribute requests by some Nginx proxy. Idea is:
>>
>> 1) proxy forward requests to node1 and optimize cores on node2
>> 2) proxy forward requests to node2 and optimize cores on node1
>>
>> But when I do optimize on node2, the node1 is doing optimization as well,
>> even if I use the "distrib=false" with curl.
>
> You are correct - with SolrCloud, any optimize command will optimize the
> entire collection, one shard replica at a time, regardless of any
> distrib parameter.  It does NOT optimize multiple replicas or shards in
> parallel.  I thought we had an issue in Jira asking to make optimize
> honor a "distrib=false" parameter, but I can't find it.  Even if that
> were fixed, it would not help you, because SolrCloud is only optimizing
> one shard replica at any given moment.
>
> Optimization does NOT directly result in downtime ... but because
> optimize generates a very large amount of disk I/O, it can be disruptive
> if your server does not have enough resources.
>
> I don't have enough information to say for sure, but I am betting that
> you don't have enough RAM in your machine to effectively cache your
> index, so anything that negatively affects performance, like an
> optimize, is too much for your server to handle at the same time as
> ongoing queries or indexing.  The info on this wiki page can help you
> determine how much total RAM you might need:
>
> http://wiki.apache.org/solr/SolrPerformanceProblems
>
> Thanks,
> Shawn
>

Re: Optimize SolrCloud without downtime

Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/25/2015 9:08 AM, pavelhladik wrote:
> Our data are changing frequently so that's why so many deletedDocs.
> Optimized core takes around 50GB on disk, we are now almost on 100GB and I'm
> looking for best solution howto optimize this huge core without downtime. I
> know optimization working in background, but anyway when the optimization is
> running our search system is slow and sometimes I receive errors - this
> behavior is like a downtime for us.
>
> I would like to switch to SolrCloud, the performance is not a issue, so I
> don't need the sharding feature at this time. I'm more interested with
> replication and distribute requests by some Nginx proxy. Idea is:
>
> 1) proxy forward requests to node1 and optimize cores on node2
> 2) proxy forward requests to node2 and optimize cores on node1
>
> But when I do optimize on node2, the node1 is doing optimization as well,
> even if I use the "distrib=false" with curl.

You are correct - with SolrCloud, any optimize command will optimize the
entire collection, one shard replica at a time, regardless of any
distrib parameter.  It does NOT optimize multiple replicas or shards in
parallel.  I thought we had an issue in Jira asking to make optimize
honor a "distrib=false" parameter, but I can't find it.  Even if that
were fixed, it would not help you, because SolrCloud is only optimizing
one shard replica at any given moment.

Optimization does NOT directly result in downtime ... but because
optimize generates a very large amount of disk I/O, it can be disruptive
if your server does not have enough resources.

I don't have enough information to say for sure, but I am betting that
you don't have enough RAM in your machine to effectively cache your
index, so anything that negatively affects performance, like an
optimize, is too much for your server to handle at the same time as
ongoing queries or indexing.  The info on this wiki page can help you
determine how much total RAM you might need:

http://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn


Re: Optimize SolrCloud without downtime

Posted by Erick Erickson <er...@gmail.com>.
I really don't have a good explanation here, those are the default
values and the folks who set them up no doubt chose them with some
care. Afraid I'll have to defer to people who actually know the
code...

Erick

On Mon, Mar 30, 2015 at 11:59 PM, Pavel Hladik
<pa...@profimedia.cz> wrote:
> When we indexing I see the deleted docs are a bit changing.. I was surprised
> when developer reindex 120M index, we had around 110M of deleted docs and
> this number was not falling. As you wrote, the typical behavior should be
> merging deleted docs to 10-20% of whole index? So it should be after two
> weeks around 20M of deleted docs.
>
> I'm not sure of settings in our solrconfig.xml:
>
>             <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>               <int name="maxMergeAtOnce">10</int>
>               <int name="segmentsPerTier">10</int>
>             </mergePolicy>
>
>             <mergeFactor>10</mergeFactor>
>
>             <mergeScheduler
> class="org.apache.lucene.index.ConcurrentMergeScheduler"/>
>
> Should we change some of them? The mergeScheduler class is empty.
>
> When I go to Core Admin, select our core I see:
>
> maxCacheMB=48.0 maxMergeSizeMB=4.0
>
> Is that ok, or the values are too low?
>
> Best,
> Pavel
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Optimize-SolrCloud-without-downtime-tp4195170p4196506.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Optimize SolrCloud without downtime

Posted by Pavel Hladik <pa...@profimedia.cz>.
When we indexing I see the deleted docs are a bit changing.. I was surprised
when developer reindex 120M index, we had around 110M of deleted docs and
this number was not falling. As you wrote, the typical behavior should be
merging deleted docs to 10-20% of whole index? So it should be after two
weeks around 20M of deleted docs.

I'm not sure of settings in our solrconfig.xml:

            <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
              <int name="maxMergeAtOnce">10</int>
              <int name="segmentsPerTier">10</int>
            </mergePolicy>

            <mergeFactor>10</mergeFactor>

            <mergeScheduler
class="org.apache.lucene.index.ConcurrentMergeScheduler"/>

Should we change some of them? The mergeScheduler class is empty.

When I go to Core Admin, select our core I see:

maxCacheMB=48.0 maxMergeSizeMB=4.0

Is that ok, or the values are too low?

Best,
Pavel



--
View this message in context: http://lucene.472066.n3.nabble.com/Optimize-SolrCloud-without-downtime-tp4195170p4196506.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Optimize SolrCloud without downtime

Posted by Erick Erickson <er...@gmail.com>.
Hmmm, are you indexing during the time you see the deleted docs not
changing? Because this is very strange. Theoretically, if you reindex
everything, that should result in segments that have _no_ live docs in
them and they should really disappear ASAP.

One way to work around this if we determine that this is a problem in
Solr might be to create a second collection and index to _that_, then
use collection aliasing to use the new index. But that's not
optimal...

Best,
Erick


On Mon, Mar 30, 2015 at 4:35 AM, Pavel Hladik
<pa...@profimedia.cz> wrote:
> Hi,
>
> thanks for reply. We have a lot of deleted docs cause we have to reindex all
> records from time to time, changing some important parameters.. When we do
> update, it means create and delete.
>
> Our deleted docs do not disappear by merging segments. I see our deleted
> docs are almost the same number after two weeks and always around 40
> segments. I ask our developers and they confirmed we use default settings
> for merging policy. Please, do you have some tips and tricks for efficient
> setting for merging segments?
>
> Our HW is RAID5 SAS disks and 40GB of memory dedicated for Solr.
>
> Regards,
> Pavel
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Optimize-SolrCloud-without-downtime-tp4195170p4196273.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Optimize SolrCloud without downtime

Posted by Pavel Hladik <pa...@profimedia.cz>.
Hi,

thanks for reply. We have a lot of deleted docs cause we have to reindex all
records from time to time, changing some important parameters.. When we do
update, it means create and delete.

Our deleted docs do not disappear by merging segments. I see our deleted
docs are almost the same number after two weeks and always around 40
segments. I ask our developers and they confirmed we use default settings
for merging policy. Please, do you have some tips and tricks for efficient
setting for merging segments?

Our HW is RAID5 SAS disks and 40GB of memory dedicated for Solr.

Regards,
Pavel





--
View this message in context: http://lucene.472066.n3.nabble.com/Optimize-SolrCloud-without-downtime-tp4195170p4196273.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Optimize SolrCloud without downtime

Posted by Erick Erickson <er...@gmail.com>.
That's a high number of deleted documents as a percentage of your
index! Or at least I find those numbers surprising. When segments are
merged in the background during normal indexing, quite a bit of weight
is given to segments that have a high percentage of deleted docs. I
usually see at most 10-20% of docs deleted.

So what kinds of things have you done to get into this state? Did you
optimize previously? Change the merge policy? Anything else?

Best,
Erick

On Wed, Mar 25, 2015 at 8:08 AM, pavelhladik <pa...@profimedia.cz> wrote:
> Hi,
>
> I didn't find the answer yet, please help. We have standalone Solr 5.0.0
> with a few cores yet. One of those cores contains:
>
> numDocs:120M
> deletedDocs:110M
>
> Our data are changing frequently so that's why so many deletedDocs.
> Optimized core takes around 50GB on disk, we are now almost on 100GB and I'm
> looking for best solution howto optimize this huge core without downtime. I
> know optimization working in background, but anyway when the optimization is
> running our search system is slow and sometimes I receive errors - this
> behavior is like a downtime for us.
>
> I would like to switch to SolrCloud, the performance is not a issue, so I
> don't need the sharding feature at this time. I'm more interested with
> replication and distribute requests by some Nginx proxy. Idea is:
>
> 1) proxy forward requests to node1 and optimize cores on node2
> 2) proxy forward requests to node2 and optimize cores on node1
>
> But when I do optimize on node2, the node1 is doing optimization as well,
> even if I use the "distrib=false" with curl.
>
> Can you please recommend architecture for optimizing without downtime? Many
> thanks.
>
> Pavel
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Optimize-SolrCloud-without-downtime-tp4195170.html
> Sent from the Solr - User mailing list archive at Nabble.com.