You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ganesh Sethuraman <ga...@gmail.com> on 2018/09/18 17:00:29 UTC
Solr 7.2.1 Collection Backup Performance issue
Hi
We are using Solr 7.2.1 with SolrCloud with 35 collections with 1 node ZK
ensemble (in lower environment, we will have 3 nodes ensemble) in AWS. We
are testing to see if we have Async Solr Cloud backup (
https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup) done
every time we are create a new collection or update an existing collection.
There are 1 replica and 8 shards per collection. Two Solr nodes.
For the largest collection (index size of 80GB), we see that BACKUP to the
EFS drive takes about ~10 mins. We are doing lot of /get (real time get)
option from the application. We are seeing that that the performance
significantly (2x) degrades on the read (get) performance when we BACK-UP
is going on in parallel.
Is there anyway to tune the system so that read does not suffer?
Any other best practices? like should we run back up during off peak load?
Is there a way to keep track of which collections are already backed up?
Re: Solr 7.2.1 Collection Backup Performance issue
Posted by Walter Underwood <wu...@wunderwood.org>.
I don’t know how well it worked, but for a while, I did this to warm up the file buffers.
It should be OK if RAM is bigger than data. Though “cat” probably opens the files with
the hint that it will never re-read the data.
find /solr-data-dir -type f | xargs cat > /dev/null
Basically, read every file with the “cat” program, which should load it into OS file buffers.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 21, 2018, at 12:53 PM, Ganesh Sethuraman <ga...@gmail.com> wrote:
>
> We don't have all the index size fit in into memory, but we still have an
> acceptable performance as of now for reads/query. But with BACKUP we are
> seeing a increase in the OS memory usage. Given that, I am sure many of
> system might be running with less memory but good enough for their
> application. But BACKUP is changing this equation now. Are there any best
> practices to do backup during off peak hours? and do some kind of warm-up?
> (if so how to warm up)
>
>
> On Tue, Sep 18, 2018 at 5:48 PM Ganesh Sethuraman <ga...@gmail.com>
> wrote:
>
>> Thanks for the information. I thought backup is going to be more of the
>> disk activity. But I understand now that RAM is involved here as well. We
>> indeed did NOT have enough memory in this box, as it is 64GB box with index
>> size of 72GB, being backed up. The read (real time GET) performance was
>> better without BACKUP, could be because there was minimal disk access, but
>> with Backup running, reads (GET) are probably doing disk read for every
>> request.
>>
>> Thanks,
>> Ganesh
>>
>> On Tue, Sep 18, 2018 at 3:43 PM Shawn Heisey <ap...@elyograg.org> wrote:
>>
>>> On 9/18/2018 11:00 AM, Ganesh Sethuraman wrote:
>>>> We are using Solr 7.2.1 with SolrCloud with 35 collections with 1 node
>>> ZK
>>>> ensemble (in lower environment, we will have 3 nodes ensemble) in AWS.
>>> We
>>>> are testing to see if we have Async Solr Cloud backup (
>>>> https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup)
>>> done
>>>> every time we are create a new collection or update an existing
>>> collection.
>>>> There are 1 replica and 8 shards per collection. Two Solr nodes.
>>>>
>>>> For the largest collection (index size of 80GB), we see that BACKUP to
>>> the
>>>> EFS drive takes about ~10 mins. We are doing lot of /get (real time get)
>>>> option from the application. We are seeing that that the performance
>>>> significantly (2x) degrades on the read (get) performance when we
>>> BACK-UP
>>>> is going on in parallel.
>>>
>>> My best guess here is that you do not have enough memory. For good
>>> performance, Solr is extremely reliant on having certain parts of the
>>> index data sitting in memory, so that it doesn't have to actually read
>>> the disk to discover matches for a query. When all is working well,
>>> that data will be read from memory instead of the disk. Memory is MUCH
>>> MUCH faster than a disk.
>>>
>>> Making a backup is going to read ALL of the index data. So if you do
>>> not have enough spare memory to cache the entire index, reading the
>>> index to make the backup is going to push the important parts of the
>>> index out of the cache, and then Solr will have to actually go and read
>>> the disk in order to satisfy a query.
>>>
>>> https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
>>>
>>> Can you gather a screenshot of your process list and put it on a file
>>> sharing website? You'll find instructions on how to do this here:
>>>
>>>
>>> https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
Re: Solr 7.2.1 Collection Backup Performance issue
Posted by Ganesh Sethuraman <ga...@gmail.com>.
We don't have all the index size fit in into memory, but we still have an
acceptable performance as of now for reads/query. But with BACKUP we are
seeing a increase in the OS memory usage. Given that, I am sure many of
system might be running with less memory but good enough for their
application. But BACKUP is changing this equation now. Are there any best
practices to do backup during off peak hours? and do some kind of warm-up?
(if so how to warm up)
On Tue, Sep 18, 2018 at 5:48 PM Ganesh Sethuraman <ga...@gmail.com>
wrote:
> Thanks for the information. I thought backup is going to be more of the
> disk activity. But I understand now that RAM is involved here as well. We
> indeed did NOT have enough memory in this box, as it is 64GB box with index
> size of 72GB, being backed up. The read (real time GET) performance was
> better without BACKUP, could be because there was minimal disk access, but
> with Backup running, reads (GET) are probably doing disk read for every
> request.
>
> Thanks,
> Ganesh
>
> On Tue, Sep 18, 2018 at 3:43 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 9/18/2018 11:00 AM, Ganesh Sethuraman wrote:
>> > We are using Solr 7.2.1 with SolrCloud with 35 collections with 1 node
>> ZK
>> > ensemble (in lower environment, we will have 3 nodes ensemble) in AWS.
>> We
>> > are testing to see if we have Async Solr Cloud backup (
>> > https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup)
>> done
>> > every time we are create a new collection or update an existing
>> collection.
>> > There are 1 replica and 8 shards per collection. Two Solr nodes.
>> >
>> > For the largest collection (index size of 80GB), we see that BACKUP to
>> the
>> > EFS drive takes about ~10 mins. We are doing lot of /get (real time get)
>> > option from the application. We are seeing that that the performance
>> > significantly (2x) degrades on the read (get) performance when we
>> BACK-UP
>> > is going on in parallel.
>>
>> My best guess here is that you do not have enough memory. For good
>> performance, Solr is extremely reliant on having certain parts of the
>> index data sitting in memory, so that it doesn't have to actually read
>> the disk to discover matches for a query. When all is working well,
>> that data will be read from memory instead of the disk. Memory is MUCH
>> MUCH faster than a disk.
>>
>> Making a backup is going to read ALL of the index data. So if you do
>> not have enough spare memory to cache the entire index, reading the
>> index to make the backup is going to push the important parts of the
>> index out of the cache, and then Solr will have to actually go and read
>> the disk in order to satisfy a query.
>>
>> https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
>>
>> Can you gather a screenshot of your process list and put it on a file
>> sharing website? You'll find instructions on how to do this here:
>>
>>
>> https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue
>>
>> Thanks,
>> Shawn
>>
>>
Re: Solr 7.2.1 Collection Backup Performance issue
Posted by Ganesh Sethuraman <ga...@gmail.com>.
Thanks for the information. I thought backup is going to be more of the
disk activity. But I understand now that RAM is involved here as well. We
indeed did NOT have enough memory in this box, as it is 64GB box with index
size of 72GB, being backed up. The read (real time GET) performance was
better without BACKUP, could be because there was minimal disk access, but
with Backup running, reads (GET) are probably doing disk read for every
request.
Thanks,
Ganesh
On Tue, Sep 18, 2018 at 3:43 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 9/18/2018 11:00 AM, Ganesh Sethuraman wrote:
> > We are using Solr 7.2.1 with SolrCloud with 35 collections with 1 node ZK
> > ensemble (in lower environment, we will have 3 nodes ensemble) in AWS. We
> > are testing to see if we have Async Solr Cloud backup (
> > https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup)
> done
> > every time we are create a new collection or update an existing
> collection.
> > There are 1 replica and 8 shards per collection. Two Solr nodes.
> >
> > For the largest collection (index size of 80GB), we see that BACKUP to
> the
> > EFS drive takes about ~10 mins. We are doing lot of /get (real time get)
> > option from the application. We are seeing that that the performance
> > significantly (2x) degrades on the read (get) performance when we BACK-UP
> > is going on in parallel.
>
> My best guess here is that you do not have enough memory. For good
> performance, Solr is extremely reliant on having certain parts of the
> index data sitting in memory, so that it doesn't have to actually read
> the disk to discover matches for a query. When all is working well,
> that data will be read from memory instead of the disk. Memory is MUCH
> MUCH faster than a disk.
>
> Making a backup is going to read ALL of the index data. So if you do
> not have enough spare memory to cache the entire index, reading the
> index to make the backup is going to push the important parts of the
> index out of the cache, and then Solr will have to actually go and read
> the disk in order to satisfy a query.
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
>
> Can you gather a screenshot of your process list and put it on a file
> sharing website? You'll find instructions on how to do this here:
>
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue
>
> Thanks,
> Shawn
>
>
Re: Solr 7.2.1 Collection Backup Performance issue
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/18/2018 11:00 AM, Ganesh Sethuraman wrote:
> We are using Solr 7.2.1 with SolrCloud with 35 collections with 1 node ZK
> ensemble (in lower environment, we will have 3 nodes ensemble) in AWS. We
> are testing to see if we have Async Solr Cloud backup (
> https://lucene.apache.org/solr/guide/7_2/collections-api.html#backup) done
> every time we are create a new collection or update an existing collection.
> There are 1 replica and 8 shards per collection. Two Solr nodes.
>
> For the largest collection (index size of 80GB), we see that BACKUP to the
> EFS drive takes about ~10 mins. We are doing lot of /get (real time get)
> option from the application. We are seeing that that the performance
> significantly (2x) degrades on the read (get) performance when we BACK-UP
> is going on in parallel.
My best guess here is that you do not have enough memory. For good
performance, Solr is extremely reliant on having certain parts of the
index data sitting in memory, so that it doesn't have to actually read
the disk to discover matches for a query. When all is working well,
that data will be read from memory instead of the disk. Memory is MUCH
MUCH faster than a disk.
Making a backup is going to read ALL of the index data. So if you do
not have enough spare memory to cache the entire index, reading the
index to make the backup is going to push the important parts of the
index out of the cache, and then Solr will have to actually go and read
the disk in order to satisfy a query.
https://wiki.apache.org/solr/SolrPerformanceProblems#RAM
Can you gather a screenshot of your process list and put it on a file
sharing website? You'll find instructions on how to do this here:
https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue
Thanks,
Shawn