You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jean-Sebastien Vachon <je...@wantedanalytics.com> on 2013/08/20 20:04:21 UTC
Flushing cache without restarting everything?
Hi All,
Is there a way to flush the cache of all nodes in a Solr Cloud (by reloading all the cores, through the collection API, ...) without having to restart all nodes?
Thanks
Re: Flushing cache without restarting everything?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2013-08-22 at 20:08 +0200, Walter Underwood wrote:
> We warm the file buffers before starting Solr to avoid spending time
> waiting for disk IO. The script is something like this:
>
> for core in core1 core2 core3
> do
> find /apps/solr/data/${core}/index -type f | xargs cat > /dev/null
> done
On that subject, I will note that the reason for not just doing a simple
"cp <index_files> > /dev/null" is that the shell (sometimes? always?) is
clever enough to skip the copying when it is done directly to /dev/null.
> It makes a big difference in the first few minutes of service. Of
> course, it helps if you have enough RAM to hold the entire index.
That is actually essential if you want to perform a reproducible test.
If there is not enough free RAM for the full amount of index files, "the
first" files from find will not be cached at all and if the index has
changed since last test, that makes it pretty arbitrary which parts are
cached and which are not.
--
Ceterum censeo spinning drives esse delendam
Re: Flushing cache without restarting everything?
Posted by Walter Underwood <wu...@wunderwood.org>.
We warm the file buffers before starting Solr to avoid spending time waiting for disk IO. The script is something like this:
for core in core1 core2 core3
do
find /apps/solr/data/${core}/index -type f | xargs cat > /dev/null
done
It makes a big difference in the first few minutes of service. Of course, it helps if you have enough RAM to hold the entire index.
wunder
On Aug 22, 2013, at 10:28 AM, Jean-Sebastien Vachon wrote:
> I was afraid someone would tell me that... thanks for your input
>
>> -----Original Message-----
>> From: Toke Eskildsen [mailto:te@statsbiblioteket.dk]
>> Sent: August-22-13 9:56 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Flushing cache without restarting everything?
>>
>> On Tue, 2013-08-20 at 20:04 +0200, Jean-Sebastien Vachon wrote:
>>> Is there a way to flush the cache of all nodes in a Solr Cloud (by
>>> reloading all the cores, through the collection API, ...) without
>>> having to restart all nodes?
>>
>> As MMapDirectory shares data with the OS disk cache, flushing of
>> Solr-related caches on a machine should involve
>>
>> 1) Shut down all Solr instances on the machine
>> 2) Clear the OS read cache ('sudo echo 1 > /proc/sys/vm/drop_caches' on
>> a Linux box)
>> 3) Start the Solr instances
>>
>> I do not know of any Solr-supported way to do step 2. For our
>> performance tests we use custom scripts to perform the steps.
>>
>> - Toke Eskildsen, State and University Library, Denmark
>>
>>
>> -----
>> Aucun virus trouvé dans ce message.
>> Analyse effectuée par AVG - www.avg.fr
>> Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date: 09/08/2013
>> La Base de données des virus a expiré.
--
Walter Underwood
wunder@wunderwood.org
Re: Flushing cache without restarting everything?
Posted by Dan Davis <da...@gmail.com>.
be careful with drop_caches - make sure you sync first
On Thu, Aug 22, 2013 at 1:28 PM, Jean-Sebastien Vachon <
jean-sebastien.vachon@wantedanalytics.com> wrote:
> I was afraid someone would tell me that... thanks for your input
>
> > -----Original Message-----
> > From: Toke Eskildsen [mailto:te@statsbiblioteket.dk]
> > Sent: August-22-13 9:56 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Flushing cache without restarting everything?
> >
> > On Tue, 2013-08-20 at 20:04 +0200, Jean-Sebastien Vachon wrote:
> > > Is there a way to flush the cache of all nodes in a Solr Cloud (by
> > > reloading all the cores, through the collection API, ...) without
> > > having to restart all nodes?
> >
> > As MMapDirectory shares data with the OS disk cache, flushing of
> > Solr-related caches on a machine should involve
> >
> > 1) Shut down all Solr instances on the machine
> > 2) Clear the OS read cache ('sudo echo 1 > /proc/sys/vm/drop_caches' on
> > a Linux box)
> > 3) Start the Solr instances
> >
> > I do not know of any Solr-supported way to do step 2. For our
> > performance tests we use custom scripts to perform the steps.
> >
> > - Toke Eskildsen, State and University Library, Denmark
> >
> >
> > -----
> > Aucun virus trouvé dans ce message.
> > Analyse effectuée par AVG - www.avg.fr
> > Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date:
> 09/08/2013
> > La Base de données des virus a expiré.
>
RE: Flushing cache without restarting everything?
Posted by Jean-Sebastien Vachon <je...@wantedanalytics.com>.
I was afraid someone would tell me that... thanks for your input
> -----Original Message-----
> From: Toke Eskildsen [mailto:te@statsbiblioteket.dk]
> Sent: August-22-13 9:56 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Flushing cache without restarting everything?
>
> On Tue, 2013-08-20 at 20:04 +0200, Jean-Sebastien Vachon wrote:
> > Is there a way to flush the cache of all nodes in a Solr Cloud (by
> > reloading all the cores, through the collection API, ...) without
> > having to restart all nodes?
>
> As MMapDirectory shares data with the OS disk cache, flushing of
> Solr-related caches on a machine should involve
>
> 1) Shut down all Solr instances on the machine
> 2) Clear the OS read cache ('sudo echo 1 > /proc/sys/vm/drop_caches' on
> a Linux box)
> 3) Start the Solr instances
>
> I do not know of any Solr-supported way to do step 2. For our
> performance tests we use custom scripts to perform the steps.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
> -----
> Aucun virus trouvé dans ce message.
> Analyse effectuée par AVG - www.avg.fr
> Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date: 09/08/2013
> La Base de données des virus a expiré.
Re: Flushing cache without restarting everything?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Tue, 2013-08-20 at 20:04 +0200, Jean-Sebastien Vachon wrote:
> Is there a way to flush the cache of all nodes in a Solr Cloud (by
> reloading all the cores, through the collection API, ...) without
> having to restart all nodes?
As MMapDirectory shares data with the OS disk cache, flushing of
Solr-related caches on a machine should involve
1) Shut down all Solr instances on the machine
2) Clear the OS read cache ('sudo echo 1 > /proc/sys/vm/drop_caches' on
a Linux box)
3) Start the Solr instances
I do not know of any Solr-supported way to do step 2. For our
performance tests we use custom scripts to perform the steps.
- Toke Eskildsen, State and University Library, Denmark
Re: Flushing cache without restarting everything?
Posted by Dmitry Kan <so...@gmail.com>.
by monitoring the original and changed systems over long enough periods,
where "long enough" is a parameter (to compute).
Or then going really low-level, if you know which component has been
changed (like they do in Lucene [1]; not always possible in Solr..)
[1] http://people.apache.org/~mikemccand/lucenebench/
On Thu, Aug 22, 2013 at 3:58 PM, Jean-Sebastien Vachon <
jean-sebastien.vachon@wantedanalytics.com> wrote:
> How can you validate that the changes you just made had any impact on the
> performance of the cloud if you don't have the same starting conditions?
>
> What we do basically is running a batch of requests to warm up the index
> and then launch the benchmark itself. That way we can measure the impact of
> our change(s). Otherwise there is absolutely no way we can be sure who is
> responsible for the gain or loss of performance.
>
> Restarting a cloud is actually a real pain, I just want to know if there
> is a faster way to proceed.
>
> > -----Original Message-----
> > From: Dmitry Kan [mailto:solrexpert@gmail.com]
> > Sent: August-22-13 7:26 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Flushing cache without restarting everything?
> >
> > But is it really a good benchmarking, if you flush the cache? Wouldn't
> you
> > want to benchmark against a system, that would be comparable to what is
> > under real (=production) load?
> >
> > Dmitry
> >
> >
> > On Tue, Aug 20, 2013 at 9:39 PM, Jean-Sebastien Vachon < jean-
> > sebastien.vachon@wantedanalytics.com> wrote:
> >
> > > I just want to run benchmarks and want to have the same starting
> > > conditions.
> > >
> > > > -----Original Message-----
> > > > From: Walter Underwood [mailto:wunder@wunderwood.org]
> > > > Sent: August-20-13 2:06 PM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: Flushing cache without restarting everything?
> > > >
> > > > Why? What are you trying to acheive with this? --wunder
> > > >
> > > > On Aug 20, 2013, at 11:04 AM, Jean-Sebastien Vachon wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Is there a way to flush the cache of all nodes in a Solr Cloud (by
> > > reloading all
> > > > the cores, through the collection API, ...) without having to
> > > > restart
> > > all nodes?
> > > > >
> > > > > Thanks
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > -----
> > > > Aucun virus trouvé dans ce message.
> > > > Analyse effectuée par AVG - www.avg.fr
> > > > Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date:
> > > 09/08/2013
> > > > La Base de données des virus a expiré.
> > >
> >
> > -----
> > Aucun virus trouvé dans ce message.
> > Analyse effectuée par AVG - www.avg.fr
> > Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date:
> 09/08/2013
> > La Base de données des virus a expiré.
>
RE: Flushing cache without restarting everything?
Posted by Jean-Sebastien Vachon <je...@wantedanalytics.com>.
How can you validate that the changes you just made had any impact on the performance of the cloud if you don't have the same starting conditions?
What we do basically is running a batch of requests to warm up the index and then launch the benchmark itself. That way we can measure the impact of our change(s). Otherwise there is absolutely no way we can be sure who is responsible for the gain or loss of performance.
Restarting a cloud is actually a real pain, I just want to know if there is a faster way to proceed.
> -----Original Message-----
> From: Dmitry Kan [mailto:solrexpert@gmail.com]
> Sent: August-22-13 7:26 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Flushing cache without restarting everything?
>
> But is it really a good benchmarking, if you flush the cache? Wouldn't you
> want to benchmark against a system, that would be comparable to what is
> under real (=production) load?
>
> Dmitry
>
>
> On Tue, Aug 20, 2013 at 9:39 PM, Jean-Sebastien Vachon < jean-
> sebastien.vachon@wantedanalytics.com> wrote:
>
> > I just want to run benchmarks and want to have the same starting
> > conditions.
> >
> > > -----Original Message-----
> > > From: Walter Underwood [mailto:wunder@wunderwood.org]
> > > Sent: August-20-13 2:06 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Flushing cache without restarting everything?
> > >
> > > Why? What are you trying to acheive with this? --wunder
> > >
> > > On Aug 20, 2013, at 11:04 AM, Jean-Sebastien Vachon wrote:
> > >
> > > > Hi All,
> > > >
> > > > Is there a way to flush the cache of all nodes in a Solr Cloud (by
> > reloading all
> > > the cores, through the collection API, ...) without having to
> > > restart
> > all nodes?
> > > >
> > > > Thanks
> > >
> > >
> > >
> > >
> > >
> > > -----
> > > Aucun virus trouvé dans ce message.
> > > Analyse effectuée par AVG - www.avg.fr
> > > Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date:
> > 09/08/2013
> > > La Base de données des virus a expiré.
> >
>
> -----
> Aucun virus trouvé dans ce message.
> Analyse effectuée par AVG - www.avg.fr
> Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date: 09/08/2013
> La Base de données des virus a expiré.
Re: Flushing cache without restarting everything?
Posted by Dmitry Kan <so...@gmail.com>.
But is it really a good benchmarking, if you flush the cache? Wouldn't you
want to benchmark against a system, that would be comparable to what is
under real (=production) load?
Dmitry
On Tue, Aug 20, 2013 at 9:39 PM, Jean-Sebastien Vachon <
jean-sebastien.vachon@wantedanalytics.com> wrote:
> I just want to run benchmarks and want to have the same starting
> conditions.
>
> > -----Original Message-----
> > From: Walter Underwood [mailto:wunder@wunderwood.org]
> > Sent: August-20-13 2:06 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Flushing cache without restarting everything?
> >
> > Why? What are you trying to acheive with this? --wunder
> >
> > On Aug 20, 2013, at 11:04 AM, Jean-Sebastien Vachon wrote:
> >
> > > Hi All,
> > >
> > > Is there a way to flush the cache of all nodes in a Solr Cloud (by
> reloading all
> > the cores, through the collection API, ...) without having to restart
> all nodes?
> > >
> > > Thanks
> >
> >
> >
> >
> >
> > -----
> > Aucun virus trouvé dans ce message.
> > Analyse effectuée par AVG - www.avg.fr
> > Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date:
> 09/08/2013
> > La Base de données des virus a expiré.
>
RE: Flushing cache without restarting everything?
Posted by Jean-Sebastien Vachon <je...@wantedanalytics.com>.
I just want to run benchmarks and want to have the same starting conditions.
> -----Original Message-----
> From: Walter Underwood [mailto:wunder@wunderwood.org]
> Sent: August-20-13 2:06 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Flushing cache without restarting everything?
>
> Why? What are you trying to acheive with this? --wunder
>
> On Aug 20, 2013, at 11:04 AM, Jean-Sebastien Vachon wrote:
>
> > Hi All,
> >
> > Is there a way to flush the cache of all nodes in a Solr Cloud (by reloading all
> the cores, through the collection API, ...) without having to restart all nodes?
> >
> > Thanks
>
>
>
>
>
> -----
> Aucun virus trouvé dans ce message.
> Analyse effectuée par AVG - www.avg.fr
> Version: 2013.0.3392 / Base de données virale: 3209/6563 - Date: 09/08/2013
> La Base de données des virus a expiré.
Re: Flushing cache without restarting everything?
Posted by Walter Underwood <wu...@wunderwood.org>.
Why? What are you trying to acheive with this? --wunder
On Aug 20, 2013, at 11:04 AM, Jean-Sebastien Vachon wrote:
> Hi All,
>
> Is there a way to flush the cache of all nodes in a Solr Cloud (by reloading all the cores, through the collection API, ...) without having to restart all nodes?
>
> Thanks