How to warm up a cold node

[Héctor Izquierdo Seliva <iz...@strands.com>]

Hi everyone, is there any recommended procedure to warm up a node before
bringing it up? 

Thanks!

Re: How to warm up a cold node

[Héctor Izquierdo Seliva <iz...@strands.com>]

How difficult do you think this could be? I would be interested into
developing this if it's feasible.

El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió:
> > Hi everyone, is there any recommended procedure to warm up a node before
> > bringing it up?
> 
> Currently the only out-of-the-box support for warming up caches is
> that implied by the key cache and row cache, which will pre-heat on
> start-up. Indexes will be indirectly preheated by index sampling, to
> the extent that they operating system retains them in page cache.
> 
> If you're wanting to pre-heat sstables there's currently no way to do
> that (but it's a useful feature to have). Pragmatically, you can
> script something that e.g. does "cat path/to/keyspace/* > /dev/null"
> or similar. But that only works if the total database size fits
> reasonably well in page cache.
> 
> Pre-heating sstables on a per-cf basis on start-up would be a nice
> feature to have.
> 

Re: How to warm up a cold node

[Héctor Izquierdo Seliva <iz...@strands.com>]

El mié, 20-04-2011 a las 07:59 +1200, aaron morton escribió:
> The dynamic snitch only reduces the chance that a node used in a read operation, it depends on the RF, the CL for the operation, the partitioner and possibly the network topology. Dropping read messages is ok, so long as your operation completes at the requested CL.  
> 
> Are you using either a key_cache or a row_cache ? If so have you enabled the background save for those as part of the column definition?
> 
> If this is about getting the OS caches warmed up you should see the pending count on the READ stage backup up and the io stats start to show longer queues. In that case Peters suggestion below is prob the best option.
> 
> Another hacky way to warm things would be to use get_count() on rows the app thiks it may need to use. This will cause all the columns to be read from disk, but not sent over the network. 
> 
> Aaron

The rows that need to be readed change over time and they are only read
for short amounts of time, so it's a bit tricky. I'll try to figure out
a way to use your suggestions.

In another thread I asked if it would be feasible to somehow store what
parts of the sstables are hot on shutdown and re read those parts of the
files on startup. Could it be done in a similar way to the work that's
being done on page migrations? What do you think?

Thanks for your time!


>   
> On 20 Apr 2011, at 00:41, Héctor Izquierdo Seliva wrote:
> 
> > Shouldn't the dynamic snitch take into account response times and ask a
> > slow node for less requests? It seems that at node startup, only a
> > handfull of requests arrive to the node and it keeps up well, but
> > there's moment where there's more than it can handle with a cold cache
> > and starts droping messages like crazy.
> > 
> > Could it be the transition from slow node to ok node is to steep?
> > 
> > 
> > El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió:
> >>> Hi everyone, is there any recommended procedure to warm up a node before
> >>> bringing it up?
> >> 
> >> Currently the only out-of-the-box support for warming up caches is
> >> that implied by the key cache and row cache, which will pre-heat on
> >> start-up. Indexes will be indirectly preheated by index sampling, to
> >> the extent that they operating system retains them in page cache.
> >> 
> >> If you're wanting to pre-heat sstables there's currently no way to do
> >> that (but it's a useful feature to have). Pragmatically, you can
> >> script something that e.g. does "cat path/to/keyspace/* > /dev/null"
> >> or similar. But that only works if the total database size fits
> >> reasonably well in page cache.
> >> 
> >> Pre-heating sstables on a per-cf basis on start-up would be a nice
> >> feature to have.
> >> 
> > 
> > 
> 

Re: How to warm up a cold node

[aaron morton <aa...@thelastpickle.com>]

The dynamic snitch only reduces the chance that a node used in a read operation, it depends on the RF, the CL for the operation, the partitioner and possibly the network topology. Dropping read messages is ok, so long as your operation completes at the requested CL.  

Are you using either a key_cache or a row_cache ? If so have you enabled the background save for those as part of the column definition?

If this is about getting the OS caches warmed up you should see the pending count on the READ stage backup up and the io stats start to show longer queues. In that case Peters suggestion below is prob the best option.

Another hacky way to warm things would be to use get_count() on rows the app thiks it may need to use. This will cause all the columns to be read from disk, but not sent over the network. 

Aaron
  
On 20 Apr 2011, at 00:41, Héctor Izquierdo Seliva wrote:

> Shouldn't the dynamic snitch take into account response times and ask a
> slow node for less requests? It seems that at node startup, only a
> handfull of requests arrive to the node and it keeps up well, but
> there's moment where there's more than it can handle with a cold cache
> and starts droping messages like crazy.
> 
> Could it be the transition from slow node to ok node is to steep?
> 
> 
> El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió:
>>> Hi everyone, is there any recommended procedure to warm up a node before
>>> bringing it up?
>> 
>> Currently the only out-of-the-box support for warming up caches is
>> that implied by the key cache and row cache, which will pre-heat on
>> start-up. Indexes will be indirectly preheated by index sampling, to
>> the extent that they operating system retains them in page cache.
>> 
>> If you're wanting to pre-heat sstables there's currently no way to do
>> that (but it's a useful feature to have). Pragmatically, you can
>> script something that e.g. does "cat path/to/keyspace/* > /dev/null"
>> or similar. But that only works if the total database size fits
>> reasonably well in page cache.
>> 
>> Pre-heating sstables on a per-cf basis on start-up would be a nice
>> feature to have.
>> 
> 
> 

Re: How to warm up a cold node

[Héctor Izquierdo Seliva <iz...@strands.com>]

Shouldn't the dynamic snitch take into account response times and ask a
slow node for less requests? It seems that at node startup, only a
handfull of requests arrive to the node and it keeps up well, but
there's moment where there's more than it can handle with a cold cache
and starts droping messages like crazy.

Could it be the transition from slow node to ok node is to steep?


El vie, 15-04-2011 a las 16:19 +0200, Peter Schuller escribió:
> > Hi everyone, is there any recommended procedure to warm up a node before
> > bringing it up?
> 
> Currently the only out-of-the-box support for warming up caches is
> that implied by the key cache and row cache, which will pre-heat on
> start-up. Indexes will be indirectly preheated by index sampling, to
> the extent that they operating system retains them in page cache.
> 
> If you're wanting to pre-heat sstables there's currently no way to do
> that (but it's a useful feature to have). Pragmatically, you can
> script something that e.g. does "cat path/to/keyspace/* > /dev/null"
> or similar. But that only works if the total database size fits
> reasonably well in page cache.
> 
> Pre-heating sstables on a per-cf basis on start-up would be a nice
> feature to have.
> 

Re: How to warm up a cold node

[Peter Schuller <pe...@infidyne.com>]

> Hi everyone, is there any recommended procedure to warm up a node before
> bringing it up?

Currently the only out-of-the-box support for warming up caches is
that implied by the key cache and row cache, which will pre-heat on
start-up. Indexes will be indirectly preheated by index sampling, to
the extent that they operating system retains them in page cache.

If you're wanting to pre-heat sstables there's currently no way to do
that (but it's a useful feature to have). Pragmatically, you can
script something that e.g. does "cat path/to/keyspace/* > /dev/null"
or similar. But that only works if the total database size fits
reasonably well in page cache.

Pre-heating sstables on a per-cf basis on start-up would be a nice
feature to have.

-- 
/ Peter Schuller