network spike every 2 hours

[Jeff Whiting <je...@qualtrics.com>]

We have a hadoop cluster that is only running hbase.  We recently installed ganglia to monitor those 
servers.  Looking over the network graphs about every 2 hours there is a very large spike.  Before 
the spike we are pushing about 5mb but during the spikes it is around 60-100mb.  I'm pretty sure we 
aren't running anything, is hbase doing something that could cause those spikes?

Thanks,
~Jeff

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com

Re: network spike every 2 hours

[Jean-Daniel Cryans <jd...@apache.org>]

How major compactions are run is a bit dumb, just every 24h and it's
done via a counter on each region. If you don't need them to run
(let's say you don't delete stuff that much), then you can disable
them and cron it to run during low traffic periods. An easy script
would be:

echo "major_compact 'my_table' " | /path/to/hbase/bin/hbase shell

J-D

On Tue, Nov 22, 2011 at 4:12 PM, Jeff Whiting <je...@qualtrics.com> wrote:
> bingo that was it, thanks!  The compaction queue spikes at the same time.  I
> also see the rpc metrics fluctuate with the major compactions (they don't go
> unreasonably high but are several times slower).  Is there any way to even
> those out? Or prevent any compactions from impacting the rpc calls?
>
> ~Jeff
>
> On 11/22/2011 5:00 PM, Jean-Daniel Cryans wrote:
>>
>> Look for the compaction queue metrics, see if it fits. Could be major
>> compactions.
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>

Re: network spike every 2 hours

[Jeff Whiting <je...@qualtrics.com>]

bingo that was it, thanks!  The compaction queue spikes at the same time.  I also see the rpc 
metrics fluctuate with the major compactions (they don't go unreasonably high but are several times 
slower).  Is there any way to even those out? Or prevent any compactions from impacting the rpc calls?

~Jeff

On 11/22/2011 5:00 PM, Jean-Daniel Cryans wrote:
> Look for the compaction queue metrics, see if it fits. Could be major
> compactions.

-- 
Jeff Whiting
Qualtrics Senior Software Engineer
jeffw@qualtrics.com

Re: network spike every 2 hours

[Jean-Daniel Cryans <jd...@apache.org>]

Look for the compaction queue metrics, see if it fits. Could be major
compactions.

J-D

On Tue, Nov 22, 2011 at 3:55 PM, Jeff Whiting <je...@qualtrics.com> wrote:
> We have a hadoop cluster that is only running hbase.  We recently installed
> ganglia to monitor those servers.  Looking over the network graphs about
> every 2 hours there is a very large spike.  Before the spike we are pushing
> about 5mb but during the spikes it is around 60-100mb.  I'm pretty sure we
> aren't running anything, is hbase doing something that could cause those
> spikes?
>
> Thanks,
> ~Jeff
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> jeffw@qualtrics.com
>
>