Question on changing block settings...

[Vidhyashankar Venkataraman <vi...@yahoo-inc.com>]

A few issues I have been observing on changing block settings:

 1.  What happens if we change the block size of a column family on an already populated database? Will this not throw apps on db out of whack because of compression and Hfile index which depend on block size?  So, once the db is populated, the user should be prevented from altering the block size..


   2.   I had disabled the block cache through Java API and through hbase shell, yet I get stats like the following: I had gone through this with Stack and he pointed out that this could be a bug:

2010-07-27 14:13:40,852 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1950.0773MB (2044804168), Free=449.7541MB (471601336), Max=2399.8313MB (2516405504), Counts: Blocks=1914, Access=1304510, Hit=1914, Miss=1302596, Evictions=3300, Evicted=788515, Ratios: Hit Ratio=0.1467217574827373%, Miss Ratio=99.8532772064209%, Evicted/Run=238.943939208984382010-07-27 14:14:40,860 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1950.0773MB (2044804168), Free=449.7541MB (471601336), Max=2399.8313MB (2516405504), Counts: Blocks=1914, Access=1306103, Hit=1914, Miss=1304189, Evictions=3300, Evicted=788515, Ratios: Hit Ratio=0.14654280385002494%, Miss Ratio=99.85345602035522%, Evicted/Run=238.943939208984382010-07-27 14:15:40,853 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1950.0773MB (2044804168), Free=449.7541MB (471601336), Max=2399.8313MB (2516405504), Counts: Blocks=1914, Access=1307783, Hit=1914, Miss=1305869, Evictions=3300, Evicted=788515, Ratios: Hit Ratio=0.14635456027463078%, Miss Ratio=99.8536467552185%, Evicted/Run=238.94393920898438

Vidhya

Re: Question on changing block settings...

[Jean-Daniel Cryans <jd...@apache.org>]

Inline.

J-D

On Tue, Aug 3, 2010 at 2:44 PM, Vidhyashankar Venkataraman
<vi...@yahoo-inc.com> wrote:
> A few issues I have been observing on changing block settings:
>
>  1.  What happens if we change the block size of a column family on an already populated database? Will this not throw apps on db out of whack because of compression and Hfile index which depend on block size?  So, once the db is populated, the user should be prevented from altering the block size..

You can, when the files are compacted they will be re-written with the
new block size. When you read files that weren't compacted, we just
read the index and figure the blocks so there's nothing wrong there
either. It works like that too when you open a file that isn't
compressed on a table that was just changed to be LZOed or else.

>
>
>   2.   I had disabled the block cache through Java API and through hbase shell, yet I get stats like the following: I had gone through this with Stack and he pointed out that this could be a bug:
>
> 2010-07-27 14:13:40,852 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1950.0773MB (2044804168), Free=449.7541MB (471601336), Max=2399.8313MB (2516405504), Counts: Blocks=1914, Access=1304510, Hit=1914, Miss=1302596, Evictions=3300, Evicted=788515, Ratios: Hit Ratio=0.1467217574827373%, Miss Ratio=99.8532772064209%, Evicted/Run=238.943939208984382010-07-27 14:14:40,860 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1950.0773MB (2044804168), Free=449.7541MB (471601336), Max=2399.8313MB (2516405504), Counts: Blocks=1914, Access=1306103, Hit=1914, Miss=1304189, Evictions=3300, Evicted=788515, Ratios: Hit Ratio=0.14654280385002494%, Miss Ratio=99.85345602035522%, Evicted/Run=238.943939208984382010-07-27 14:15:40,853 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=1950.0773MB (2044804168), Free=449.7541MB (471601336), Max=2399.8313MB (2516405504), Counts: Blocks=1914, Access=1307783, Hit=1914, Miss=1305869, Evictions=3300, Evicted=788515, Ratios: Hit Ratio=0.14635456027463078%, Miss Ratio=99.8536467552185%, Evicted/Run=238.94393920898438

If you didn't reboot the region server processes, then the stats won't
be cleared (since other tables could be using the block cache). Is it
the case?