You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Liu, Ming (HPIT-GADSC)" <mi...@hp.com> on 2014/11/20 17:05:52 UTC
how to explain read/write performance change after modifying the
hfile.block.cache.size?
Hello, all,
I am playing with YCSB to test HBase performance. I am using HBase 0.98.5. I tried to adjust the hfile.block.cache.size to see the difference, when I set hfile.block.cache.size to 0, read performance is very bad, but write performance is very very very good....; when I set hfile.block.cache.size to 0.4, read is better, but write performance drop dramatically. I disable the client side writebuffer already.
This is hard to understand for me:
The HBase guide just said hfile.block.cache.size setting is about how much memory used as block cache used by StoreFile. I have no idea of how HBase works internally. Typically, it is easy to understand that increase the size of cache should help the read, but why it will harm the write operation? The write performance down from 30,000 to 4,000 for your reference, just by changing the hfile.block.cache.size from 0 to 0.4.
Could anyone give me a brief explanation about this observation or give me some advices about what to study to understand what is block cache used for?
Another question: HBase write will first write to WAL then to memstore. Will the write to WAL go to disk directly before hbase write memstore, a sync operation or it is possible that write to WAL is still buffered somewhere when hbase put the data into the memstore?
Reading src code may cost me months, so a kindly reply will help me a lot... ...
Thanks very much!
Best Regards,
Ming
Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
Posted by Kevin O'dell <ke...@cloudera.com>.
until you kill all the cache right? Or was this an old JIRA I was thinking
of?
On Thu, Nov 20, 2014 at 3:37 PM, Ted Yu <yu...@gmail.com> wrote:
> The indices are always cached.
>
> Cheers
>
> On Nov 20, 2014, at 12:33 PM, "Kevin O'dell" <ke...@cloudera.com>
> wrote:
>
> > I am also under the impression that HBase reads should basically not work
> > with block cache set to 0 since we store the indexes in block cache
> right?
> >
> > On Thu, Nov 20, 2014 at 3:31 PM, lars hofhansl <la...@apache.org> wrote:
> >
> >> That would explain it if memstores are flushed due to global memory
> >> pressure.
> >>
> >> But cache and memstore size are (unfortunately) configured
> independently.
> >> The memstore heap portion would be 40% (by default) in either case.So
> this
> >> is a bit curious still.
> >> Ming, can you tell us more details?- RAM on the boxes- heap setup for
> the
> >> region servers- any other relevant settings on hbase-site.xml- configs
> on
> >> the table/column family you're writing to (like bloom filters, etc).
> >>
> >> That would help us diagnose this.
> >>
> >> -- Lars
> >>
> >> From: Ted Yu <yu...@gmail.com>
> >> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> >> Sent: Thursday, November 20, 2014 9:32 AM
> >> Subject: Re: how to explain read/write performance change after
> modifying
> >> the hfile.block.cache.size?
> >>
> >> When block cache size increases from 0 to 0.4, the amount of heap given
> to
> >> memstore decreases. This would slow down the writes.
> >> Please see:
> >> http://hbase.apache.org/book.html#store.memstore
> >>
> >> For your second question, see this thread:
> >>
> >>
> http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
> >>
> >> Cheers
> >>
> >> On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <
> ming.liu2@hp.com>
> >> wrote:
> >>
> >>> Hello, all,
> >>>
> >>> I am playing with YCSB to test HBase performance. I am using HBase
> >> 0.98.5.
> >>> I tried to adjust the hfile.block.cache.size to see the difference,
> when
> >> I
> >>> set hfile.block.cache.size to 0, read performance is very bad, but
> write
> >>> performance is very very very good....; when I set
> hfile.block.cache.size
> >>> to 0.4, read is better, but write performance drop dramatically. I
> >> disable
> >>> the client side writebuffer already.
> >>> This is hard to understand for me:
> >>> The HBase guide just said hfile.block.cache.size setting is about how
> >> much
> >>> memory used as block cache used by StoreFile. I have no idea of how
> HBase
> >>> works internally. Typically, it is easy to understand that increase the
> >>> size of cache should help the read, but why it will harm the write
> >>> operation? The write performance down from 30,000 to 4,000 for your
> >>> reference, just by changing the hfile.block.cache.size from 0 to 0.4.
> >>> Could anyone give me a brief explanation about this observation or give
> >> me
> >>> some advices about what to study to understand what is block cache used
> >> for?
> >>>
> >>> Another question: HBase write will first write to WAL then to memstore.
> >>> Will the write to WAL go to disk directly before hbase write memstore,
> a
> >>> sync operation or it is possible that write to WAL is still buffered
> >>> somewhere when hbase put the data into the memstore?
> >>>
> >>> Reading src code may cost me months, so a kindly reply will help me a
> >>> lot... ...
> >>> Thanks very much!
> >>>
> >>> Best Regards,
> >>> Ming
> >
> >
> >
> > --
> > Kevin O'Dell
> > Systems Engineer, Cloudera
>
--
Kevin O'Dell
Systems Engineer, Cloudera
Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
Posted by Ted Yu <yu...@gmail.com>.
The indices are always cached.
Cheers
On Nov 20, 2014, at 12:33 PM, "Kevin O'dell" <ke...@cloudera.com> wrote:
> I am also under the impression that HBase reads should basically not work
> with block cache set to 0 since we store the indexes in block cache right?
>
> On Thu, Nov 20, 2014 at 3:31 PM, lars hofhansl <la...@apache.org> wrote:
>
>> That would explain it if memstores are flushed due to global memory
>> pressure.
>>
>> But cache and memstore size are (unfortunately) configured independently.
>> The memstore heap portion would be 40% (by default) in either case.So this
>> is a bit curious still.
>> Ming, can you tell us more details?- RAM on the boxes- heap setup for the
>> region servers- any other relevant settings on hbase-site.xml- configs on
>> the table/column family you're writing to (like bloom filters, etc).
>>
>> That would help us diagnose this.
>>
>> -- Lars
>>
>> From: Ted Yu <yu...@gmail.com>
>> To: "user@hbase.apache.org" <us...@hbase.apache.org>
>> Sent: Thursday, November 20, 2014 9:32 AM
>> Subject: Re: how to explain read/write performance change after modifying
>> the hfile.block.cache.size?
>>
>> When block cache size increases from 0 to 0.4, the amount of heap given to
>> memstore decreases. This would slow down the writes.
>> Please see:
>> http://hbase.apache.org/book.html#store.memstore
>>
>> For your second question, see this thread:
>>
>> http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
>>
>> Cheers
>>
>> On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
>> wrote:
>>
>>> Hello, all,
>>>
>>> I am playing with YCSB to test HBase performance. I am using HBase
>> 0.98.5.
>>> I tried to adjust the hfile.block.cache.size to see the difference, when
>> I
>>> set hfile.block.cache.size to 0, read performance is very bad, but write
>>> performance is very very very good....; when I set hfile.block.cache.size
>>> to 0.4, read is better, but write performance drop dramatically. I
>> disable
>>> the client side writebuffer already.
>>> This is hard to understand for me:
>>> The HBase guide just said hfile.block.cache.size setting is about how
>> much
>>> memory used as block cache used by StoreFile. I have no idea of how HBase
>>> works internally. Typically, it is easy to understand that increase the
>>> size of cache should help the read, but why it will harm the write
>>> operation? The write performance down from 30,000 to 4,000 for your
>>> reference, just by changing the hfile.block.cache.size from 0 to 0.4.
>>> Could anyone give me a brief explanation about this observation or give
>> me
>>> some advices about what to study to understand what is block cache used
>> for?
>>>
>>> Another question: HBase write will first write to WAL then to memstore.
>>> Will the write to WAL go to disk directly before hbase write memstore, a
>>> sync operation or it is possible that write to WAL is still buffered
>>> somewhere when hbase put the data into the memstore?
>>>
>>> Reading src code may cost me months, so a kindly reply will help me a
>>> lot... ...
>>> Thanks very much!
>>>
>>> Best Regards,
>>> Ming
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
Posted by Kevin O'dell <ke...@cloudera.com>.
I am also under the impression that HBase reads should basically not work
with block cache set to 0 since we store the indexes in block cache right?
On Thu, Nov 20, 2014 at 3:31 PM, lars hofhansl <la...@apache.org> wrote:
> That would explain it if memstores are flushed due to global memory
> pressure.
>
> But cache and memstore size are (unfortunately) configured independently.
> The memstore heap portion would be 40% (by default) in either case.So this
> is a bit curious still.
> Ming, can you tell us more details?- RAM on the boxes- heap setup for the
> region servers- any other relevant settings on hbase-site.xml- configs on
> the table/column family you're writing to (like bloom filters, etc).
>
> That would help us diagnose this.
>
> -- Lars
>
> From: Ted Yu <yu...@gmail.com>
> To: "user@hbase.apache.org" <us...@hbase.apache.org>
> Sent: Thursday, November 20, 2014 9:32 AM
> Subject: Re: how to explain read/write performance change after modifying
> the hfile.block.cache.size?
>
> When block cache size increases from 0 to 0.4, the amount of heap given to
> memstore decreases. This would slow down the writes.
> Please see:
> http://hbase.apache.org/book.html#store.memstore
>
> For your second question, see this thread:
>
> http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
>
> Cheers
>
> On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
> wrote:
>
> > Hello, all,
> >
> > I am playing with YCSB to test HBase performance. I am using HBase
> 0.98.5.
> > I tried to adjust the hfile.block.cache.size to see the difference, when
> I
> > set hfile.block.cache.size to 0, read performance is very bad, but write
> > performance is very very very good....; when I set hfile.block.cache.size
> > to 0.4, read is better, but write performance drop dramatically. I
> disable
> > the client side writebuffer already.
> > This is hard to understand for me:
> > The HBase guide just said hfile.block.cache.size setting is about how
> much
> > memory used as block cache used by StoreFile. I have no idea of how HBase
> > works internally. Typically, it is easy to understand that increase the
> > size of cache should help the read, but why it will harm the write
> > operation? The write performance down from 30,000 to 4,000 for your
> > reference, just by changing the hfile.block.cache.size from 0 to 0.4.
> > Could anyone give me a brief explanation about this observation or give
> me
> > some advices about what to study to understand what is block cache used
> for?
> >
> > Another question: HBase write will first write to WAL then to memstore.
> > Will the write to WAL go to disk directly before hbase write memstore, a
> > sync operation or it is possible that write to WAL is still buffered
> > somewhere when hbase put the data into the memstore?
> >
> > Reading src code may cost me months, so a kindly reply will help me a
> > lot... ...
> > Thanks very much!
> >
> > Best Regards,
> > Ming
> >
>
>
>
>
--
Kevin O'Dell
Systems Engineer, Cloudera
Re: how to explain read/write performance change after modifying
the hfile.block.cache.size?
Posted by lars hofhansl <la...@apache.org>.
Yep. Came here to say that. The block cache is 40% of the heap, so if that's 400mb with the old default setting of 1g.
From: "Liu, Ming (HPIT-GADSC)" <mi...@hp.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc: lars hofhansl <la...@apache.org>
Sent: Saturday, November 22, 2014 3:35 PM
Subject: RE: how to explain read/write performance change after modifying the hfile.block.cache.size?
Thank you Nick,
I will increase the heap size.
The workstation is a development workstation. People use it to code and build software, and do unit testing. And I use 'who' and 'ps' to make sure it is exclusive to me when I did the test, but it is not accurate. You are right, the benchmark is not accepted in that env. I just use this to 'develop' a benchmark tool, since I wrote a new YCSB driver for your system on HBase, but the following test is using native HBase driver and test on Hbase. For the real benchmark, we will run it on a separated cluster later. But I hope the data at least make sensor even on a shared env.
The heap configuration is something I really need to check , thank you.
Best Regards,
Ming
-----Original Message-----
From: Nick Dimiduk [mailto:ndimiduk@gmail.com]
Sent: Saturday, November 22, 2014 5:57 AM
To: user@hbase.apache.org
Cc: lars hofhansl
Subject: Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
400mb blockcache? Ouch. What's your hbase-env.sh? Have you configured a heap size? My guess is you're using the un configured default of 1G. Should be at least 8G, and maybe more like 30G with this kind of host.
How many users are sharing it and with what kinds of tasks? If there's no IO isolation between processes, I suspect your benchmarks will be worthless on this shared environment.
-n
On Friday, November 21, 2014, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Thank you Lars,
>
> There must be something wrong with my testing yesterday. I cannot
> reproduce the issue anymore. Now, changing the cache.size from 0 to
> 0.4 will not slow down the write perf dramatically, but still will
> slow down write (Yesterday I saw a 7x slowdown, today it is about 1.3x
> slowdown which is acceptable for me). As Ted pointed out, it is
> possible that memstore cannot get enough memory when more RAM give to
> block cache so it flush more frequently, but I really need more
> reading to understand when memstore will flush.
> And one thing I noticed is when I restart hbase for the very first
> test, the performance is best, then the second time, it is slower,
> both read and write, and slower and slower in the following tests and
> get to a stable point after about 3 or 4 times, in each run I will
> read 5,000,000 rows and update 5,000,000 rows. There are too many
> factors affect the read/write OPS in hbase...
>
> My purpose is to find a proper way to evaluate performance, since we
> are going to change something in hbase and it is good to have a base
> benchmark so we can compare the performance after change. So I must
> make sure the perf test itself make sense and should be trusted.
>
> I saw an entry in the log may help to see the cache settings in my system:
> hfile.LruBlockCache: Total=373.54 MB, free=13.13 MB, max=386.68 MB,
> blocks=5655, accesses=17939947, hits=14065015, hitRatio=78.40%, ,
> cachingAccesses=17934857, cachingHits=14064420,
> cachingHitsRatio=78.42%, evictions=15646, evicted=3861343,
> evictedPerRun=246.7942657470703
>
> My testing environment is a workstation with 12 core CPU, 96G memory
> and
> 1.7 T disk. But it is a shared workstation, many users share it and I
> started hbase in standalone mode with hbase-site.xml as below :
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://localhost:24400/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>hdfs://localhost:24400/zookeeper</value>
> </property>
> <property>
> <name>hbase.master.port</name>
> <value>24560</value>
> </property>
> <property>
> <name>hbase.master.info.port</name>
> <value>24561</value>
> </property>
> <property>
> <name>hbase.regionserver.port</name>
> <value>24562</value>
> </property>
> <property>
> <name>hbase.regionserver.info.port</name>
> <value>24563</value>
> </property>
> <property>
> <name>hbase.zookeeper.peerport</name>
> <value>24567</value>
> </property>
> <property>
> <name>hbase.zookeeper.leaderport</name>
> <value>24568</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>24570</value>
> </property>
> <property>
> <name>hbase.rest.port</name>
> <value>24571</value>
> </property>
> <property>
> <name>hbase.client.scanner.caching</name>
> <value>100</value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.bulkload.staging.dir</name>
> <value>hdfs://localhost:24400/hbase-staging</value>
> </property>
> <property>
> <name>hbase.snapshot.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.master.distributed.log.splitting</name>
> <value>false</value>
> </property>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>90000000</value> :-) I just want to make sure never timeout
> here, since I get timeout so many times...
> </property>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.4</value>
> </property>
> </configuration>
>
>
> [liuliumi@ YCSB]$ free -m
> total used free shared buffers cached
> Mem: 96731 46828 49903 0 984 32525
> -/+ buffers/cache: 13317 83413
> Swap: 48432 12 48420
>
>
> [liuliumi@ YCSB]$ lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> CPU(s): 12
> On-line CPU(s) list: 0-11
> Thread(s) per core: 1
> Core(s) per socket: 6
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 44
> Stepping: 2
> CPU MHz: 3065.894
> BogoMIPS: 6130.96
> Virtualization: VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 256K
> L3 cache: 12288K
> NUMA node0 CPU(s): 0,2,4,6,8,10
> NUMA node1 CPU(s): 1,3,5,7,9,11
>
>
> Thanks,
> Ming
>
> -----Original Message-----
> From: lars hofhansl [mailto:larsh@apache.org <javascript:;>]
> Sent: Friday, November 21, 2014 4:31 AM
> To: user@hbase.apache.org <javascript:;>
> Subject: Re: how to explain read/write performance change after
> modifying the hfile.block.cache.size?
>
> That would explain it if memstores are flushed due to global memory
> pressure.
>
> But cache and memstore size are (unfortunately) configured independently.
> The memstore heap portion would be 40% (by default) in either case.So
> this is a bit curious still.
> Ming, can you tell us more details?- RAM on the boxes- heap setup for
> the region servers- any other relevant settings on hbase-site.xml-
> configs on the table/column family you're writing to (like bloom filters, etc).
>
> That would help us diagnose this.
>
> -- Lars
>
> From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> To: "user@hbase.apache.org <javascript:;>" <user@hbase.apache.org
> <javascript:;>>
> Sent: Thursday, November 20, 2014 9:32 AM
> Subject: Re: how to explain read/write performance change after
> modifying the hfile.block.cache.size?
>
> When block cache size increases from 0 to 0.4, the amount of heap
> given to memstore decreases. This would slow down the writes.
> Please see:
> http://hbase.apache.org/book.html#store.memstore
>
> For your second question, see this thread:
>
> http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clari
> fications+on+HBase+Durability
>
> Cheers
>
> On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC)
> <ming.liu2@hp.com <javascript:;>>
> wrote:
>
> > Hello, all,
> >
> > I am playing with YCSB to test HBase performance. I am using HBase
> 0.98.5.
> > I tried to adjust the hfile.block.cache.size to see the difference,
> > when I set hfile.block.cache.size to 0, read performance is very
> > bad, but write performance is very very very good....; when I set
> > hfile.block.cache.size to 0.4, read is better, but write performance
> > drop dramatically. I disable the client side writebuffer already.
> > This is hard to understand for me:
> > The HBase guide just said hfile.block.cache.size setting is about
> > how much memory used as block cache used by StoreFile. I have no
> > idea of how HBase works internally. Typically, it is easy to
> > understand that increase the size of cache should help the read, but
> > why it will harm the write operation? The write performance down
> > from 30,000 to 4,000 for your reference, just by changing the
> > hfile.block.cache.size from 0
> to 0.4.
> > Could anyone give me a brief explanation about this observation or
> > give me some advices about what to study to understand what is block
> cache used for?
> >
> > Another question: HBase write will first write to WAL then to memstore.
> > Will the write to WAL go to disk directly before hbase write
> > memstore, a sync operation or it is possible that write to WAL is
> > still buffered somewhere when hbase put the data into the memstore?
> >
> > Reading src code may cost me months, so a kindly reply will help me
> > a lot... ...
> > Thanks very much!
> >
> > Best Regards,
> > Ming
> >
>
>
>
>
RE: how to explain read/write performance change after modifying
the hfile.block.cache.size?
Posted by "Liu, Ming (HPIT-GADSC)" <mi...@hp.com>.
Thank you Nick,
I will increase the heap size.
The workstation is a development workstation. People use it to code and build software, and do unit testing. And I use 'who' and 'ps' to make sure it is exclusive to me when I did the test, but it is not accurate. You are right, the benchmark is not accepted in that env. I just use this to 'develop' a benchmark tool, since I wrote a new YCSB driver for your system on HBase, but the following test is using native HBase driver and test on Hbase. For the real benchmark, we will run it on a separated cluster later. But I hope the data at least make sensor even on a shared env.
The heap configuration is something I really need to check , thank you.
Best Regards,
Ming
-----Original Message-----
From: Nick Dimiduk [mailto:ndimiduk@gmail.com]
Sent: Saturday, November 22, 2014 5:57 AM
To: user@hbase.apache.org
Cc: lars hofhansl
Subject: Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
400mb blockcache? Ouch. What's your hbase-env.sh? Have you configured a heap size? My guess is you're using the un configured default of 1G. Should be at least 8G, and maybe more like 30G with this kind of host.
How many users are sharing it and with what kinds of tasks? If there's no IO isolation between processes, I suspect your benchmarks will be worthless on this shared environment.
-n
On Friday, November 21, 2014, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Thank you Lars,
>
> There must be something wrong with my testing yesterday. I cannot
> reproduce the issue anymore. Now, changing the cache.size from 0 to
> 0.4 will not slow down the write perf dramatically, but still will
> slow down write (Yesterday I saw a 7x slowdown, today it is about 1.3x
> slowdown which is acceptable for me). As Ted pointed out, it is
> possible that memstore cannot get enough memory when more RAM give to
> block cache so it flush more frequently, but I really need more
> reading to understand when memstore will flush.
> And one thing I noticed is when I restart hbase for the very first
> test, the performance is best, then the second time, it is slower,
> both read and write, and slower and slower in the following tests and
> get to a stable point after about 3 or 4 times, in each run I will
> read 5,000,000 rows and update 5,000,000 rows. There are too many
> factors affect the read/write OPS in hbase...
>
> My purpose is to find a proper way to evaluate performance, since we
> are going to change something in hbase and it is good to have a base
> benchmark so we can compare the performance after change. So I must
> make sure the perf test itself make sense and should be trusted.
>
> I saw an entry in the log may help to see the cache settings in my system:
> hfile.LruBlockCache: Total=373.54 MB, free=13.13 MB, max=386.68 MB,
> blocks=5655, accesses=17939947, hits=14065015, hitRatio=78.40%, ,
> cachingAccesses=17934857, cachingHits=14064420,
> cachingHitsRatio=78.42%, evictions=15646, evicted=3861343,
> evictedPerRun=246.7942657470703
>
> My testing environment is a workstation with 12 core CPU, 96G memory
> and
> 1.7 T disk. But it is a shared workstation, many users share it and I
> started hbase in standalone mode with hbase-site.xml as below :
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://localhost:24400/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>hdfs://localhost:24400/zookeeper</value>
> </property>
> <property>
> <name>hbase.master.port</name>
> <value>24560</value>
> </property>
> <property>
> <name>hbase.master.info.port</name>
> <value>24561</value>
> </property>
> <property>
> <name>hbase.regionserver.port</name>
> <value>24562</value>
> </property>
> <property>
> <name>hbase.regionserver.info.port</name>
> <value>24563</value>
> </property>
> <property>
> <name>hbase.zookeeper.peerport</name>
> <value>24567</value>
> </property>
> <property>
> <name>hbase.zookeeper.leaderport</name>
> <value>24568</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>24570</value>
> </property>
> <property>
> <name>hbase.rest.port</name>
> <value>24571</value>
> </property>
> <property>
> <name>hbase.client.scanner.caching</name>
> <value>100</value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.bulkload.staging.dir</name>
> <value>hdfs://localhost:24400/hbase-staging</value>
> </property>
> <property>
> <name>hbase.snapshot.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.master.distributed.log.splitting</name>
> <value>false</value>
> </property>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>90000000</value> :-) I just want to make sure never timeout
> here, since I get timeout so many times...
> </property>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.4</value>
> </property>
> </configuration>
>
>
> [liuliumi@ YCSB]$ free -m
> total used free shared buffers cached
> Mem: 96731 46828 49903 0 984 32525
> -/+ buffers/cache: 13317 83413
> Swap: 48432 12 48420
>
>
> [liuliumi@ YCSB]$ lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> CPU(s): 12
> On-line CPU(s) list: 0-11
> Thread(s) per core: 1
> Core(s) per socket: 6
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 44
> Stepping: 2
> CPU MHz: 3065.894
> BogoMIPS: 6130.96
> Virtualization: VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 256K
> L3 cache: 12288K
> NUMA node0 CPU(s): 0,2,4,6,8,10
> NUMA node1 CPU(s): 1,3,5,7,9,11
>
>
> Thanks,
> Ming
>
> -----Original Message-----
> From: lars hofhansl [mailto:larsh@apache.org <javascript:;>]
> Sent: Friday, November 21, 2014 4:31 AM
> To: user@hbase.apache.org <javascript:;>
> Subject: Re: how to explain read/write performance change after
> modifying the hfile.block.cache.size?
>
> That would explain it if memstores are flushed due to global memory
> pressure.
>
> But cache and memstore size are (unfortunately) configured independently.
> The memstore heap portion would be 40% (by default) in either case.So
> this is a bit curious still.
> Ming, can you tell us more details?- RAM on the boxes- heap setup for
> the region servers- any other relevant settings on hbase-site.xml-
> configs on the table/column family you're writing to (like bloom filters, etc).
>
> That would help us diagnose this.
>
> -- Lars
>
> From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> To: "user@hbase.apache.org <javascript:;>" <user@hbase.apache.org
> <javascript:;>>
> Sent: Thursday, November 20, 2014 9:32 AM
> Subject: Re: how to explain read/write performance change after
> modifying the hfile.block.cache.size?
>
> When block cache size increases from 0 to 0.4, the amount of heap
> given to memstore decreases. This would slow down the writes.
> Please see:
> http://hbase.apache.org/book.html#store.memstore
>
> For your second question, see this thread:
>
> http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clari
> fications+on+HBase+Durability
>
> Cheers
>
> On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC)
> <ming.liu2@hp.com <javascript:;>>
> wrote:
>
> > Hello, all,
> >
> > I am playing with YCSB to test HBase performance. I am using HBase
> 0.98.5.
> > I tried to adjust the hfile.block.cache.size to see the difference,
> > when I set hfile.block.cache.size to 0, read performance is very
> > bad, but write performance is very very very good....; when I set
> > hfile.block.cache.size to 0.4, read is better, but write performance
> > drop dramatically. I disable the client side writebuffer already.
> > This is hard to understand for me:
> > The HBase guide just said hfile.block.cache.size setting is about
> > how much memory used as block cache used by StoreFile. I have no
> > idea of how HBase works internally. Typically, it is easy to
> > understand that increase the size of cache should help the read, but
> > why it will harm the write operation? The write performance down
> > from 30,000 to 4,000 for your reference, just by changing the
> > hfile.block.cache.size from 0
> to 0.4.
> > Could anyone give me a brief explanation about this observation or
> > give me some advices about what to study to understand what is block
> cache used for?
> >
> > Another question: HBase write will first write to WAL then to memstore.
> > Will the write to WAL go to disk directly before hbase write
> > memstore, a sync operation or it is possible that write to WAL is
> > still buffered somewhere when hbase put the data into the memstore?
> >
> > Reading src code may cost me months, so a kindly reply will help me
> > a lot... ...
> > Thanks very much!
> >
> > Best Regards,
> > Ming
> >
>
>
>
>
Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
Posted by Nick Dimiduk <nd...@gmail.com>.
400mb blockcache? Ouch. What's your hbase-env.sh? Have you configured a
heap size? My guess is you're using the un configured default of 1G. Should
be at least 8G, and maybe more like 30G with this kind of host.
How many users are sharing it and with what kinds of tasks? If there's no
IO isolation between processes, I suspect your benchmarks will be worthless
on this shared environment.
-n
On Friday, November 21, 2014, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Thank you Lars,
>
> There must be something wrong with my testing yesterday. I cannot
> reproduce the issue anymore. Now, changing the cache.size from 0 to 0.4
> will not slow down the write perf dramatically, but still will slow down
> write (Yesterday I saw a 7x slowdown, today it is about 1.3x slowdown which
> is acceptable for me). As Ted pointed out, it is possible that memstore
> cannot get enough memory when more RAM give to block cache so it flush more
> frequently, but I really need more reading to understand when memstore will
> flush.
> And one thing I noticed is when I restart hbase for the very first test,
> the performance is best, then the second time, it is slower, both read and
> write, and slower and slower in the following tests and get to a stable
> point after about 3 or 4 times, in each run I will read 5,000,000 rows and
> update 5,000,000 rows. There are too many factors affect the read/write OPS
> in hbase...
>
> My purpose is to find a proper way to evaluate performance, since we are
> going to change something in hbase and it is good to have a base benchmark
> so we can compare the performance after change. So I must make sure the
> perf test itself make sense and should be trusted.
>
> I saw an entry in the log may help to see the cache settings in my system:
> hfile.LruBlockCache: Total=373.54 MB, free=13.13 MB, max=386.68 MB,
> blocks=5655, accesses=17939947, hits=14065015, hitRatio=78.40%, ,
> cachingAccesses=17934857, cachingHits=14064420, cachingHitsRatio=78.42%,
> evictions=15646, evicted=3861343, evictedPerRun=246.7942657470703
>
> My testing environment is a workstation with 12 core CPU, 96G memory and
> 1.7 T disk. But it is a shared workstation, many users share it and I
> started hbase in standalone mode with hbase-site.xml as below :
>
> <configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://localhost:24400/hbase</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>hdfs://localhost:24400/zookeeper</value>
> </property>
> <property>
> <name>hbase.master.port</name>
> <value>24560</value>
> </property>
> <property>
> <name>hbase.master.info.port</name>
> <value>24561</value>
> </property>
> <property>
> <name>hbase.regionserver.port</name>
> <value>24562</value>
> </property>
> <property>
> <name>hbase.regionserver.info.port</name>
> <value>24563</value>
> </property>
> <property>
> <name>hbase.zookeeper.peerport</name>
> <value>24567</value>
> </property>
> <property>
> <name>hbase.zookeeper.leaderport</name>
> <value>24568</value>
> </property>
> <property>
> <name>hbase.zookeeper.property.clientPort</name>
> <value>24570</value>
> </property>
> <property>
> <name>hbase.rest.port</name>
> <value>24571</value>
> </property>
> <property>
> <name>hbase.client.scanner.caching</name>
> <value>100</value>
> </property>
> <property>
> <name>hbase.client.scanner.timeout.period</name>
> <value>60000</value>
> </property>
> <property>
> <name>hbase.bulkload.staging.dir</name>
> <value>hdfs://localhost:24400/hbase-staging</value>
> </property>
> <property>
> <name>hbase.snapshot.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>hbase.master.distributed.log.splitting</name>
> <value>false</value>
> </property>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>90000000</value> :-) I just want to make sure never timeout
> here, since I get timeout so many times...
> </property>
> <property>
> <name>hfile.block.cache.size</name>
> <value>0.4</value>
> </property>
> </configuration>
>
>
> [liuliumi@ YCSB]$ free -m
> total used free shared buffers cached
> Mem: 96731 46828 49903 0 984 32525
> -/+ buffers/cache: 13317 83413
> Swap: 48432 12 48420
>
>
> [liuliumi@ YCSB]$ lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> CPU(s): 12
> On-line CPU(s) list: 0-11
> Thread(s) per core: 1
> Core(s) per socket: 6
> Socket(s): 2
> NUMA node(s): 2
> Vendor ID: GenuineIntel
> CPU family: 6
> Model: 44
> Stepping: 2
> CPU MHz: 3065.894
> BogoMIPS: 6130.96
> Virtualization: VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache: 256K
> L3 cache: 12288K
> NUMA node0 CPU(s): 0,2,4,6,8,10
> NUMA node1 CPU(s): 1,3,5,7,9,11
>
>
> Thanks,
> Ming
>
> -----Original Message-----
> From: lars hofhansl [mailto:larsh@apache.org <javascript:;>]
> Sent: Friday, November 21, 2014 4:31 AM
> To: user@hbase.apache.org <javascript:;>
> Subject: Re: how to explain read/write performance change after modifying
> the hfile.block.cache.size?
>
> That would explain it if memstores are flushed due to global memory
> pressure.
>
> But cache and memstore size are (unfortunately) configured independently.
> The memstore heap portion would be 40% (by default) in either case.So this
> is a bit curious still.
> Ming, can you tell us more details?- RAM on the boxes- heap setup for the
> region servers- any other relevant settings on hbase-site.xml- configs on
> the table/column family you're writing to (like bloom filters, etc).
>
> That would help us diagnose this.
>
> -- Lars
>
> From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> To: "user@hbase.apache.org <javascript:;>" <user@hbase.apache.org
> <javascript:;>>
> Sent: Thursday, November 20, 2014 9:32 AM
> Subject: Re: how to explain read/write performance change after modifying
> the hfile.block.cache.size?
>
> When block cache size increases from 0 to 0.4, the amount of heap given to
> memstore decreases. This would slow down the writes.
> Please see:
> http://hbase.apache.org/book.html#store.memstore
>
> For your second question, see this thread:
>
> http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
>
> Cheers
>
> On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <ming.liu2@hp.com
> <javascript:;>>
> wrote:
>
> > Hello, all,
> >
> > I am playing with YCSB to test HBase performance. I am using HBase
> 0.98.5.
> > I tried to adjust the hfile.block.cache.size to see the difference,
> > when I set hfile.block.cache.size to 0, read performance is very bad,
> > but write performance is very very very good....; when I set
> > hfile.block.cache.size to 0.4, read is better, but write performance
> > drop dramatically. I disable the client side writebuffer already.
> > This is hard to understand for me:
> > The HBase guide just said hfile.block.cache.size setting is about how
> > much memory used as block cache used by StoreFile. I have no idea of
> > how HBase works internally. Typically, it is easy to understand that
> > increase the size of cache should help the read, but why it will harm
> > the write operation? The write performance down from 30,000 to 4,000
> > for your reference, just by changing the hfile.block.cache.size from 0
> to 0.4.
> > Could anyone give me a brief explanation about this observation or
> > give me some advices about what to study to understand what is block
> cache used for?
> >
> > Another question: HBase write will first write to WAL then to memstore.
> > Will the write to WAL go to disk directly before hbase write memstore,
> > a sync operation or it is possible that write to WAL is still buffered
> > somewhere when hbase put the data into the memstore?
> >
> > Reading src code may cost me months, so a kindly reply will help me a
> > lot... ...
> > Thanks very much!
> >
> > Best Regards,
> > Ming
> >
>
>
>
>
RE: how to explain read/write performance change after modifying
the hfile.block.cache.size?
Posted by "Liu, Ming (HPIT-GADSC)" <mi...@hp.com>.
Thank you Lars,
There must be something wrong with my testing yesterday. I cannot reproduce the issue anymore. Now, changing the cache.size from 0 to 0.4 will not slow down the write perf dramatically, but still will slow down write (Yesterday I saw a 7x slowdown, today it is about 1.3x slowdown which is acceptable for me). As Ted pointed out, it is possible that memstore cannot get enough memory when more RAM give to block cache so it flush more frequently, but I really need more reading to understand when memstore will flush.
And one thing I noticed is when I restart hbase for the very first test, the performance is best, then the second time, it is slower, both read and write, and slower and slower in the following tests and get to a stable point after about 3 or 4 times, in each run I will read 5,000,000 rows and update 5,000,000 rows. There are too many factors affect the read/write OPS in hbase...
My purpose is to find a proper way to evaluate performance, since we are going to change something in hbase and it is good to have a base benchmark so we can compare the performance after change. So I must make sure the perf test itself make sense and should be trusted.
I saw an entry in the log may help to see the cache settings in my system:
hfile.LruBlockCache: Total=373.54 MB, free=13.13 MB, max=386.68 MB, blocks=5655, accesses=17939947, hits=14065015, hitRatio=78.40%, , cachingAccesses=17934857, cachingHits=14064420, cachingHitsRatio=78.42%, evictions=15646, evicted=3861343, evictedPerRun=246.7942657470703
My testing environment is a workstation with 12 core CPU, 96G memory and 1.7 T disk. But it is a shared workstation, many users share it and I started hbase in standalone mode with hbase-site.xml as below :
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:24400/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>hdfs://localhost:24400/zookeeper</value>
</property>
<property>
<name>hbase.master.port</name>
<value>24560</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>24561</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>24562</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>24563</value>
</property>
<property>
<name>hbase.zookeeper.peerport</name>
<value>24567</value>
</property>
<property>
<name>hbase.zookeeper.leaderport</name>
<value>24568</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>24570</value>
</property>
<property>
<name>hbase.rest.port</name>
<value>24571</value>
</property>
<property>
<name>hbase.client.scanner.caching</name>
<value>100</value>
</property>
<property>
<name>hbase.client.scanner.timeout.period</name>
<value>60000</value>
</property>
<property>
<name>hbase.bulkload.staging.dir</name>
<value>hdfs://localhost:24400/hbase-staging</value>
</property>
<property>
<name>hbase.snapshot.enabled</name>
<value>true</value>
</property>
<property>
<name>hbase.master.distributed.log.splitting</name>
<value>false</value>
</property>
<property>
<name>zookeeper.session.timeout</name>
<value>90000000</value> :-) I just want to make sure never timeout here, since I get timeout so many times...
</property>
<property>
<name>hfile.block.cache.size</name>
<value>0.4</value>
</property>
</configuration>
[liuliumi@ YCSB]$ free -m
total used free shared buffers cached
Mem: 96731 46828 49903 0 984 32525
-/+ buffers/cache: 13317 83413
Swap: 48432 12 48420
[liuliumi@ YCSB]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 1
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 44
Stepping: 2
CPU MHz: 3065.894
BogoMIPS: 6130.96
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0,2,4,6,8,10
NUMA node1 CPU(s): 1,3,5,7,9,11
Thanks,
Ming
-----Original Message-----
From: lars hofhansl [mailto:larsh@apache.org]
Sent: Friday, November 21, 2014 4:31 AM
To: user@hbase.apache.org
Subject: Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
That would explain it if memstores are flushed due to global memory pressure.
But cache and memstore size are (unfortunately) configured independently. The memstore heap portion would be 40% (by default) in either case.So this is a bit curious still.
Ming, can you tell us more details?- RAM on the boxes- heap setup for the region servers- any other relevant settings on hbase-site.xml- configs on the table/column family you're writing to (like bloom filters, etc).
That would help us diagnose this.
-- Lars
From: Ted Yu <yu...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Thursday, November 20, 2014 9:32 AM
Subject: Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
When block cache size increases from 0 to 0.4, the amount of heap given to memstore decreases. This would slow down the writes.
Please see:
http://hbase.apache.org/book.html#store.memstore
For your second question, see this thread:
http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
Cheers
On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Hello, all,
>
> I am playing with YCSB to test HBase performance. I am using HBase 0.98.5.
> I tried to adjust the hfile.block.cache.size to see the difference,
> when I set hfile.block.cache.size to 0, read performance is very bad,
> but write performance is very very very good....; when I set
> hfile.block.cache.size to 0.4, read is better, but write performance
> drop dramatically. I disable the client side writebuffer already.
> This is hard to understand for me:
> The HBase guide just said hfile.block.cache.size setting is about how
> much memory used as block cache used by StoreFile. I have no idea of
> how HBase works internally. Typically, it is easy to understand that
> increase the size of cache should help the read, but why it will harm
> the write operation? The write performance down from 30,000 to 4,000
> for your reference, just by changing the hfile.block.cache.size from 0 to 0.4.
> Could anyone give me a brief explanation about this observation or
> give me some advices about what to study to understand what is block cache used for?
>
> Another question: HBase write will first write to WAL then to memstore.
> Will the write to WAL go to disk directly before hbase write memstore,
> a sync operation or it is possible that write to WAL is still buffered
> somewhere when hbase put the data into the memstore?
>
> Reading src code may cost me months, so a kindly reply will help me a
> lot... ...
> Thanks very much!
>
> Best Regards,
> Ming
>
Re: how to explain read/write performance change after modifying
the hfile.block.cache.size?
Posted by lars hofhansl <la...@apache.org>.
That would explain it if memstores are flushed due to global memory pressure.
But cache and memstore size are (unfortunately) configured independently. The memstore heap portion would be 40% (by default) in either case.So this is a bit curious still.
Ming, can you tell us more details?- RAM on the boxes- heap setup for the region servers- any other relevant settings on hbase-site.xml- configs on the table/column family you're writing to (like bloom filters, etc).
That would help us diagnose this.
-- Lars
From: Ted Yu <yu...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Thursday, November 20, 2014 9:32 AM
Subject: Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
When block cache size increases from 0 to 0.4, the amount of heap given to
memstore decreases. This would slow down the writes.
Please see:
http://hbase.apache.org/book.html#store.memstore
For your second question, see this thread:
http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
Cheers
On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Hello, all,
>
> I am playing with YCSB to test HBase performance. I am using HBase 0.98.5.
> I tried to adjust the hfile.block.cache.size to see the difference, when I
> set hfile.block.cache.size to 0, read performance is very bad, but write
> performance is very very very good....; when I set hfile.block.cache.size
> to 0.4, read is better, but write performance drop dramatically. I disable
> the client side writebuffer already.
> This is hard to understand for me:
> The HBase guide just said hfile.block.cache.size setting is about how much
> memory used as block cache used by StoreFile. I have no idea of how HBase
> works internally. Typically, it is easy to understand that increase the
> size of cache should help the read, but why it will harm the write
> operation? The write performance down from 30,000 to 4,000 for your
> reference, just by changing the hfile.block.cache.size from 0 to 0.4.
> Could anyone give me a brief explanation about this observation or give me
> some advices about what to study to understand what is block cache used for?
>
> Another question: HBase write will first write to WAL then to memstore.
> Will the write to WAL go to disk directly before hbase write memstore, a
> sync operation or it is possible that write to WAL is still buffered
> somewhere when hbase put the data into the memstore?
>
> Reading src code may cost me months, so a kindly reply will help me a
> lot... ...
> Thanks very much!
>
> Best Regards,
> Ming
>
RE: how to explain read/write performance change after modifying
the hfile.block.cache.size?
Posted by "Liu, Ming (HPIT-GADSC)" <mi...@hp.com>.
Thank you Ted,
It is a great explanation. You are always very helpful ^_^
I will study the link carefully.
Thanks,
Ming
-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Friday, November 21, 2014 1:32 AM
To: user@hbase.apache.org
Subject: Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
When block cache size increases from 0 to 0.4, the amount of heap given to memstore decreases. This would slow down the writes.
Please see:
http://hbase.apache.org/book.html#store.memstore
For your second question, see this thread:
http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
Cheers
On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Hello, all,
>
> I am playing with YCSB to test HBase performance. I am using HBase 0.98.5.
> I tried to adjust the hfile.block.cache.size to see the difference,
> when I set hfile.block.cache.size to 0, read performance is very bad,
> but write performance is very very very good....; when I set
> hfile.block.cache.size to 0.4, read is better, but write performance
> drop dramatically. I disable the client side writebuffer already.
> This is hard to understand for me:
> The HBase guide just said hfile.block.cache.size setting is about how
> much memory used as block cache used by StoreFile. I have no idea of
> how HBase works internally. Typically, it is easy to understand that
> increase the size of cache should help the read, but why it will harm
> the write operation? The write performance down from 30,000 to 4,000
> for your reference, just by changing the hfile.block.cache.size from 0 to 0.4.
> Could anyone give me a brief explanation about this observation or
> give me some advices about what to study to understand what is block cache used for?
>
> Another question: HBase write will first write to WAL then to memstore.
> Will the write to WAL go to disk directly before hbase write memstore,
> a sync operation or it is possible that write to WAL is still buffered
> somewhere when hbase put the data into the memstore?
>
> Reading src code may cost me months, so a kindly reply will help me a
> lot... ...
> Thanks very much!
>
> Best Regards,
> Ming
>
Re: how to explain read/write performance change after modifying the hfile.block.cache.size?
Posted by Ted Yu <yu...@gmail.com>.
When block cache size increases from 0 to 0.4, the amount of heap given to
memstore decreases. This would slow down the writes.
Please see:
http://hbase.apache.org/book.html#store.memstore
For your second question, see this thread:
http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflush&subj=Re+Clarifications+on+HBase+Durability
Cheers
On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) <mi...@hp.com>
wrote:
> Hello, all,
>
> I am playing with YCSB to test HBase performance. I am using HBase 0.98.5.
> I tried to adjust the hfile.block.cache.size to see the difference, when I
> set hfile.block.cache.size to 0, read performance is very bad, but write
> performance is very very very good....; when I set hfile.block.cache.size
> to 0.4, read is better, but write performance drop dramatically. I disable
> the client side writebuffer already.
> This is hard to understand for me:
> The HBase guide just said hfile.block.cache.size setting is about how much
> memory used as block cache used by StoreFile. I have no idea of how HBase
> works internally. Typically, it is easy to understand that increase the
> size of cache should help the read, but why it will harm the write
> operation? The write performance down from 30,000 to 4,000 for your
> reference, just by changing the hfile.block.cache.size from 0 to 0.4.
> Could anyone give me a brief explanation about this observation or give me
> some advices about what to study to understand what is block cache used for?
>
> Another question: HBase write will first write to WAL then to memstore.
> Will the write to WAL go to disk directly before hbase write memstore, a
> sync operation or it is possible that write to WAL is still buffered
> somewhere when hbase put the data into the memstore?
>
> Reading src code may cost me months, so a kindly reply will help me a
> lot... ...
> Thanks very much!
>
> Best Regards,
> Ming
>