You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Wayne <wa...@gmail.com> on 2011/01/31 22:38:25 UTC
Open Scanner Latency
After doing many tests (10k serialized scans) we see that on average opening
the scanner takes 2/3 of the read time if the read is fresh
(scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time around (1
minute later) we assume the region cache is "hot" and the open scanner is
much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After 1-2
hours the cache is no longer "hot" and we are back to the initial numbers.
We assume this is due to finding where the data is located in the cluster.
We have cache turned off on our tables, but have 2% cache for hbase and the
.META. table region server is showing 98% hit rate (.META. is served out of
cache). How can we pre-warm the cache to speed up our reads? It does not
seem correct that 2/3 of our read time is always finding where the data is
located. We have played with the prefetch.limit with various different
settings without much difference. How can we warm up the cache? Per the
#2468 wording we need "Clients could prewarm cache by doing a large scan of
all the meta for the table instead of random reads for each miss". We
definitely do not want to pay this price on each read, but would like to
maybe set up a cron job to update once an hour for the tables this is needed
for. It would be great to have a way to pin the region locations to memory
or at least a method to heat it up before a big read process gets kicked
off. A read's latency for our type of usage pattern should be based
primarily on disk i/o latency and not looking around for where the data is
located in the cluster. Adding SSD disks wouldn't help us much at all to
lower read latency given what we are seeing.
Any help or suggestions would be greatly appreciated.
Thanks.
RE: HBase client side configuration example?
Posted by Peter Haidinyak <ph...@local.com>.
I connect to HBase from Java using the following
final Configuration configuration = HBaseConfiguration.create();
configuration.clear();
configuration.set("hbase.zookeeper.quorum", "caiss01a,caiss01b")); // - Our two zookeeper machines
configuration.set("hbase.zookeeper.property.clientPort", "2222")); // - Port they are listening on.
m_logger.info("Opening myTable Tables");
m_myTable = new HTable(configuration, "myTable");
Hope it helps
-Pete
-----Original Message-----
From: Jérôme Verstrynge [mailto:jverstry@gmail.com]
Sent: Wednesday, February 02, 2011 1:58 PM
To: user@hbase.apache.org
Subject: HBase client side configuration example?
Hi,
I have managed to successfully install HBase on a remote linux node
(using Cloudera's CDH3). The next step is to implement a small Java
application to connect to it. I have tried to find some documentation to
configure HBaseConfiguration, but I could not find proper examples for
hbase-site.xml and hbase-default.xml. Does anyone have any to share?
Thanks,
JVerstry
Re: HBase client side configuration example?
Posted by Ted Yu <yu...@gmail.com>.
You should be able to find them under remote linux node's $HBASE_HOME/conf
On Wed, Feb 2, 2011 at 1:57 PM, Jérôme Verstrynge <jv...@gmail.com>wrote:
> Hi,
>
> I have managed to successfully install HBase on a remote linux node (using
> Cloudera's CDH3). The next step is to implement a small Java application to
> connect to it. I have tried to find some documentation to configure
> HBaseConfiguration, but I could not find proper examples for hbase-site.xml
> and hbase-default.xml. Does anyone have any to share?
>
> Thanks,
>
> JVerstry
>
>
>
HBase client side configuration example?
Posted by Jérôme Verstrynge <jv...@gmail.com>.
Hi,
I have managed to successfully install HBase on a remote linux node
(using Cloudera's CDH3). The next step is to implement a small Java
application to connect to it. I have tried to find some documentation to
configure HBaseConfiguration, but I could not find proper examples for
hbase-site.xml and hbase-default.xml. Does anyone have any to share?
Thanks,
JVerstry
Re: Open Scanner Latency
Posted by Wayne <wa...@gmail.com>.
On Mon, Jan 31, 2011 at 4:54 PM, Stack <st...@duboce.net> wrote:
> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> > After doing many tests (10k serialized scans) we see that on average
> opening
> > the scanner takes 2/3 of the read time if the read is fresh
> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms).
>
> I've saw that this w/e. The getScanner takes all the time. Tracing,
> it did not seem to be locating regions in cluster but suspect would
> seem to be down in StoreScanner when we seek all the StoreFiles. I
> didn't look beyond that (this w/e that is).
>
Very interesting. We have written our own Thrift method getRowsWithColumns
which opens a scanner, does one get (we know how many rows we need), and
then closes without ever having to store the scanner. We have yet to push to
the cluster. We realized the benefit of never having to store the scanner
but I did not consider it might be our problem here. We will find out if
this is the problem. Why would StoreScanner be slow the first time around
and then speed up?
>
> What if you do full table scan of the data that you want hot on a
> period (making sure you Scan with the skip cache button OFF)? Is the
> data you want cached all in one table? Does marking the table
> in-memory help?
>
We have blockcache turned off for all of our tables. A full table scan would
be very expensive and timely. Especially since it seems to be flushed
quickly the cost/benefit would be negligible I assume. We have lots of
tables we need to do this with and memory based is not an option. We read a
lot of our data and a smaller cache actually seemed worse in initial tests.
We have enough problems with memory, we are happy to do disk reads (we just
want out bottleneck to be disk i/o).
>
> > A read's latency for our type of usage pattern should be based
> > primarily on disk i/o latency and not looking around for where the data
> is
> > located in the cluster. Adding SSD disks wouldn't help us much at all to
> > lower read latency given what we are seeing.
> >
>
> You think that its locating data in the cluster? Long-lived clients
> shouldn't be doing lookups, they should have cached all seen region
> locations, not unless the region moved. Do you think that is what is
> happening Wayne?
>
I think it is finding it, and it works fast once hot but our write load is
so heavy we assume it pushes the location out of memory. Even if I wait 10
min to rerun the test the numbers start to creep up quickly for the open
scanner. Our test is opening 10k separate scanner objects, reading data, and
then closing.
>
> Here is an interesting article on SSDs and Cassandra:
> http://blog.kosmix.com/?p=1445 Speculation is that SSDs don't really
> improve latency given the size of reads done by cass (and hbase) but
> rather, they help keep latency about constant when lots of contending
> cilents; i.e. maybe we could have one cluster at SU only if we used
> SSDs.
>
We were actually looking to go to 2.5" 10k disks (velociraptor) in those
nifty SuperMicro quad nodes, but from what I have seen this or SSDs would
not have much affect.
>
> St.Ack
>
Re: Open Scanner Latency
Posted by Stack <st...@duboce.net>.
On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> After doing many tests (10k serialized scans) we see that on average opening
> the scanner takes 2/3 of the read time if the read is fresh
> (scannerOpenWithStop=~35ms, scannerGetList=~10ms).
I've saw that this w/e. The getScanner takes all the time. Tracing,
it did not seem to be locating regions in cluster but suspect would
seem to be down in StoreScanner when we seek all the StoreFiles. I
didn't look beyond that (this w/e that is).
What if you do full table scan of the data that you want hot on a
period (making sure you Scan with the skip cache button OFF)? Is the
data you want cached all in one table? Does marking the table
in-memory help?
> A read's latency for our type of usage pattern should be based
> primarily on disk i/o latency and not looking around for where the data is
> located in the cluster. Adding SSD disks wouldn't help us much at all to
> lower read latency given what we are seeing.
>
You think that its locating data in the cluster? Long-lived clients
shouldn't be doing lookups, they should have cached all seen region
locations, not unless the region moved. Do you think that is what is
happening Wayne?
Here is an interesting article on SSDs and Cassandra:
http://blog.kosmix.com/?p=1445 Speculation is that SSDs don't really
improve latency given the size of reads done by cass (and hbase) but
rather, they help keep latency about constant when lots of contending
cilents; i.e. maybe we could have one cluster at SU only if we used
SSDs.
St.Ack
Re: Open Scanner Latency
Posted by 陈加俊 <cj...@gmail.com>.
terrible ...
org.apache.hadoop.hbase.client.ScannerTimeoutException: 338424ms passed
since the last invocation, timeout is currently set to 300000
On Tue, Feb 1, 2011 at 6:45 AM, Wayne <wa...@gmail.com> wrote:
> The file system buffer cache explains what is going on. The open scanner
> reads the first block and the subsequent read goes against the same block
> thereby getting out of the file buffer cache.
>
> Thanks.
>
>
> On Mon, Jan 31, 2011 at 5:22 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
> > Even without block caching, the linux buffer cache is still a factor,
> > and your reads still go through them (via the datanode).
> >
> > When Stack is talking about the StoreScanner, this is a particular
> > class inside of HBase that does the job of reading from 1 column
> > family. The first time you instantiate it, it will read the first
> > block it needs. This is done during the 'openScanner' call, and would
> > explain the latency you are seeing in openScanner.
> >
> > -ryan
> >
> > On Mon, Jan 31, 2011 at 2:17 PM, Wayne <wa...@gmail.com> wrote:
> > > I assume BLOCKCACHE => 'false' would turn this off? We have turned off
> > cache
> > > on all tables.
> > >
> > > On Mon, Jan 31, 2011 at 4:54 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> > >
> > >> The Regionserver caches blocks, so a second read would benefit from
> > >> the caching of the first read. Over time blocks get evicted in a LRU
> > >> manner, and things would get slow again.
> > >>
> > >> Does this make sense to you?
> > >>
> > >> On Mon, Jan 31, 2011 at 1:50 PM, Wayne <wa...@gmail.com> wrote:
> > >> > We have heavy writes always going on so there is always memory
> > pressure.
> > >> >
> > >> > If the open scanner reads the first block maybe that explains the
> 8ms
> > the
> > >> > second time a test is run, but why is the first run averaging 35ms
> to
> > >> open
> > >> > and when the same read requests are sent again the open is only 8ms?
> > >> There
> > >> > is a difference between read #1 and read #2 that I can only explain
> by
> > >> > region location search. Our writes our so heavy I assume this region
> > >> > location information flushed always in 30-60 minutes.
> > >> >
> > >> >
> > >> > On Mon, Jan 31, 2011 at 4:44 PM, Ryan Rawson <ry...@gmail.com>
> > wrote:
> > >> >
> > >> >> Hey,
> > >> >>
> > >> >> The region location cache is held by a soft reference, so as long
> as
> > >> >> you dont have memory pressure, it will never get invalidated just
> > >> >> because of time.
> > >> >>
> > >> >> Another thing to consider, in HBase, the open scanner code also
> seeks
> > >> >> and reads the first block of the scan. This may incur a read to
> disk
> > >> >> and might explain the hot vs cold you are seeing below.
> > >> >>
> > >> >> -ryan
> > >> >>
> > >> >> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> > >> >> > After doing many tests (10k serialized scans) we see that on
> > average
> > >> >> opening
> > >> >> > the scanner takes 2/3 of the read time if the read is fresh
> > >> >> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second
> time
> > >> around
> > >> >> (1
> > >> >> > minute later) we assume the region cache is "hot" and the open
> > scanner
> > >> is
> > >> >> > much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms).
> After
> > >> 1-2
> > >> >> > hours the cache is no longer "hot" and we are back to the initial
> > >> >> numbers.
> > >> >> > We assume this is due to finding where the data is located in the
> > >> >> cluster.
> > >> >> > We have cache turned off on our tables, but have 2% cache for
> hbase
> > >> and
> > >> >> the
> > >> >> > .META. table region server is showing 98% hit rate (.META. is
> > served
> > >> out
> > >> >> of
> > >> >> > cache). How can we pre-warm the cache to speed up our reads? It
> > does
> > >> not
> > >> >> > seem correct that 2/3 of our read time is always finding where
> the
> > >> data
> > >> >> is
> > >> >> > located. We have played with the prefetch.limit with various
> > different
> > >> >> > settings without much difference. How can we warm up the cache?
> Per
> > >> the
> > >> >> > #2468 wording we need "Clients could prewarm cache by doing a
> large
> > >> scan
> > >> >> of
> > >> >> > all the meta for the table instead of random reads for each
> miss".
> > We
> > >> >> > definitely do not want to pay this price on each read, but would
> > like
> > >> to
> > >> >> > maybe set up a cron job to update once an hour for the tables
> this
> > is
> > >> >> needed
> > >> >> > for. It would be great to have a way to pin the region locations
> to
> > >> >> memory
> > >> >> > or at least a method to heat it up before a big read process gets
> > >> kicked
> > >> >> > off. A read's latency for our type of usage pattern should be
> based
> > >> >> > primarily on disk i/o latency and not looking around for where
> the
> > >> data
> > >> >> is
> > >> >> > located in the cluster. Adding SSD disks wouldn't help us much at
> > all
> > >> to
> > >> >> > lower read latency given what we are seeing.
> > >> >> >
> > >> >> > Any help or suggestions would be greatly appreciated.
> > >> >> >
> > >> >> > Thanks.
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> >
>
--
Thanks & Best regards
jiajun
Re: Open Scanner Latency
Posted by Wayne <wa...@gmail.com>.
The file system buffer cache explains what is going on. The open scanner
reads the first block and the subsequent read goes against the same block
thereby getting out of the file buffer cache.
Thanks.
On Mon, Jan 31, 2011 at 5:22 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Even without block caching, the linux buffer cache is still a factor,
> and your reads still go through them (via the datanode).
>
> When Stack is talking about the StoreScanner, this is a particular
> class inside of HBase that does the job of reading from 1 column
> family. The first time you instantiate it, it will read the first
> block it needs. This is done during the 'openScanner' call, and would
> explain the latency you are seeing in openScanner.
>
> -ryan
>
> On Mon, Jan 31, 2011 at 2:17 PM, Wayne <wa...@gmail.com> wrote:
> > I assume BLOCKCACHE => 'false' would turn this off? We have turned off
> cache
> > on all tables.
> >
> > On Mon, Jan 31, 2011 at 4:54 PM, Ryan Rawson <ry...@gmail.com> wrote:
> >
> >> The Regionserver caches blocks, so a second read would benefit from
> >> the caching of the first read. Over time blocks get evicted in a LRU
> >> manner, and things would get slow again.
> >>
> >> Does this make sense to you?
> >>
> >> On Mon, Jan 31, 2011 at 1:50 PM, Wayne <wa...@gmail.com> wrote:
> >> > We have heavy writes always going on so there is always memory
> pressure.
> >> >
> >> > If the open scanner reads the first block maybe that explains the 8ms
> the
> >> > second time a test is run, but why is the first run averaging 35ms to
> >> open
> >> > and when the same read requests are sent again the open is only 8ms?
> >> There
> >> > is a difference between read #1 and read #2 that I can only explain by
> >> > region location search. Our writes our so heavy I assume this region
> >> > location information flushed always in 30-60 minutes.
> >> >
> >> >
> >> > On Mon, Jan 31, 2011 at 4:44 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> >> >
> >> >> Hey,
> >> >>
> >> >> The region location cache is held by a soft reference, so as long as
> >> >> you dont have memory pressure, it will never get invalidated just
> >> >> because of time.
> >> >>
> >> >> Another thing to consider, in HBase, the open scanner code also seeks
> >> >> and reads the first block of the scan. This may incur a read to disk
> >> >> and might explain the hot vs cold you are seeing below.
> >> >>
> >> >> -ryan
> >> >>
> >> >> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> >> >> > After doing many tests (10k serialized scans) we see that on
> average
> >> >> opening
> >> >> > the scanner takes 2/3 of the read time if the read is fresh
> >> >> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time
> >> around
> >> >> (1
> >> >> > minute later) we assume the region cache is "hot" and the open
> scanner
> >> is
> >> >> > much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After
> >> 1-2
> >> >> > hours the cache is no longer "hot" and we are back to the initial
> >> >> numbers.
> >> >> > We assume this is due to finding where the data is located in the
> >> >> cluster.
> >> >> > We have cache turned off on our tables, but have 2% cache for hbase
> >> and
> >> >> the
> >> >> > .META. table region server is showing 98% hit rate (.META. is
> served
> >> out
> >> >> of
> >> >> > cache). How can we pre-warm the cache to speed up our reads? It
> does
> >> not
> >> >> > seem correct that 2/3 of our read time is always finding where the
> >> data
> >> >> is
> >> >> > located. We have played with the prefetch.limit with various
> different
> >> >> > settings without much difference. How can we warm up the cache? Per
> >> the
> >> >> > #2468 wording we need "Clients could prewarm cache by doing a large
> >> scan
> >> >> of
> >> >> > all the meta for the table instead of random reads for each miss".
> We
> >> >> > definitely do not want to pay this price on each read, but would
> like
> >> to
> >> >> > maybe set up a cron job to update once an hour for the tables this
> is
> >> >> needed
> >> >> > for. It would be great to have a way to pin the region locations to
> >> >> memory
> >> >> > or at least a method to heat it up before a big read process gets
> >> kicked
> >> >> > off. A read's latency for our type of usage pattern should be based
> >> >> > primarily on disk i/o latency and not looking around for where the
> >> data
> >> >> is
> >> >> > located in the cluster. Adding SSD disks wouldn't help us much at
> all
> >> to
> >> >> > lower read latency given what we are seeing.
> >> >> >
> >> >> > Any help or suggestions would be greatly appreciated.
> >> >> >
> >> >> > Thanks.
> >> >> >
> >> >>
> >> >
> >>
> >
>
Re: Open Scanner Latency
Posted by Ryan Rawson <ry...@gmail.com>.
Even without block caching, the linux buffer cache is still a factor,
and your reads still go through them (via the datanode).
When Stack is talking about the StoreScanner, this is a particular
class inside of HBase that does the job of reading from 1 column
family. The first time you instantiate it, it will read the first
block it needs. This is done during the 'openScanner' call, and would
explain the latency you are seeing in openScanner.
-ryan
On Mon, Jan 31, 2011 at 2:17 PM, Wayne <wa...@gmail.com> wrote:
> I assume BLOCKCACHE => 'false' would turn this off? We have turned off cache
> on all tables.
>
> On Mon, Jan 31, 2011 at 4:54 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> The Regionserver caches blocks, so a second read would benefit from
>> the caching of the first read. Over time blocks get evicted in a LRU
>> manner, and things would get slow again.
>>
>> Does this make sense to you?
>>
>> On Mon, Jan 31, 2011 at 1:50 PM, Wayne <wa...@gmail.com> wrote:
>> > We have heavy writes always going on so there is always memory pressure.
>> >
>> > If the open scanner reads the first block maybe that explains the 8ms the
>> > second time a test is run, but why is the first run averaging 35ms to
>> open
>> > and when the same read requests are sent again the open is only 8ms?
>> There
>> > is a difference between read #1 and read #2 that I can only explain by
>> > region location search. Our writes our so heavy I assume this region
>> > location information flushed always in 30-60 minutes.
>> >
>> >
>> > On Mon, Jan 31, 2011 at 4:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
>> >
>> >> Hey,
>> >>
>> >> The region location cache is held by a soft reference, so as long as
>> >> you dont have memory pressure, it will never get invalidated just
>> >> because of time.
>> >>
>> >> Another thing to consider, in HBase, the open scanner code also seeks
>> >> and reads the first block of the scan. This may incur a read to disk
>> >> and might explain the hot vs cold you are seeing below.
>> >>
>> >> -ryan
>> >>
>> >> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
>> >> > After doing many tests (10k serialized scans) we see that on average
>> >> opening
>> >> > the scanner takes 2/3 of the read time if the read is fresh
>> >> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time
>> around
>> >> (1
>> >> > minute later) we assume the region cache is "hot" and the open scanner
>> is
>> >> > much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After
>> 1-2
>> >> > hours the cache is no longer "hot" and we are back to the initial
>> >> numbers.
>> >> > We assume this is due to finding where the data is located in the
>> >> cluster.
>> >> > We have cache turned off on our tables, but have 2% cache for hbase
>> and
>> >> the
>> >> > .META. table region server is showing 98% hit rate (.META. is served
>> out
>> >> of
>> >> > cache). How can we pre-warm the cache to speed up our reads? It does
>> not
>> >> > seem correct that 2/3 of our read time is always finding where the
>> data
>> >> is
>> >> > located. We have played with the prefetch.limit with various different
>> >> > settings without much difference. How can we warm up the cache? Per
>> the
>> >> > #2468 wording we need "Clients could prewarm cache by doing a large
>> scan
>> >> of
>> >> > all the meta for the table instead of random reads for each miss". We
>> >> > definitely do not want to pay this price on each read, but would like
>> to
>> >> > maybe set up a cron job to update once an hour for the tables this is
>> >> needed
>> >> > for. It would be great to have a way to pin the region locations to
>> >> memory
>> >> > or at least a method to heat it up before a big read process gets
>> kicked
>> >> > off. A read's latency for our type of usage pattern should be based
>> >> > primarily on disk i/o latency and not looking around for where the
>> data
>> >> is
>> >> > located in the cluster. Adding SSD disks wouldn't help us much at all
>> to
>> >> > lower read latency given what we are seeing.
>> >> >
>> >> > Any help or suggestions would be greatly appreciated.
>> >> >
>> >> > Thanks.
>> >> >
>> >>
>> >
>>
>
Re: Open Scanner Latency
Posted by Wayne <wa...@gmail.com>.
I assume BLOCKCACHE => 'false' would turn this off? We have turned off cache
on all tables.
On Mon, Jan 31, 2011 at 4:54 PM, Ryan Rawson <ry...@gmail.com> wrote:
> The Regionserver caches blocks, so a second read would benefit from
> the caching of the first read. Over time blocks get evicted in a LRU
> manner, and things would get slow again.
>
> Does this make sense to you?
>
> On Mon, Jan 31, 2011 at 1:50 PM, Wayne <wa...@gmail.com> wrote:
> > We have heavy writes always going on so there is always memory pressure.
> >
> > If the open scanner reads the first block maybe that explains the 8ms the
> > second time a test is run, but why is the first run averaging 35ms to
> open
> > and when the same read requests are sent again the open is only 8ms?
> There
> > is a difference between read #1 and read #2 that I can only explain by
> > region location search. Our writes our so heavy I assume this region
> > location information flushed always in 30-60 minutes.
> >
> >
> > On Mon, Jan 31, 2011 at 4:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
> >
> >> Hey,
> >>
> >> The region location cache is held by a soft reference, so as long as
> >> you dont have memory pressure, it will never get invalidated just
> >> because of time.
> >>
> >> Another thing to consider, in HBase, the open scanner code also seeks
> >> and reads the first block of the scan. This may incur a read to disk
> >> and might explain the hot vs cold you are seeing below.
> >>
> >> -ryan
> >>
> >> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> >> > After doing many tests (10k serialized scans) we see that on average
> >> opening
> >> > the scanner takes 2/3 of the read time if the read is fresh
> >> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time
> around
> >> (1
> >> > minute later) we assume the region cache is "hot" and the open scanner
> is
> >> > much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After
> 1-2
> >> > hours the cache is no longer "hot" and we are back to the initial
> >> numbers.
> >> > We assume this is due to finding where the data is located in the
> >> cluster.
> >> > We have cache turned off on our tables, but have 2% cache for hbase
> and
> >> the
> >> > .META. table region server is showing 98% hit rate (.META. is served
> out
> >> of
> >> > cache). How can we pre-warm the cache to speed up our reads? It does
> not
> >> > seem correct that 2/3 of our read time is always finding where the
> data
> >> is
> >> > located. We have played with the prefetch.limit with various different
> >> > settings without much difference. How can we warm up the cache? Per
> the
> >> > #2468 wording we need "Clients could prewarm cache by doing a large
> scan
> >> of
> >> > all the meta for the table instead of random reads for each miss". We
> >> > definitely do not want to pay this price on each read, but would like
> to
> >> > maybe set up a cron job to update once an hour for the tables this is
> >> needed
> >> > for. It would be great to have a way to pin the region locations to
> >> memory
> >> > or at least a method to heat it up before a big read process gets
> kicked
> >> > off. A read's latency for our type of usage pattern should be based
> >> > primarily on disk i/o latency and not looking around for where the
> data
> >> is
> >> > located in the cluster. Adding SSD disks wouldn't help us much at all
> to
> >> > lower read latency given what we are seeing.
> >> >
> >> > Any help or suggestions would be greatly appreciated.
> >> >
> >> > Thanks.
> >> >
> >>
> >
>
Re: Open Scanner Latency
Posted by Ryan Rawson <ry...@gmail.com>.
The Regionserver caches blocks, so a second read would benefit from
the caching of the first read. Over time blocks get evicted in a LRU
manner, and things would get slow again.
Does this make sense to you?
On Mon, Jan 31, 2011 at 1:50 PM, Wayne <wa...@gmail.com> wrote:
> We have heavy writes always going on so there is always memory pressure.
>
> If the open scanner reads the first block maybe that explains the 8ms the
> second time a test is run, but why is the first run averaging 35ms to open
> and when the same read requests are sent again the open is only 8ms? There
> is a difference between read #1 and read #2 that I can only explain by
> region location search. Our writes our so heavy I assume this region
> location information flushed always in 30-60 minutes.
>
>
> On Mon, Jan 31, 2011 at 4:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> Hey,
>>
>> The region location cache is held by a soft reference, so as long as
>> you dont have memory pressure, it will never get invalidated just
>> because of time.
>>
>> Another thing to consider, in HBase, the open scanner code also seeks
>> and reads the first block of the scan. This may incur a read to disk
>> and might explain the hot vs cold you are seeing below.
>>
>> -ryan
>>
>> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
>> > After doing many tests (10k serialized scans) we see that on average
>> opening
>> > the scanner takes 2/3 of the read time if the read is fresh
>> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time around
>> (1
>> > minute later) we assume the region cache is "hot" and the open scanner is
>> > much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After 1-2
>> > hours the cache is no longer "hot" and we are back to the initial
>> numbers.
>> > We assume this is due to finding where the data is located in the
>> cluster.
>> > We have cache turned off on our tables, but have 2% cache for hbase and
>> the
>> > .META. table region server is showing 98% hit rate (.META. is served out
>> of
>> > cache). How can we pre-warm the cache to speed up our reads? It does not
>> > seem correct that 2/3 of our read time is always finding where the data
>> is
>> > located. We have played with the prefetch.limit with various different
>> > settings without much difference. How can we warm up the cache? Per the
>> > #2468 wording we need "Clients could prewarm cache by doing a large scan
>> of
>> > all the meta for the table instead of random reads for each miss". We
>> > definitely do not want to pay this price on each read, but would like to
>> > maybe set up a cron job to update once an hour for the tables this is
>> needed
>> > for. It would be great to have a way to pin the region locations to
>> memory
>> > or at least a method to heat it up before a big read process gets kicked
>> > off. A read's latency for our type of usage pattern should be based
>> > primarily on disk i/o latency and not looking around for where the data
>> is
>> > located in the cluster. Adding SSD disks wouldn't help us much at all to
>> > lower read latency given what we are seeing.
>> >
>> > Any help or suggestions would be greatly appreciated.
>> >
>> > Thanks.
>> >
>>
>
Re: Open Scanner Latency
Posted by Wayne <wa...@gmail.com>.
We have heavy writes always going on so there is always memory pressure.
If the open scanner reads the first block maybe that explains the 8ms the
second time a test is run, but why is the first run averaging 35ms to open
and when the same read requests are sent again the open is only 8ms? There
is a difference between read #1 and read #2 that I can only explain by
region location search. Our writes our so heavy I assume this region
location information flushed always in 30-60 minutes.
On Mon, Jan 31, 2011 at 4:44 PM, Ryan Rawson <ry...@gmail.com> wrote:
> Hey,
>
> The region location cache is held by a soft reference, so as long as
> you dont have memory pressure, it will never get invalidated just
> because of time.
>
> Another thing to consider, in HBase, the open scanner code also seeks
> and reads the first block of the scan. This may incur a read to disk
> and might explain the hot vs cold you are seeing below.
>
> -ryan
>
> On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> > After doing many tests (10k serialized scans) we see that on average
> opening
> > the scanner takes 2/3 of the read time if the read is fresh
> > (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time around
> (1
> > minute later) we assume the region cache is "hot" and the open scanner is
> > much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After 1-2
> > hours the cache is no longer "hot" and we are back to the initial
> numbers.
> > We assume this is due to finding where the data is located in the
> cluster.
> > We have cache turned off on our tables, but have 2% cache for hbase and
> the
> > .META. table region server is showing 98% hit rate (.META. is served out
> of
> > cache). How can we pre-warm the cache to speed up our reads? It does not
> > seem correct that 2/3 of our read time is always finding where the data
> is
> > located. We have played with the prefetch.limit with various different
> > settings without much difference. How can we warm up the cache? Per the
> > #2468 wording we need "Clients could prewarm cache by doing a large scan
> of
> > all the meta for the table instead of random reads for each miss". We
> > definitely do not want to pay this price on each read, but would like to
> > maybe set up a cron job to update once an hour for the tables this is
> needed
> > for. It would be great to have a way to pin the region locations to
> memory
> > or at least a method to heat it up before a big read process gets kicked
> > off. A read's latency for our type of usage pattern should be based
> > primarily on disk i/o latency and not looking around for where the data
> is
> > located in the cluster. Adding SSD disks wouldn't help us much at all to
> > lower read latency given what we are seeing.
> >
> > Any help or suggestions would be greatly appreciated.
> >
> > Thanks.
> >
>
Re: Open Scanner Latency
Posted by Ryan Rawson <ry...@gmail.com>.
Hey,
The region location cache is held by a soft reference, so as long as
you dont have memory pressure, it will never get invalidated just
because of time.
Another thing to consider, in HBase, the open scanner code also seeks
and reads the first block of the scan. This may incur a read to disk
and might explain the hot vs cold you are seeing below.
-ryan
On Mon, Jan 31, 2011 at 1:38 PM, Wayne <wa...@gmail.com> wrote:
> After doing many tests (10k serialized scans) we see that on average opening
> the scanner takes 2/3 of the read time if the read is fresh
> (scannerOpenWithStop=~35ms, scannerGetList=~10ms). The second time around (1
> minute later) we assume the region cache is "hot" and the open scanner is
> much faster (scannerOpenWithStop=~8ms, scannerGetList=~10ms). After 1-2
> hours the cache is no longer "hot" and we are back to the initial numbers.
> We assume this is due to finding where the data is located in the cluster.
> We have cache turned off on our tables, but have 2% cache for hbase and the
> .META. table region server is showing 98% hit rate (.META. is served out of
> cache). How can we pre-warm the cache to speed up our reads? It does not
> seem correct that 2/3 of our read time is always finding where the data is
> located. We have played with the prefetch.limit with various different
> settings without much difference. How can we warm up the cache? Per the
> #2468 wording we need "Clients could prewarm cache by doing a large scan of
> all the meta for the table instead of random reads for each miss". We
> definitely do not want to pay this price on each read, but would like to
> maybe set up a cron job to update once an hour for the tables this is needed
> for. It would be great to have a way to pin the region locations to memory
> or at least a method to heat it up before a big read process gets kicked
> off. A read's latency for our type of usage pattern should be based
> primarily on disk i/o latency and not looking around for where the data is
> located in the cluster. Adding SSD disks wouldn't help us much at all to
> lower read latency given what we are seeing.
>
> Any help or suggestions would be greatly appreciated.
>
> Thanks.
>