You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vibhav Mundra <mu...@gmail.com> on 2013/04/01 12:09:04 UTC
Read thruput
Hi All,
I am trying to use Hbase for real-time data retrieval with a timeout of 50
ms.
I am using 2 machines as datanode and regionservers,
and one machine as a master for hadoop and Hbase.
But I am able to fire only 3000 queries per sec and 10% of them are timing
out.
The database has 60 million rows.
Are these figure okie, or I am missing something.
I have used the scanner caching to be equal to one, because for each time
we are fetching a single row only.
Here are the various configurations:
*Our schema
*{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
*Configuration*
1 Machine having both hbase and hadoop master
2 machines having both region server node and datanode
total 285 region servers
*Machine Level Optimizations:*
a)No of file descriptors is 1000000(ulimit -n gives 1000000)
b)Increase the read-ahead value to 4096
c)Added noatime,nodiratime to the disks
*Hadoop Optimizations:*
dfs.datanode.max.xcievers = 4096
dfs.block.size = 33554432
dfs.datanode.handler.count = 256
io.file.buffer.size = 65536
hadoop data is split on 4 directories, so that different disks are being
accessed
*Hbase Optimizations*:
hbase.client.scanner.caching=1 #We have specifcally added this, as we
return always one row.
hbase.regionserver.handler.count=3200
hfile.block.cache.size=0.35
hbase.hregion.memstore.mslab.enabled=true
hfile.min.blocksize.size=16384
hfile.min.blocksize.size=4
hbase.hstore.blockingStoreFiles=200
hbase.regionserver.optionallogflushinterval=60000
hbase.hregion.majorcompaction=0
hbase.hstore.compaction.max=100
hbase.hstore.compactionThreshold=100
*Hbase-GC
*-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
*Hadoop-GC*
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-Vibhav
Re: Read thruput
Posted by Vibhav Mundra <mu...@gmail.com>.
Yes, we are running with 8GB, so I dont think that should be a concern.
-Vibhav
On Tue, Apr 2, 2013 at 12:03 AM, Ted Yu <yu...@gmail.com> wrote:
> Your hbase.regionserver.handler.count seems very high. The following is
> from hbase-default.xml:
>
> For an estimate of server-side memory-used, evaluate
> hbase.client.write.buffer * hbase.regionserver.handler.count
>
> In your case, the above product would be 6GB :-)
>
>
> On Mon, Apr 1, 2013 at 3:09 AM, Vibhav Mundra <mu...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am trying to use Hbase for real-time data retrieval with a timeout of
> 50
> > ms.
> >
> > I am using 2 machines as datanode and regionservers,
> > and one machine as a master for hadoop and Hbase.
> >
> > But I am able to fire only 3000 queries per sec and 10% of them are
> timing
> > out.
> > The database has 60 million rows.
> >
> > Are these figure okie, or I am missing something.
> > I have used the scanner caching to be equal to one, because for each time
> > we are fetching a single row only.
> >
> > Here are the various configurations:
> >
> > *Our schema
> > *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
> > 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
> > 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> > P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
> 'true',
> > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
> >
> > *Configuration*
> > 1 Machine having both hbase and hadoop master
> > 2 machines having both region server node and datanode
> > total 285 region servers
> >
> > *Machine Level Optimizations:*
> > a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> > b)Increase the read-ahead value to 4096
> > c)Added noatime,nodiratime to the disks
> >
> > *Hadoop Optimizations:*
> > dfs.datanode.max.xcievers = 4096
> > dfs.block.size = 33554432
> > dfs.datanode.handler.count = 256
> > io.file.buffer.size = 65536
> > hadoop data is split on 4 directories, so that different disks are being
> > accessed
> >
> > *Hbase Optimizations*:
> >
> > hbase.client.scanner.caching=1 #We have specifcally added this, as we
> > return always one row.
> > hbase.regionserver.handler.count=3200
> > hfile.block.cache.size=0.35
> > hbase.hregion.memstore.mslab.enabled=true
> > hfile.min.blocksize.size=16384
> > hfile.min.blocksize.size=4
> > hbase.hstore.blockingStoreFiles=200
> > hbase.regionserver.optionallogflushinterval=60000
> > hbase.hregion.majorcompaction=0
> > hbase.hstore.compaction.max=100
> > hbase.hstore.compactionThreshold=100
> >
> > *Hbase-GC
> > *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
> > -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> > *Hadoop-GC*
> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> >
> > -Vibhav
> >
>
Re: Read thruput
Posted by Ted Yu <yu...@gmail.com>.
Your hbase.regionserver.handler.count seems very high. The following is
from hbase-default.xml:
For an estimate of server-side memory-used, evaluate
hbase.client.write.buffer * hbase.regionserver.handler.count
In your case, the above product would be 6GB :-)
On Mon, Apr 1, 2013 at 3:09 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> Hi All,
>
> I am trying to use Hbase for real-time data retrieval with a timeout of 50
> ms.
>
> I am using 2 machines as datanode and regionservers,
> and one machine as a master for hadoop and Hbase.
>
> But I am able to fire only 3000 queries per sec and 10% of them are timing
> out.
> The database has 60 million rows.
>
> Are these figure okie, or I am missing something.
> I have used the scanner caching to be equal to one, because for each time
> we are fetching a single row only.
>
> Here are the various configurations:
>
> *Our schema
> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>
> *Configuration*
> 1 Machine having both hbase and hadoop master
> 2 machines having both region server node and datanode
> total 285 region servers
>
> *Machine Level Optimizations:*
> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> b)Increase the read-ahead value to 4096
> c)Added noatime,nodiratime to the disks
>
> *Hadoop Optimizations:*
> dfs.datanode.max.xcievers = 4096
> dfs.block.size = 33554432
> dfs.datanode.handler.count = 256
> io.file.buffer.size = 65536
> hadoop data is split on 4 directories, so that different disks are being
> accessed
>
> *Hbase Optimizations*:
>
> hbase.client.scanner.caching=1 #We have specifcally added this, as we
> return always one row.
> hbase.regionserver.handler.count=3200
> hfile.block.cache.size=0.35
> hbase.hregion.memstore.mslab.enabled=true
> hfile.min.blocksize.size=16384
> hfile.min.blocksize.size=4
> hbase.hstore.blockingStoreFiles=200
> hbase.regionserver.optionallogflushinterval=60000
> hbase.hregion.majorcompaction=0
> hbase.hstore.compaction.max=100
> hbase.hstore.compactionThreshold=100
>
> *Hbase-GC
> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> *Hadoop-GC*
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>
> -Vibhav
>
Re: Read thruput
Posted by Azuryy Yu <az...@gmail.com>.
can you output GC log? CMS GC should be optimized futher. please find it on
official site. another, use vmstat monitor page rate during query.
On Apr 1, 2013 6:09 PM, "Vibhav Mundra" <mu...@gmail.com> wrote:
> Hi All,
>
> I am trying to use Hbase for real-time data retrieval with a timeout of 50
> ms.
>
> I am using 2 machines as datanode and regionservers,
> and one machine as a master for hadoop and Hbase.
>
> But I am able to fire only 3000 queries per sec and 10% of them are timing
> out.
> The database has 60 million rows.
>
> Are these figure okie, or I am missing something.
> I have used the scanner caching to be equal to one, because for each time
> we are fetching a single row only.
>
> Here are the various configurations:
>
> *Our schema
> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>
> *Configuration*
> 1 Machine having both hbase and hadoop master
> 2 machines having both region server node and datanode
> total 285 region servers
>
> *Machine Level Optimizations:*
> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> b)Increase the read-ahead value to 4096
> c)Added noatime,nodiratime to the disks
>
> *Hadoop Optimizations:*
> dfs.datanode.max.xcievers = 4096
> dfs.block.size = 33554432
> dfs.datanode.handler.count = 256
> io.file.buffer.size = 65536
> hadoop data is split on 4 directories, so that different disks are being
> accessed
>
> *Hbase Optimizations*:
>
> hbase.client.scanner.caching=1 #We have specifcally added this, as we
> return always one row.
> hbase.regionserver.handler.count=3200
> hfile.block.cache.size=0.35
> hbase.hregion.memstore.mslab.enabled=true
> hfile.min.blocksize.size=16384
> hfile.min.blocksize.size=4
> hbase.hstore.blockingStoreFiles=200
> hbase.regionserver.optionallogflushinterval=60000
> hbase.hregion.majorcompaction=0
> hbase.hstore.compaction.max=100
> hbase.hstore.compactionThreshold=100
>
> *Hbase-GC
> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> *Hadoop-GC*
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>
> -Vibhav
>
Re: Read thruput
Posted by lars hofhansl <la...@apache.org>.
If you are concerned about latencies < 50ms you should disable Nagle's.
In hbase-site.xml:
<property>
<name>hbase.ipc.client.tcpnodelay</name>
<value>true</value>
</property>
<property>
<name>ipc.server.tcpnodelay</name>
<value>true</value>
</property>
You might get a further latency improvement if you do that same for HDFS:
In hdfs-site.xml:
<property>
<name>ipc.server.tcpnodelay</name>
<value>true</value>
</property>
<property>
<name>ipc.client.tcpnodelay</name>
<value>true</value>
</property>
Also (as other's have pointed out) you need to carefully control your garbage collections.
Watch the HDFS replication count (3 by default, which does not make any sense with only 2 DNs), but since your reading that should make no difference.
-- Lars
________________________________
From: Vibhav Mundra <mu...@gmail.com>
To: user@hbase.apache.org
Sent: Monday, April 1, 2013 3:09 AM
Subject: Read thruput
Hi All,
I am trying to use Hbase for real-time data retrieval with a timeout of 50
ms.
I am using 2 machines as datanode and regionservers,
and one machine as a master for hadoop and Hbase.
But I am able to fire only 3000 queries per sec and 10% of them are timing
out.
The database has 60 million rows.
Are these figure okie, or I am missing something.
I have used the scanner caching to be equal to one, because for each time
we are fetching a single row only.
Here are the various configurations:
*Our schema
*{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
*Configuration*
1 Machine having both hbase and hadoop master
2 machines having both region server node and datanode
total 285 region servers
*Machine Level Optimizations:*
a)No of file descriptors is 1000000(ulimit -n gives 1000000)
b)Increase the read-ahead value to 4096
c)Added noatime,nodiratime to the disks
*Hadoop Optimizations:*
dfs.datanode.max.xcievers = 4096
dfs.block.size = 33554432
dfs.datanode.handler.count = 256
io.file.buffer.size = 65536
hadoop data is split on 4 directories, so that different disks are being
accessed
*Hbase Optimizations*:
hbase.client.scanner.caching=1 #We have specifcally added this, as we
return always one row.
hbase.regionserver.handler.count=3200
hfile.block.cache.size=0.35
hbase.hregion.memstore.mslab.enabled=true
hfile.min.blocksize.size=16384
hfile.min.blocksize.size=4
hbase.hstore.blockingStoreFiles=200
hbase.regionserver.optionallogflushinterval=60000
hbase.hregion.majorcompaction=0
hbase.hstore.compaction.max=100
hbase.hstore.compactionThreshold=100
*Hbase-GC
*-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
*Hadoop-GC*
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-Vibhav
Re: Read thruput
Posted by Asaf Mesika <as...@gmail.com>.
Can you possible batch some Get calls to a Scan with a Filter that contains
the list of row keys you need?
For example, if you have 100 Gets, you can create a start key and end key
from getting the max and mix from those 100 row keys list. Next, you need
to write a filter which saves this 100 row keys to a private member and
uses the hint method in the Filter interface to jump to the closest rowkey
in the region it scans.
If you need help with that I can add a more detailed description of that
Filter.
This should reduce most of the heavy weight over head processing of each
Get.
On Tuesday, April 2, 2013, Vibhav Mundra wrote:
> How does your client call looks like? Get? Scan? Filters?
> --My client keeps doing the Get request. Each time a single row is fetched.
> Essentially we are using Hbase as key value retrieval.
>
> Is 3000/sec is client side calls or is it in numbers of rows per sec?
> --3000/sec is the client side calls.
>
> If you measure in MB/sec how much read throughput do you get?
> --Each client request's response is at maximum 1 KB so its the MB/sec is
> 3MB { 3000 * 1 KB }.
>
> Where is your client located? Same router as the cluster?
> --It is routed on the same cluster, on the same subnet.
>
> Have you activated dfs read short circuit? Of not try it.
> --I have not tried this. Let me try this also.
>
> Compression - try switching to Snappy - should be faster.
> What else is running on the cluster parallel to your reading client?
> --There is small upload code running. I have never seen the CPU usage more
> than 5%, so actually didnt bother to look at this angle.
>
> -Vibhav
>
>
> On Tue, Apr 2, 2013 at 1:42 AM, Asaf Mesika <as...@gmail.com> wrote:
>
> > How does your client call looks like? Get? Scan? Filters?
> > Is 3000/sec is client side calls or is it in numbers of rows per sec?
> > If you measure in MB/sec how much read throughput do you get?
> > Where is your client located? Same router as the cluster?
> > Have you activated dfs read short circuit? Of not try it.
> > Compression - try switching to Snappy - should be faster.
> > What else is running on the cluster parallel to your reading client?
> >
> > On Monday, April 1, 2013, Vibhav Mundra wrote:
> >
> > > What is the general read-thru put that one gets when using Hbase.
> > >
> > > I am not to able to achieve more than 3000/secs with a timeout of 50
> > > millisecs.
> > > In this case also there is 10% of them are timing-out.
> > >
> > > -Vibhav
> > >
> > >
> > > On Mon, Apr 1, 2013 at 11:20 PM, Vibhav Mundra <mu...@gmail.com>
> wrote:
> > >
> > > > yes, I have changes the BLOCK CACHE % to 0.35.
> > > >
> > > > -Vibhav
> > > >
> > > >
> > > > On Mon, Apr 1, 2013 at 10:20 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > >> I was aware of that discussion which was about MAX_FILESIZE and
> > > BLOCKSIZE
> > > >>
> > > >> My suggestion was about block cache percentage.
> > > >>
> > > >> Cheers
> > > >>
> > > >>
> > > >> On Mon, Apr 1, 2013 at 4:57 AM, Vibhav Mundra <mu...@gmail.com>
> > wrote:
> > > >>
> > > >> > I have used the following site:
> > > >> > http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
> > > >> >
> > > >> > to lessen the value of block cache.
> > > >> >
> > > >> > -Vibhav
> > > >> >
> > > >> >
> > > >> > On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
> > > >> >
> > > >> > > Can you increase block cache size ?
> > > >> > >
> > > >> > > What version of hbase are you using ?
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com>
> > wrote:
> > > >> > >
> > > >> > > > The typical size of each of my row is less than 1KB.
> > > >> > > >
> > > >> > > > Regarding the memory, I have used 8GB for Hbase regionservers
> > and
> > > 4
> > > >> GB
> > > >> > > for
> > > >> > > > datanodes and I dont see them completely used. So I ruled out
> > the
> > > GC
> > > >> > > aspect.
> > > >> > > >
> > > >> > > > In case u still believe that GC is an issue, I will upload the
> > gc
> > > >> logs.
> > > >> > > >
> > > >> > > > -Vibhav
> > > >> > > >
> > > >> > > >
> > > >> > > > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> > > >> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > >> > > >
> > > >> > > >> Hi
> > > >> > > >>
> > > >> > > >> How big is your row? Are they wider rows and what would be
> the
> > > >> size
> > > >> > of
> > > >> > > >> every cell?
> > > >> > > >> How many read threads are getting used?
> > > >> > > >>
> > > >> > > >>
> > > >
Re: Read thruput
Posted by Vibhav Mundra <mu...@gmail.com>.
How does your client call looks like? Get? Scan? Filters?
--My client keeps doing the Get request. Each time a single row is fetched.
Essentially we are using Hbase as key value retrieval.
Is 3000/sec is client side calls or is it in numbers of rows per sec?
--3000/sec is the client side calls.
If you measure in MB/sec how much read throughput do you get?
--Each client request's response is at maximum 1 KB so its the MB/sec is
3MB { 3000 * 1 KB }.
Where is your client located? Same router as the cluster?
--It is routed on the same cluster, on the same subnet.
Have you activated dfs read short circuit? Of not try it.
--I have not tried this. Let me try this also.
Compression - try switching to Snappy - should be faster.
What else is running on the cluster parallel to your reading client?
--There is small upload code running. I have never seen the CPU usage more
than 5%, so actually didnt bother to look at this angle.
-Vibhav
On Tue, Apr 2, 2013 at 1:42 AM, Asaf Mesika <as...@gmail.com> wrote:
> How does your client call looks like? Get? Scan? Filters?
> Is 3000/sec is client side calls or is it in numbers of rows per sec?
> If you measure in MB/sec how much read throughput do you get?
> Where is your client located? Same router as the cluster?
> Have you activated dfs read short circuit? Of not try it.
> Compression - try switching to Snappy - should be faster.
> What else is running on the cluster parallel to your reading client?
>
> On Monday, April 1, 2013, Vibhav Mundra wrote:
>
> > What is the general read-thru put that one gets when using Hbase.
> >
> > I am not to able to achieve more than 3000/secs with a timeout of 50
> > millisecs.
> > In this case also there is 10% of them are timing-out.
> >
> > -Vibhav
> >
> >
> > On Mon, Apr 1, 2013 at 11:20 PM, Vibhav Mundra <mu...@gmail.com> wrote:
> >
> > > yes, I have changes the BLOCK CACHE % to 0.35.
> > >
> > > -Vibhav
> > >
> > >
> > > On Mon, Apr 1, 2013 at 10:20 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > >> I was aware of that discussion which was about MAX_FILESIZE and
> > BLOCKSIZE
> > >>
> > >> My suggestion was about block cache percentage.
> > >>
> > >> Cheers
> > >>
> > >>
> > >> On Mon, Apr 1, 2013 at 4:57 AM, Vibhav Mundra <mu...@gmail.com>
> wrote:
> > >>
> > >> > I have used the following site:
> > >> > http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
> > >> >
> > >> > to lessen the value of block cache.
> > >> >
> > >> > -Vibhav
> > >> >
> > >> >
> > >> > On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
> > >> >
> > >> > > Can you increase block cache size ?
> > >> > >
> > >> > > What version of hbase are you using ?
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com>
> wrote:
> > >> > >
> > >> > > > The typical size of each of my row is less than 1KB.
> > >> > > >
> > >> > > > Regarding the memory, I have used 8GB for Hbase regionservers
> and
> > 4
> > >> GB
> > >> > > for
> > >> > > > datanodes and I dont see them completely used. So I ruled out
> the
> > GC
> > >> > > aspect.
> > >> > > >
> > >> > > > In case u still believe that GC is an issue, I will upload the
> gc
> > >> logs.
> > >> > > >
> > >> > > > -Vibhav
> > >> > > >
> > >> > > >
> > >> > > > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> > >> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > >> > > >
> > >> > > >> Hi
> > >> > > >>
> > >> > > >> How big is your row? Are they wider rows and what would be the
> > >> size
> > >> > of
> > >> > > >> every cell?
> > >> > > >> How many read threads are getting used?
> > >> > > >>
> > >> > > >>
> > >> > > >> Were you able to take a thread dump when this was happening?
> > Have
> > >> you
> > >> > > seen
> > >> > > >> the GC log?
> > >> > > >> May be need some more info before we can think of the problem.
> > >> > > >>
> > >> > > >> Regards
> > >> > > >> Ram
> > >> > > >>
> > >> > > >>
> > >> > > >> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <
> mundra@gmail.com>
> > >> > wrote:
> > >> > > >>
> > >> > > >>> Hi All,
> > >> > > >>>
> > >> > > >>> I am trying to use Hbase for real-time data retrieval with a
> > >> timeout
> > >> > of
> > >> > > >> 50
> > >> > > >>> ms.
> > >> > > >>>
> > >> > > >>> I am using 2 machines as datanode and regionservers,
> > >> > > >>> and one machine as a master for hadoop and Hbase.
> > >> > > >>>
> > >> > > >>> But I am able to fire only 3000 queries per sec and 10% of
> them
> > >> are
> > >> > > >> timing
> > >> > > >>> out.
> > >> > > >>> The database has 60 million rows.
> > >> > > >>>
> > >> > > >>> Are these figure okie, or I am missing something.
> > >> > > >>> I have used the scanner caching to be equal to one, because
> for
> > >> each
> > >> > > time
> > >> > >
>
Re: Read thruput
Posted by Asaf Mesika <as...@gmail.com>.
How does your client call looks like? Get? Scan? Filters?
Is 3000/sec is client side calls or is it in numbers of rows per sec?
If you measure in MB/sec how much read throughput do you get?
Where is your client located? Same router as the cluster?
Have you activated dfs read short circuit? Of not try it.
Compression - try switching to Snappy - should be faster.
What else is running on the cluster parallel to your reading client?
On Monday, April 1, 2013, Vibhav Mundra wrote:
> What is the general read-thru put that one gets when using Hbase.
>
> I am not to able to achieve more than 3000/secs with a timeout of 50
> millisecs.
> In this case also there is 10% of them are timing-out.
>
> -Vibhav
>
>
> On Mon, Apr 1, 2013 at 11:20 PM, Vibhav Mundra <mu...@gmail.com> wrote:
>
> > yes, I have changes the BLOCK CACHE % to 0.35.
> >
> > -Vibhav
> >
> >
> > On Mon, Apr 1, 2013 at 10:20 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> I was aware of that discussion which was about MAX_FILESIZE and
> BLOCKSIZE
> >>
> >> My suggestion was about block cache percentage.
> >>
> >> Cheers
> >>
> >>
> >> On Mon, Apr 1, 2013 at 4:57 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> >>
> >> > I have used the following site:
> >> > http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
> >> >
> >> > to lessen the value of block cache.
> >> >
> >> > -Vibhav
> >> >
> >> >
> >> > On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
> >> >
> >> > > Can you increase block cache size ?
> >> > >
> >> > > What version of hbase are you using ?
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> >> > >
> >> > > > The typical size of each of my row is less than 1KB.
> >> > > >
> >> > > > Regarding the memory, I have used 8GB for Hbase regionservers and
> 4
> >> GB
> >> > > for
> >> > > > datanodes and I dont see them completely used. So I ruled out the
> GC
> >> > > aspect.
> >> > > >
> >> > > > In case u still believe that GC is an issue, I will upload the gc
> >> logs.
> >> > > >
> >> > > > -Vibhav
> >> > > >
> >> > > >
> >> > > > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> >> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> >> > > >
> >> > > >> Hi
> >> > > >>
> >> > > >> How big is your row? Are they wider rows and what would be the
> >> size
> >> > of
> >> > > >> every cell?
> >> > > >> How many read threads are getting used?
> >> > > >>
> >> > > >>
> >> > > >> Were you able to take a thread dump when this was happening?
> Have
> >> you
> >> > > seen
> >> > > >> the GC log?
> >> > > >> May be need some more info before we can think of the problem.
> >> > > >>
> >> > > >> Regards
> >> > > >> Ram
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com>
> >> > wrote:
> >> > > >>
> >> > > >>> Hi All,
> >> > > >>>
> >> > > >>> I am trying to use Hbase for real-time data retrieval with a
> >> timeout
> >> > of
> >> > > >> 50
> >> > > >>> ms.
> >> > > >>>
> >> > > >>> I am using 2 machines as datanode and regionservers,
> >> > > >>> and one machine as a master for hadoop and Hbase.
> >> > > >>>
> >> > > >>> But I am able to fire only 3000 queries per sec and 10% of them
> >> are
> >> > > >> timing
> >> > > >>> out.
> >> > > >>> The database has 60 million rows.
> >> > > >>>
> >> > > >>> Are these figure okie, or I am missing something.
> >> > > >>> I have used the scanner caching to be equal to one, because for
> >> each
> >> > > time
> >> > >
Re: Read thruput
Posted by Vibhav Mundra <mu...@gmail.com>.
What is the general read-thru put that one gets when using Hbase.
I am not to able to achieve more than 3000/secs with a timeout of 50
millisecs.
In this case also there is 10% of them are timing-out.
-Vibhav
On Mon, Apr 1, 2013 at 11:20 PM, Vibhav Mundra <mu...@gmail.com> wrote:
> yes, I have changes the BLOCK CACHE % to 0.35.
>
> -Vibhav
>
>
> On Mon, Apr 1, 2013 at 10:20 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> I was aware of that discussion which was about MAX_FILESIZE and BLOCKSIZE
>>
>> My suggestion was about block cache percentage.
>>
>> Cheers
>>
>>
>> On Mon, Apr 1, 2013 at 4:57 AM, Vibhav Mundra <mu...@gmail.com> wrote:
>>
>> > I have used the following site:
>> > http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
>> >
>> > to lessen the value of block cache.
>> >
>> > -Vibhav
>> >
>> >
>> > On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
>> >
>> > > Can you increase block cache size ?
>> > >
>> > > What version of hbase are you using ?
>> > >
>> > > Thanks
>> > >
>> > > On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com> wrote:
>> > >
>> > > > The typical size of each of my row is less than 1KB.
>> > > >
>> > > > Regarding the memory, I have used 8GB for Hbase regionservers and 4
>> GB
>> > > for
>> > > > datanodes and I dont see them completely used. So I ruled out the GC
>> > > aspect.
>> > > >
>> > > > In case u still believe that GC is an issue, I will upload the gc
>> logs.
>> > > >
>> > > > -Vibhav
>> > > >
>> > > >
>> > > > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
>> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
>> > > >
>> > > >> Hi
>> > > >>
>> > > >> How big is your row? Are they wider rows and what would be the
>> size
>> > of
>> > > >> every cell?
>> > > >> How many read threads are getting used?
>> > > >>
>> > > >>
>> > > >> Were you able to take a thread dump when this was happening? Have
>> you
>> > > seen
>> > > >> the GC log?
>> > > >> May be need some more info before we can think of the problem.
>> > > >>
>> > > >> Regards
>> > > >> Ram
>> > > >>
>> > > >>
>> > > >> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com>
>> > wrote:
>> > > >>
>> > > >>> Hi All,
>> > > >>>
>> > > >>> I am trying to use Hbase for real-time data retrieval with a
>> timeout
>> > of
>> > > >> 50
>> > > >>> ms.
>> > > >>>
>> > > >>> I am using 2 machines as datanode and regionservers,
>> > > >>> and one machine as a master for hadoop and Hbase.
>> > > >>>
>> > > >>> But I am able to fire only 3000 queries per sec and 10% of them
>> are
>> > > >> timing
>> > > >>> out.
>> > > >>> The database has 60 million rows.
>> > > >>>
>> > > >>> Are these figure okie, or I am missing something.
>> > > >>> I have used the scanner caching to be equal to one, because for
>> each
>> > > time
>> > > >>> we are fetching a single row only.
>> > > >>>
>> > > >>> Here are the various configurations:
>> > > >>>
>> > > >>> *Our schema
>> > > >>> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf',
>> DATA_BLOCK_ENCODING
>> > =>
>> > > >>> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0',
>> > COMPRESSION
>> > > =>
>> > > >>> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0',
>> KEE
>> > > >>> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
>> > > >> 'true',
>> > > >>> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>> > > >>>
>> > > >>> *Configuration*
>> > > >>> 1 Machine having both hbase and hadoop master
>> > > >>> 2 machines having both region server node and datanode
>> > > >>> total 285 region servers
>> > > >>>
>> > > >>> *Machine Level Optimizations:*
>> > > >>> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
>> > > >>> b)Increase the read-ahead value to 4096
>> > > >>> c)Added noatime,nodiratime to the disks
>> > > >>>
>> > > >>> *Hadoop Optimizations:*
>> > > >>> dfs.datanode.max.xcievers = 4096
>> > > >>> dfs.block.size = 33554432
>> > > >>> dfs.datanode.handler.count = 256
>> > > >>> io.file.buffer.size = 65536
>> > > >>> hadoop data is split on 4 directories, so that different disks are
>> > > being
>> > > >>> accessed
>> > > >>>
>> > > >>> *Hbase Optimizations*:
>> > > >>>
>> > > >>> hbase.client.scanner.caching=1 #We have specifcally added this,
>> as
>> > we
>> > > >>> return always one row.
>> > > >>> hbase.regionserver.handler.count=3200
>> > > >>> hfile.block.cache.size=0.35
>> > > >>> hbase.hregion.memstore.mslab.enabled=true
>> > > >>> hfile.min.blocksize.size=16384
>> > > >>> hfile.min.blocksize.size=4
>> > > >>> hbase.hstore.blockingStoreFiles=200
>> > > >>> hbase.regionserver.optionallogflushinterval=60000
>> > > >>> hbase.hregion.majorcompaction=0
>> > > >>> hbase.hstore.compaction.max=100
>> > > >>> hbase.hstore.compactionThreshold=100
>> > > >>>
>> > > >>> *Hbase-GC
>> > > >>> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>> > -XX:+CMSParallelRemarkEnabled
>> > > >>> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
>> > > >>> *Hadoop-GC*
>> > > >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>> > > >>>
>> > > >>> -Vibhav
>> > > >>
>> > >
>> >
>>
>
>
Re: Read thruput
Posted by Vibhav Mundra <mu...@gmail.com>.
yes, I have changes the BLOCK CACHE % to 0.35.
-Vibhav
On Mon, Apr 1, 2013 at 10:20 PM, Ted Yu <yu...@gmail.com> wrote:
> I was aware of that discussion which was about MAX_FILESIZE and BLOCKSIZE
>
> My suggestion was about block cache percentage.
>
> Cheers
>
>
> On Mon, Apr 1, 2013 at 4:57 AM, Vibhav Mundra <mu...@gmail.com> wrote:
>
> > I have used the following site:
> > http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
> >
> > to lessen the value of block cache.
> >
> > -Vibhav
> >
> >
> > On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
> >
> > > Can you increase block cache size ?
> > >
> > > What version of hbase are you using ?
> > >
> > > Thanks
> > >
> > > On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> > >
> > > > The typical size of each of my row is less than 1KB.
> > > >
> > > > Regarding the memory, I have used 8GB for Hbase regionservers and 4
> GB
> > > for
> > > > datanodes and I dont see them completely used. So I ruled out the GC
> > > aspect.
> > > >
> > > > In case u still believe that GC is an issue, I will upload the gc
> logs.
> > > >
> > > > -Vibhav
> > > >
> > > >
> > > > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> How big is your row? Are they wider rows and what would be the size
> > of
> > > >> every cell?
> > > >> How many read threads are getting used?
> > > >>
> > > >>
> > > >> Were you able to take a thread dump when this was happening? Have
> you
> > > seen
> > > >> the GC log?
> > > >> May be need some more info before we can think of the problem.
> > > >>
> > > >> Regards
> > > >> Ram
> > > >>
> > > >>
> > > >> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com>
> > wrote:
> > > >>
> > > >>> Hi All,
> > > >>>
> > > >>> I am trying to use Hbase for real-time data retrieval with a
> timeout
> > of
> > > >> 50
> > > >>> ms.
> > > >>>
> > > >>> I am using 2 machines as datanode and regionservers,
> > > >>> and one machine as a master for hadoop and Hbase.
> > > >>>
> > > >>> But I am able to fire only 3000 queries per sec and 10% of them are
> > > >> timing
> > > >>> out.
> > > >>> The database has 60 million rows.
> > > >>>
> > > >>> Are these figure okie, or I am missing something.
> > > >>> I have used the scanner caching to be equal to one, because for
> each
> > > time
> > > >>> we are fetching a single row only.
> > > >>>
> > > >>> Here are the various configurations:
> > > >>>
> > > >>> *Our schema
> > > >>> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf',
> DATA_BLOCK_ENCODING
> > =>
> > > >>> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0',
> > COMPRESSION
> > > =>
> > > >>> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0',
> KEE
> > > >>> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
> > > >> 'true',
> > > >>> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
> > > >>>
> > > >>> *Configuration*
> > > >>> 1 Machine having both hbase and hadoop master
> > > >>> 2 machines having both region server node and datanode
> > > >>> total 285 region servers
> > > >>>
> > > >>> *Machine Level Optimizations:*
> > > >>> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> > > >>> b)Increase the read-ahead value to 4096
> > > >>> c)Added noatime,nodiratime to the disks
> > > >>>
> > > >>> *Hadoop Optimizations:*
> > > >>> dfs.datanode.max.xcievers = 4096
> > > >>> dfs.block.size = 33554432
> > > >>> dfs.datanode.handler.count = 256
> > > >>> io.file.buffer.size = 65536
> > > >>> hadoop data is split on 4 directories, so that different disks are
> > > being
> > > >>> accessed
> > > >>>
> > > >>> *Hbase Optimizations*:
> > > >>>
> > > >>> hbase.client.scanner.caching=1 #We have specifcally added this, as
> > we
> > > >>> return always one row.
> > > >>> hbase.regionserver.handler.count=3200
> > > >>> hfile.block.cache.size=0.35
> > > >>> hbase.hregion.memstore.mslab.enabled=true
> > > >>> hfile.min.blocksize.size=16384
> > > >>> hfile.min.blocksize.size=4
> > > >>> hbase.hstore.blockingStoreFiles=200
> > > >>> hbase.regionserver.optionallogflushinterval=60000
> > > >>> hbase.hregion.majorcompaction=0
> > > >>> hbase.hstore.compaction.max=100
> > > >>> hbase.hstore.compactionThreshold=100
> > > >>>
> > > >>> *Hbase-GC
> > > >>> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> > -XX:+CMSParallelRemarkEnabled
> > > >>> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> > > >>> *Hadoop-GC*
> > > >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> > > >>>
> > > >>> -Vibhav
> > > >>
> > >
> >
>
Re: Read thruput
Posted by Ted Yu <yu...@gmail.com>.
I was aware of that discussion which was about MAX_FILESIZE and BLOCKSIZE
My suggestion was about block cache percentage.
Cheers
On Mon, Apr 1, 2013 at 4:57 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> I have used the following site:
> http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
>
> to lessen the value of block cache.
>
> -Vibhav
>
>
> On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
>
> > Can you increase block cache size ?
> >
> > What version of hbase are you using ?
> >
> > Thanks
> >
> > On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> >
> > > The typical size of each of my row is less than 1KB.
> > >
> > > Regarding the memory, I have used 8GB for Hbase regionservers and 4 GB
> > for
> > > datanodes and I dont see them completely used. So I ruled out the GC
> > aspect.
> > >
> > > In case u still believe that GC is an issue, I will upload the gc logs.
> > >
> > > -Vibhav
> > >
> > >
> > > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > >
> > >> Hi
> > >>
> > >> How big is your row? Are they wider rows and what would be the size
> of
> > >> every cell?
> > >> How many read threads are getting used?
> > >>
> > >>
> > >> Were you able to take a thread dump when this was happening? Have you
> > seen
> > >> the GC log?
> > >> May be need some more info before we can think of the problem.
> > >>
> > >> Regards
> > >> Ram
> > >>
> > >>
> > >> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com>
> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> I am trying to use Hbase for real-time data retrieval with a timeout
> of
> > >> 50
> > >>> ms.
> > >>>
> > >>> I am using 2 machines as datanode and regionservers,
> > >>> and one machine as a master for hadoop and Hbase.
> > >>>
> > >>> But I am able to fire only 3000 queries per sec and 10% of them are
> > >> timing
> > >>> out.
> > >>> The database has 60 million rows.
> > >>>
> > >>> Are these figure okie, or I am missing something.
> > >>> I have used the scanner caching to be equal to one, because for each
> > time
> > >>> we are fetching a single row only.
> > >>>
> > >>> Here are the various configurations:
> > >>>
> > >>> *Our schema
> > >>> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING
> =>
> > >>> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0',
> COMPRESSION
> > =>
> > >>> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> > >>> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
> > >> 'true',
> > >>> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
> > >>>
> > >>> *Configuration*
> > >>> 1 Machine having both hbase and hadoop master
> > >>> 2 machines having both region server node and datanode
> > >>> total 285 region servers
> > >>>
> > >>> *Machine Level Optimizations:*
> > >>> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> > >>> b)Increase the read-ahead value to 4096
> > >>> c)Added noatime,nodiratime to the disks
> > >>>
> > >>> *Hadoop Optimizations:*
> > >>> dfs.datanode.max.xcievers = 4096
> > >>> dfs.block.size = 33554432
> > >>> dfs.datanode.handler.count = 256
> > >>> io.file.buffer.size = 65536
> > >>> hadoop data is split on 4 directories, so that different disks are
> > being
> > >>> accessed
> > >>>
> > >>> *Hbase Optimizations*:
> > >>>
> > >>> hbase.client.scanner.caching=1 #We have specifcally added this, as
> we
> > >>> return always one row.
> > >>> hbase.regionserver.handler.count=3200
> > >>> hfile.block.cache.size=0.35
> > >>> hbase.hregion.memstore.mslab.enabled=true
> > >>> hfile.min.blocksize.size=16384
> > >>> hfile.min.blocksize.size=4
> > >>> hbase.hstore.blockingStoreFiles=200
> > >>> hbase.regionserver.optionallogflushinterval=60000
> > >>> hbase.hregion.majorcompaction=0
> > >>> hbase.hstore.compaction.max=100
> > >>> hbase.hstore.compactionThreshold=100
> > >>>
> > >>> *Hbase-GC
> > >>> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> -XX:+CMSParallelRemarkEnabled
> > >>> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> > >>> *Hadoop-GC*
> > >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> > >>>
> > >>> -Vibhav
> > >>
> >
>
Re: Read thruput
Posted by Vibhav Mundra <mu...@gmail.com>.
I have used the following site:
http://grokbase.com/t/hbase/user/11bat80x7m/row-get-very-slow
to lessen the value of block cache.
-Vibhav
On Mon, Apr 1, 2013 at 4:23 PM, Ted <yu...@gmail.com> wrote:
> Can you increase block cache size ?
>
> What version of hbase are you using ?
>
> Thanks
>
> On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com> wrote:
>
> > The typical size of each of my row is less than 1KB.
> >
> > Regarding the memory, I have used 8GB for Hbase regionservers and 4 GB
> for
> > datanodes and I dont see them completely used. So I ruled out the GC
> aspect.
> >
> > In case u still believe that GC is an issue, I will upload the gc logs.
> >
> > -Vibhav
> >
> >
> > On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> >> Hi
> >>
> >> How big is your row? Are they wider rows and what would be the size of
> >> every cell?
> >> How many read threads are getting used?
> >>
> >>
> >> Were you able to take a thread dump when this was happening? Have you
> seen
> >> the GC log?
> >> May be need some more info before we can think of the problem.
> >>
> >> Regards
> >> Ram
> >>
> >>
> >> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com> wrote:
> >>
> >>> Hi All,
> >>>
> >>> I am trying to use Hbase for real-time data retrieval with a timeout of
> >> 50
> >>> ms.
> >>>
> >>> I am using 2 machines as datanode and regionservers,
> >>> and one machine as a master for hadoop and Hbase.
> >>>
> >>> But I am able to fire only 3000 queries per sec and 10% of them are
> >> timing
> >>> out.
> >>> The database has 60 million rows.
> >>>
> >>> Are these figure okie, or I am missing something.
> >>> I have used the scanner caching to be equal to one, because for each
> time
> >>> we are fetching a single row only.
> >>>
> >>> Here are the various configurations:
> >>>
> >>> *Our schema
> >>> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
> >>> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION
> =>
> >>> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> >>> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
> >> 'true',
> >>> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
> >>>
> >>> *Configuration*
> >>> 1 Machine having both hbase and hadoop master
> >>> 2 machines having both region server node and datanode
> >>> total 285 region servers
> >>>
> >>> *Machine Level Optimizations:*
> >>> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> >>> b)Increase the read-ahead value to 4096
> >>> c)Added noatime,nodiratime to the disks
> >>>
> >>> *Hadoop Optimizations:*
> >>> dfs.datanode.max.xcievers = 4096
> >>> dfs.block.size = 33554432
> >>> dfs.datanode.handler.count = 256
> >>> io.file.buffer.size = 65536
> >>> hadoop data is split on 4 directories, so that different disks are
> being
> >>> accessed
> >>>
> >>> *Hbase Optimizations*:
> >>>
> >>> hbase.client.scanner.caching=1 #We have specifcally added this, as we
> >>> return always one row.
> >>> hbase.regionserver.handler.count=3200
> >>> hfile.block.cache.size=0.35
> >>> hbase.hregion.memstore.mslab.enabled=true
> >>> hfile.min.blocksize.size=16384
> >>> hfile.min.blocksize.size=4
> >>> hbase.hstore.blockingStoreFiles=200
> >>> hbase.regionserver.optionallogflushinterval=60000
> >>> hbase.hregion.majorcompaction=0
> >>> hbase.hstore.compaction.max=100
> >>> hbase.hstore.compactionThreshold=100
> >>>
> >>> *Hbase-GC
> >>> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
> >>> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> >>> *Hadoop-GC*
> >>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> >>>
> >>> -Vibhav
> >>
>
Re: Read thruput
Posted by Ted <yu...@gmail.com>.
Can you increase block cache size ?
What version of hbase are you using ?
Thanks
On Apr 1, 2013, at 3:47 AM, Vibhav Mundra <mu...@gmail.com> wrote:
> The typical size of each of my row is less than 1KB.
>
> Regarding the memory, I have used 8GB for Hbase regionservers and 4 GB for
> datanodes and I dont see them completely used. So I ruled out the GC aspect.
>
> In case u still believe that GC is an issue, I will upload the gc logs.
>
> -Vibhav
>
>
> On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
>> Hi
>>
>> How big is your row? Are they wider rows and what would be the size of
>> every cell?
>> How many read threads are getting used?
>>
>>
>> Were you able to take a thread dump when this was happening? Have you seen
>> the GC log?
>> May be need some more info before we can think of the problem.
>>
>> Regards
>> Ram
>>
>>
>> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I am trying to use Hbase for real-time data retrieval with a timeout of
>> 50
>>> ms.
>>>
>>> I am using 2 machines as datanode and regionservers,
>>> and one machine as a master for hadoop and Hbase.
>>>
>>> But I am able to fire only 3000 queries per sec and 10% of them are
>> timing
>>> out.
>>> The database has 60 million rows.
>>>
>>> Are these figure okie, or I am missing something.
>>> I have used the scanner caching to be equal to one, because for each time
>>> we are fetching a single row only.
>>>
>>> Here are the various configurations:
>>>
>>> *Our schema
>>> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
>>> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
>>> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
>>> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
>> 'true',
>>> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>>>
>>> *Configuration*
>>> 1 Machine having both hbase and hadoop master
>>> 2 machines having both region server node and datanode
>>> total 285 region servers
>>>
>>> *Machine Level Optimizations:*
>>> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
>>> b)Increase the read-ahead value to 4096
>>> c)Added noatime,nodiratime to the disks
>>>
>>> *Hadoop Optimizations:*
>>> dfs.datanode.max.xcievers = 4096
>>> dfs.block.size = 33554432
>>> dfs.datanode.handler.count = 256
>>> io.file.buffer.size = 65536
>>> hadoop data is split on 4 directories, so that different disks are being
>>> accessed
>>>
>>> *Hbase Optimizations*:
>>>
>>> hbase.client.scanner.caching=1 #We have specifcally added this, as we
>>> return always one row.
>>> hbase.regionserver.handler.count=3200
>>> hfile.block.cache.size=0.35
>>> hbase.hregion.memstore.mslab.enabled=true
>>> hfile.min.blocksize.size=16384
>>> hfile.min.blocksize.size=4
>>> hbase.hstore.blockingStoreFiles=200
>>> hbase.regionserver.optionallogflushinterval=60000
>>> hbase.hregion.majorcompaction=0
>>> hbase.hstore.compaction.max=100
>>> hbase.hstore.compactionThreshold=100
>>>
>>> *Hbase-GC
>>> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
>>> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
>>> *Hadoop-GC*
>>> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>>>
>>> -Vibhav
>>
Re: Read thruput
Posted by Vibhav Mundra <mu...@gmail.com>.
The typical size of each of my row is less than 1KB.
Regarding the memory, I have used 8GB for Hbase regionservers and 4 GB for
datanodes and I dont see them completely used. So I ruled out the GC aspect.
In case u still believe that GC is an issue, I will upload the gc logs.
-Vibhav
On Mon, Apr 1, 2013 at 3:46 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Hi
>
> How big is your row? Are they wider rows and what would be the size of
> every cell?
> How many read threads are getting used?
>
>
> Were you able to take a thread dump when this was happening? Have you seen
> the GC log?
> May be need some more info before we can think of the problem.
>
> Regards
> Ram
>
>
> On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am trying to use Hbase for real-time data retrieval with a timeout of
> 50
> > ms.
> >
> > I am using 2 machines as datanode and regionservers,
> > and one machine as a master for hadoop and Hbase.
> >
> > But I am able to fire only 3000 queries per sec and 10% of them are
> timing
> > out.
> > The database has 60 million rows.
> >
> > Are these figure okie, or I am missing something.
> > I have used the scanner caching to be equal to one, because for each time
> > we are fetching a single row only.
> >
> > Here are the various configurations:
> >
> > *Our schema
> > *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
> > 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
> > 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> > P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK =>
> 'true',
> > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
> >
> > *Configuration*
> > 1 Machine having both hbase and hadoop master
> > 2 machines having both region server node and datanode
> > total 285 region servers
> >
> > *Machine Level Optimizations:*
> > a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> > b)Increase the read-ahead value to 4096
> > c)Added noatime,nodiratime to the disks
> >
> > *Hadoop Optimizations:*
> > dfs.datanode.max.xcievers = 4096
> > dfs.block.size = 33554432
> > dfs.datanode.handler.count = 256
> > io.file.buffer.size = 65536
> > hadoop data is split on 4 directories, so that different disks are being
> > accessed
> >
> > *Hbase Optimizations*:
> >
> > hbase.client.scanner.caching=1 #We have specifcally added this, as we
> > return always one row.
> > hbase.regionserver.handler.count=3200
> > hfile.block.cache.size=0.35
> > hbase.hregion.memstore.mslab.enabled=true
> > hfile.min.blocksize.size=16384
> > hfile.min.blocksize.size=4
> > hbase.hstore.blockingStoreFiles=200
> > hbase.regionserver.optionallogflushinterval=60000
> > hbase.hregion.majorcompaction=0
> > hbase.hstore.compaction.max=100
> > hbase.hstore.compactionThreshold=100
> >
> > *Hbase-GC
> > *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
> > -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> > *Hadoop-GC*
> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
> >
> > -Vibhav
> >
>
Re: Read thruput
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Hi
How big is your row? Are they wider rows and what would be the size of
every cell?
How many read threads are getting used?
Were you able to take a thread dump when this was happening? Have you seen
the GC log?
May be need some more info before we can think of the problem.
Regards
Ram
On Mon, Apr 1, 2013 at 3:39 PM, Vibhav Mundra <mu...@gmail.com> wrote:
> Hi All,
>
> I am trying to use Hbase for real-time data retrieval with a timeout of 50
> ms.
>
> I am using 2 machines as datanode and regionservers,
> and one machine as a master for hadoop and Hbase.
>
> But I am able to fire only 3000 queries per sec and 10% of them are timing
> out.
> The database has 60 million rows.
>
> Are these figure okie, or I am missing something.
> I have used the scanner caching to be equal to one, because for each time
> we are fetching a single row only.
>
> Here are the various configurations:
>
> *Our schema
> *{NAME => 'mytable', FAMILIES => [{NAME => 'cf', DATA_BLOCK_ENCODING =>
> 'NONE', BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '0', COMPRESSION =>
> 'GZ', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0', KEE
> P_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>
> *Configuration*
> 1 Machine having both hbase and hadoop master
> 2 machines having both region server node and datanode
> total 285 region servers
>
> *Machine Level Optimizations:*
> a)No of file descriptors is 1000000(ulimit -n gives 1000000)
> b)Increase the read-ahead value to 4096
> c)Added noatime,nodiratime to the disks
>
> *Hadoop Optimizations:*
> dfs.datanode.max.xcievers = 4096
> dfs.block.size = 33554432
> dfs.datanode.handler.count = 256
> io.file.buffer.size = 65536
> hadoop data is split on 4 directories, so that different disks are being
> accessed
>
> *Hbase Optimizations*:
>
> hbase.client.scanner.caching=1 #We have specifcally added this, as we
> return always one row.
> hbase.regionserver.handler.count=3200
> hfile.block.cache.size=0.35
> hbase.hregion.memstore.mslab.enabled=true
> hfile.min.blocksize.size=16384
> hfile.min.blocksize.size=4
> hbase.hstore.blockingStoreFiles=200
> hbase.regionserver.optionallogflushinterval=60000
> hbase.hregion.majorcompaction=0
> hbase.hstore.compaction.max=100
> hbase.hstore.compactionThreshold=100
>
> *Hbase-GC
> *-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=20 -XX:ParallelGCThreads=16
> *Hadoop-GC*
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
>
> -Vibhav
>