You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by y_...@tsmc.com on 2010/03/01 03:53:49 UTC
Re: HBase reading performance
Hi,
I started 200 clients(spread it to 20 machines) to run NewHTableTest like
following code, which took 983 seconds.
META table just resides in one region that machine CPU and network traffic
are very high
while running NewHTableTest,so I guess there is a bottleneck from Zookeeper
or META table server.
Any suggestion?
My Cluster:
1U servers(4core,12G ram): 20
zookeepers : 3
region servers : 10
regions :1500
public void NewHTableTest() throws IOException {
IndexedTable idxTable1= new IndexedTable(config,
Bytes.toBytes("Table1"));
IndexedTable idxTable2= new IndexedTable(config,
Bytes.toBytes("Table2"));
IndexedTable idxTable3= new IndexedTable(config,
Bytes.toBytes("Table3"));
IndexedTable idxTable4= new IndexedTable(config,
Bytes.toBytes("Table4"));
IndexedTable idxTable5= new IndexedTable(config,
Bytes.toBytes("Table5"));
IndexedTable idxTable6= new IndexedTable(config,
Bytes.toBytes("Table6"));
IndexedTable idxTable7= new IndexedTable(config,
Bytes.toBytes("Table7"));
IndexedTable idxTable8= new IndexedTable(config,
Bytes.toBytes("Table8"));
IndexedTable idxTable9= new IndexedTable(config,
Bytes.toBytes("Table9"));
IndexedTable idxTable10= new IndexedTable(config,
Bytes.toBytes("Table10"));
IndexedTable idxTable11= new IndexedTable(config,
Bytes.toBytes("Table11"));
IndexedTable idxTable12= new IndexedTable(config,
Bytes.toBytes("Table12"));
IndexedTable idxTable13= new IndexedTable(config,
Bytes.toBytes("Table13"));
IndexedTable idxTable14= new IndexedTable(config,
Bytes.toBytes("Table14"));
IndexedTable idxTable15= new IndexedTable(config,
Bytes.toBytes("Table15"));
IndexedTable idxTable16= new IndexedTable(config,
Bytes.toBytes("Table16"));
IndexedTable idxTable17= new IndexedTable(config,
Bytes.toBytes("Table17"));
IndexedTable idxTable18= new IndexedTable(config,
Bytes.toBytes("Table18"));
IndexedTable idxTable19= new IndexedTable(config,
Bytes.toBytes("Table19"));
IndexedTable idxTable20= new IndexedTable(config,
Bytes.toBytes("Table20"));
IndexedTable idxTable21= new IndexedTable(config,
Bytes.toBytes("Table21"));
IndexedTable idxTable22= new IndexedTable(config,
Bytes.toBytes("Table22"));
IndexedTable idxTable23= new IndexedTable(config,
Bytes.toBytes("Table23"));
IndexedTable idxTable24= new IndexedTable(config,
Bytes.toBytes("Table24"));
IndexedTable idxTable25= new IndexedTable(config,
Bytes.toBytes("Table25"));
IndexedTable idxTable26= new IndexedTable(config,
Bytes.toBytes("Table26"));
IndexedTable idxTable27= new IndexedTable(config,
Bytes.toBytes("Table27"));
IndexedTable idxTable28= new IndexedTable(config,
Bytes.toBytes("Table28"));
IndexedTable idxTable29= new IndexedTable(config,
Bytes.toBytes("Table29"));
IndexedTable idxTable30= new IndexedTable(config,
Bytes.toBytes("Table30"));
}
Fleming Chiu(邱宏明)
707-6128
y_823910@tsmc.com
週一無肉日吃素救地球(Meat Free Monday Taiwan)
---------------------------------------------------------------------------
TSMC PROPERTY
This email communication (and any attachments) is proprietary information
for the sole use of its
intended recipient. Any unauthorized review, use or distribution by anyone
other than the intended
recipient is strictly prohibited. If you are not the intended recipient,
please notify the sender by
replying to this email, and then delete this email and any copies of it
immediately. Thank you.
---------------------------------------------------------------------------
Re: HBase reading performance
Posted by Jean-Daniel Cryans <jd...@apache.org>.
In this particular case a lot of things come in action:
- Creating a table is a long process because the client sleeps a lot,
6 seconds before 0.20.3, 2 seconds in 0.20.3 and even less than that
in the current head of branch.
- in 0.20, without the HDFS-200 patch, HDFS doesn't support fs syncs
so we force memstore flushes at something like 8MB so that you don't
lose too much data on that very important table (hopefully in 0.21
it's supported, no data loss yeah!). So all those memstore flushes can
account for a lot of traffic and can generate a lot more compactions.
What exactly is your test trying to show? I'm really not sure... that
tables with very small memstores take edits at a slower rate?
J-D
2010/2/28 <y_...@tsmc.com>:
> Hi,
> I started 200 clients(spread it to 20 machines) to run NewHTableTest like
> following code, which took 983 seconds.
> META table just resides in one region that machine CPU and network traffic
> are very high
> while running NewHTableTest,so I guess there is a bottleneck from Zookeeper
> or META table server.
> Any suggestion?
>
>
> My Cluster:
> 1U servers(4core,12G ram): 20
> zookeepers : 3
> region servers : 10
> regions :1500
>
>
> public void NewHTableTest() throws IOException {
> IndexedTable idxTable1= new IndexedTable(config,
> Bytes.toBytes("Table1"));
> IndexedTable idxTable2= new IndexedTable(config,
> Bytes.toBytes("Table2"));
> IndexedTable idxTable3= new IndexedTable(config,
> Bytes.toBytes("Table3"));
> IndexedTable idxTable4= new IndexedTable(config,
> Bytes.toBytes("Table4"));
> IndexedTable idxTable5= new IndexedTable(config,
> Bytes.toBytes("Table5"));
> IndexedTable idxTable6= new IndexedTable(config,
> Bytes.toBytes("Table6"));
> IndexedTable idxTable7= new IndexedTable(config,
> Bytes.toBytes("Table7"));
> IndexedTable idxTable8= new IndexedTable(config,
> Bytes.toBytes("Table8"));
> IndexedTable idxTable9= new IndexedTable(config,
> Bytes.toBytes("Table9"));
> IndexedTable idxTable10= new IndexedTable(config,
> Bytes.toBytes("Table10"));
> IndexedTable idxTable11= new IndexedTable(config,
> Bytes.toBytes("Table11"));
> IndexedTable idxTable12= new IndexedTable(config,
> Bytes.toBytes("Table12"));
> IndexedTable idxTable13= new IndexedTable(config,
> Bytes.toBytes("Table13"));
> IndexedTable idxTable14= new IndexedTable(config,
> Bytes.toBytes("Table14"));
> IndexedTable idxTable15= new IndexedTable(config,
> Bytes.toBytes("Table15"));
> IndexedTable idxTable16= new IndexedTable(config,
> Bytes.toBytes("Table16"));
> IndexedTable idxTable17= new IndexedTable(config,
> Bytes.toBytes("Table17"));
> IndexedTable idxTable18= new IndexedTable(config,
> Bytes.toBytes("Table18"));
> IndexedTable idxTable19= new IndexedTable(config,
> Bytes.toBytes("Table19"));
> IndexedTable idxTable20= new IndexedTable(config,
> Bytes.toBytes("Table20"));
> IndexedTable idxTable21= new IndexedTable(config,
> Bytes.toBytes("Table21"));
> IndexedTable idxTable22= new IndexedTable(config,
> Bytes.toBytes("Table22"));
> IndexedTable idxTable23= new IndexedTable(config,
> Bytes.toBytes("Table23"));
> IndexedTable idxTable24= new IndexedTable(config,
> Bytes.toBytes("Table24"));
> IndexedTable idxTable25= new IndexedTable(config,
> Bytes.toBytes("Table25"));
> IndexedTable idxTable26= new IndexedTable(config,
> Bytes.toBytes("Table26"));
> IndexedTable idxTable27= new IndexedTable(config,
> Bytes.toBytes("Table27"));
> IndexedTable idxTable28= new IndexedTable(config,
> Bytes.toBytes("Table28"));
> IndexedTable idxTable29= new IndexedTable(config,
> Bytes.toBytes("Table29"));
> IndexedTable idxTable30= new IndexedTable(config,
> Bytes.toBytes("Table30"));
> }
>
>
>
>
> Fleming Chiu(邱宏明)
> 707-6128
> y_823910@tsmc.com
> 週一無肉日吃素救地球(Meat Free Monday Taiwan)
>
>
> ---------------------------------------------------------------------------
> TSMC PROPERTY
> This email communication (and any attachments) is proprietary information
> for the sole use of its
> intended recipient. Any unauthorized review, use or distribution by anyone
> other than the intended
> recipient is strictly prohibited. If you are not the intended recipient,
> please notify the sender by
> replying to this email, and then delete this email and any copies of it
> immediately. Thank you.
> ---------------------------------------------------------------------------
>
>
>
>