Concurrent reads throw thrift

[ruslan usifov <ru...@gmail.com>]

Hi!

I have cluster from 3 machines:

firts(srv1) have NameNode, DataNode, Hmaster, Hregionserver, Thrift on it
second(srv2) have DataNode, Hregionserver, HQuorumPerr, Thrift on it.
third(srv3) have DataNode, Hregionserver

and have follow python program, that run in 8 separate process

import random;
import time;

from thrift import Thrift
from thrift.transport import TSocket, TTransport
from thrift.protocol import TBinaryProtocol
from hbase import ttypes
from hbase.Hbase import Client, ColumnDescriptor, Mutation, BatchMutation

def get_client(host):
  transport = TSocket.TSocket(host, 9090)

  # Buffering is critical. Raw sockets are very slow
  transport = TTransport.TBufferedTransport(transport)

  # Wrap in a protocol
  protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport)

  # Create a client to use the protocol encoder
  client = Client(protocol)

  # Connect!
  transport.open()

  return client

clients = [get_client("192.168.0.241"), get_client("192.168.0.242")];
begin = time.time();
maxrange = 20000000 - 1;

for i in xrange(500000):
  if (i != 0) and ((i % 10000) == 0):
    tm = time.time();
    print tm - begin;
    begin = tm;

  client = clients[random.randint(0, len(clients) - 1)];
  client.getRow("test", str(random.randint(0, maxrange)));

print time.time() - begin;


And i have very big latency (about 300 request per seconds). If i run that
program in 4 process latency is slow but i think is not good (making about
1000 request per second) It's very strange that and cpu and discs are not
100% load
Please help!!! What i do wrong?