Insert, update performance

[Adam Silberstein <si...@yahoo-inc.com>]

Hi,
I¹m currently doing some tests comparing 2 use cases:
6 servers
2400 ops per second from 100 clients
Enough ops total to take 30 minutes (2400 * 60 * 30)
120 million 1KB records to start

Case A: Mix of 95% reads, 5% updates
Case B: Mix of 95% reads, 5% inserts

>From a client API standpoint, these cases are identical. The only difference
is that in Case A I only do operations on the 120 million keys I already
inserted.  In Case B, I do operations on new keys.

I was expecting performance to be similar, but I¹m having problems with Case
B.  In particular, after running continuously for 5-10, the latencies go way
up and my throughput goes way down to a couple of operations a second.  I
left it like this for a while in case there were compactions going on, but
it never returned to its original throughput.  I have no problems with Case
A.  

Any ideas on what¹s going on here?  It may be related to the concurrent read
workload, since I successfully loaded 120 million records originally, and
that took several hours.

Thanks,
Adam

Re: Insert, update performance

[stack <st...@duboce.net>]

On Wed, Dec 16, 2009 at 7:52 AM, Adam Silberstein <si...@yahoo-inc.com>wrote:

> I was expecting performance to be similar, but I¹m having problems with
> Case
> B.  In particular, after running continuously for 5-10, the latencies go
> way
> up and my throughput goes way down to a couple of operations a second.  I
> left it like this for a while in case there were compactions going on, but
> it never returned to its original throughput.  I have no problems with Case
> A.
>

So, if you look in the logs on the regionservers, do you see anything?  In
particular, there is a writes barrier that we'll bring down if we are not
flushing fast enough.  The default is pretty conservative so the barrier
comes down pretty quick.  The read load might be slowing the flushing?

Or are splits happening at this time?  Are number of regions climbing during
test B?

If you want, post a few regionserver logs somewhere we can pull it and we'll
take a look at it for you.




> Any ideas on what¹s going on here?  It may be related to the concurrent
> read
> workload, since I successfully loaded 120 million records originally, and
> that took several hours.



"serveral hours" is good or bad by your estimation?  Should it be faster?

Thanks,
St.Ack