write performance regression in CDH 5.9?

[Graham Baecher <gb...@hubspot.com>]

Hi all,

We're looking to update the HBase version we're running on our servers, but
noticed while comparing performance test results a the previous version
that Puts now seem to be slower.
Specifically, comparing HBase 1.2.0-cdh5.9.0 to 1.2.0-cdh5.8.0 using YCSB,
it looks like mean read latency is improved in cdh5.9 but mean write
latency is worse, and the net effect in a 50% read/50% write workload is a
performance degradation. FWIW, we've been testing with
`hbase.ipc.server.callqueue.type` = "deadline" on both versions.

Has anyone else using CDH 5.9.0 HBase run into this, or know what might be
the cause?

~Graham

Re: write performance regression in CDH 5.9?

[Stack <st...@duboce.net>]

On Fri, Nov 18, 2016 at 7:46 AM, Graham Baecher <gb...@hubspot.com>
wrote:

> Hi all,
>
> We're looking to update the HBase version we're running on our servers, but
> noticed while comparing performance test results a the previous version
> that Puts now seem to be slower.
> Specifically, comparing HBase 1.2.0-cdh5.9.0 to 1.2.0-cdh5.8.0 using YCSB,
> it looks like mean read latency is improved in cdh5.9 but mean write
> latency is worse, and the net effect in a 50% read/50% write workload is a
> performance degradation. FWIW, we've been testing with
> `hbase.ipc.server.callqueue.type` = "deadline" on both versions.
>
> Has anyone else using CDH 5.9.0 HBase run into this, or know what might be
> the cause?
>
>
Hey Graham:

If you do pure YCSB write, do you see a difference? You updated HBase and
HDFS? Whats cluster size and commands you are running so can try replicate?

You are using a vendor's hbase so we'd suggest you go ping the vendor on
their mailing lists. That said, I'm interested if a regression in the 1.2
line (I'm not sure anyone has run compares of patch versions on 1.2 line).
I'm game for trying to figure which commit was responsible. I
opened HBASE-17128 as a working issue on which we can ping/pong
configs/test runs.

S



> ~Graham
>

Re: write performance regression in CDH 5.9?

[Graham Baecher <gb...@hubspot.com>]

Thanks Jeremy.
I ran the regressed workload this morning with FIFO enabled and though it
was faster, it looks like it was a flat ~1ms faster for both reads and
writes. This doesn't compensate for the write performance dropoff, so even
with FIFO our 5.9 writes are slower than 5.8.

On Fri, Nov 18, 2016 at 11:29 AM, Jeremy Carroll <ph...@gmail.com>
wrote:

> Use FIFO. Much better in our testing
>
> On Fri, Nov 18, 2016 at 7:46 AM Graham Baecher <gb...@hubspot.com>
> wrote:
>
> > Hi all,
> >
> > We're looking to update the HBase version we're running on our servers,
> but
> > noticed while comparing performance test results a the previous version
> > that Puts now seem to be slower.
> > Specifically, comparing HBase 1.2.0-cdh5.9.0 to 1.2.0-cdh5.8.0 using
> YCSB,
> > it looks like mean read latency is improved in cdh5.9 but mean write
> > latency is worse, and the net effect in a 50% read/50% write workload is
> a
> > performance degradation. FWIW, we've been testing with
> > `hbase.ipc.server.callqueue.type` = "deadline" on both versions.
> >
> > Has anyone else using CDH 5.9.0 HBase run into this, or know what might
> be
> > the cause?
> >
> > ~Graham
> >
>

Re: write performance regression in CDH 5.9?

[Jeremy Carroll <ph...@gmail.com>]

Use FIFO. Much better in our testing

On Fri, Nov 18, 2016 at 7:46 AM Graham Baecher <gb...@hubspot.com> wrote:

> Hi all,
>
> We're looking to update the HBase version we're running on our servers, but
> noticed while comparing performance test results a the previous version
> that Puts now seem to be slower.
> Specifically, comparing HBase 1.2.0-cdh5.9.0 to 1.2.0-cdh5.8.0 using YCSB,
> it looks like mean read latency is improved in cdh5.9 but mean write
> latency is worse, and the net effect in a 50% read/50% write workload is a
> performance degradation. FWIW, we've been testing with
> `hbase.ipc.server.callqueue.type` = "deadline" on both versions.
>
> Has anyone else using CDH 5.9.0 HBase run into this, or know what might be
> the cause?
>
> ~Graham
>