You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Todd Lipcon <to...@cloudera.com> on 2010/04/15 10:59:29 UTC
HDFS-895 hflush speedup
Any of you out there who are running the 0.20+200+etc hadoop with 0.20
branch hbase want to try this patch? I think with it on, we can practically
turn off group commit and get quite a bit speedup.
--
Todd Lipcon
Software Engineer, Cloudera
Re: HDFS-895 hflush speedup
Posted by Ryan Rawson <ry...@gmail.com>.
I'm running whats on here:
http://github.com/ryanobjc/hadoop-cloudera
Which is basically the patches you sent me previously.
I have a cluster I would be willing to test with...
-ryan
On Thu, Apr 15, 2010 at 11:54 AM, Todd Lipcon <to...@cloudera.com> wrote:
> On Thu, Apr 15, 2010 at 10:51 AM, Kannan Muthukkaruppan <Kannan@facebook.com
>> wrote:
>
>> Todd: Good stuff! Have you tried this out on a HBase test too?
>>
>>
> Not yet - I basically had a burst of inspiration last night around 11pm,
> coded til I couldn't stay awake, and threw up a patch :) I hope to give it a
> try today (and also update patch to trunk so I can get some reviews there)
>
>
>> > I think with it on, we can practically turn off group commit and get
>> quite a bit speedup.
>>
>> Will batching now happen underneath the DFSClient automatically now that
>> syncs/appends are not serialized.
>>
>>
> It's not quite batching, but essentially I think we get the same benefit.
> While one thread is waiting for *its* packet/write to be synced, another
> thread can still append to the stream and call sync for its own data.
>
> -Todd
>
> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Thursday, April 15, 2010 1:59 AM
>> To: hbase-dev@hadoop.apache.org
>> Subject: HDFS-895 hflush speedup
>>
>> Any of you out there who are running the 0.20+200+etc hadoop with 0.20
>> branch hbase want to try this patch? I think with it on, we can practically
>> turn off group commit and get quite a bit speedup.
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: HDFS-895 hflush speedup
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Apr 15, 2010 at 10:51 AM, Kannan Muthukkaruppan <Kannan@facebook.com
> wrote:
> Todd: Good stuff! Have you tried this out on a HBase test too?
>
>
Not yet - I basically had a burst of inspiration last night around 11pm,
coded til I couldn't stay awake, and threw up a patch :) I hope to give it a
try today (and also update patch to trunk so I can get some reviews there)
> > I think with it on, we can practically turn off group commit and get
> quite a bit speedup.
>
> Will batching now happen underneath the DFSClient automatically now that
> syncs/appends are not serialized.
>
>
It's not quite batching, but essentially I think we get the same benefit.
While one thread is waiting for *its* packet/write to be synced, another
thread can still append to the stream and call sync for its own data.
-Todd
-----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, April 15, 2010 1:59 AM
> To: hbase-dev@hadoop.apache.org
> Subject: HDFS-895 hflush speedup
>
> Any of you out there who are running the 0.20+200+etc hadoop with 0.20
> branch hbase want to try this patch? I think with it on, we can practically
> turn off group commit and get quite a bit speedup.
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Todd Lipcon
Software Engineer, Cloudera
RE: HDFS-895 hflush speedup
Posted by Andrew Purtell <ap...@apache.org>.
I'll try this out when I get a chance. Probably not until the weekend. This looks great.
- Andy
> From: Kannan Muthukkaruppan <Ka...@facebook.com>
> Subject: RE: HDFS-895 hflush speedup
> To: "hbase-dev@hadoop.apache.org" <hb...@hadoop.apache.org>
> Date: Thursday, April 15, 2010, 10:51 AM
> Todd: Good stuff! Have you tried this
> out on a HBase test too?
>
> > I think with it on, we can practically turn off group
> > commit and get quite a bit speedup.
>
> Will batching now happen underneath the DFSClient
> automatically now that syncs/appends are not serialized.
>
> regards,
> Kannan
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, April 15, 2010 1:59 AM
> To: hbase-dev@hadoop.apache.org
> Subject: HDFS-895 hflush speedup
>
> Any of you out there who are running the 0.20+200+etc
> hadoop with 0.20 branch hbase want to try this patch?
> I think with it on, we can practically
> turn off group commit and get quite a bit speedup.
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
RE: HDFS-895 hflush speedup
Posted by Kannan Muthukkaruppan <Ka...@facebook.com>.
Todd: Good stuff! Have you tried this out on a HBase test too?
> I think with it on, we can practically turn off group commit and get quite a bit speedup.
Will batching now happen underneath the DFSClient automatically now that syncs/appends are not serialized.
regards,
Kannan
-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Thursday, April 15, 2010 1:59 AM
To: hbase-dev@hadoop.apache.org
Subject: HDFS-895 hflush speedup
Any of you out there who are running the 0.20+200+etc hadoop with 0.20
branch hbase want to try this patch? I think with it on, we can practically
turn off group commit and get quite a bit speedup.
--
Todd Lipcon
Software Engineer, Cloudera