Client Side buffering vs WAL

[Michael Segel <mi...@hotmail.com>]

Hi,

Came across a problem that I need to walk through.

On the client side, when you instantiate an HTable object, you can specify HTable.setAutoFlush(true/false).  Setting the boolean value to true means that when you execute a put(), the write is not buffered on the client and will be written directly to HBase. This overrides the client side buffering that you can set in your configuration files.


While for many applications its ok for the app to buffer up its writes, however there's a set of apps where you don't want to do this. That is when your app writes a record to HBase, you want it exposed ASAP.

On the server side, you have the Write Ahead Log.

If I understand the WAL, it abstracts the actual process of writing to disk so that as far as your application is concerned, when you write to the WAL, its in HBase.

So, my question is how long does it take for a record in the WAL to be written to Disk?

Also if a record is in the WAL, if I did a get() will the record be found?

Its possible that in a m/r job that client side buffering could mean that it could take a relatively 'long' time to actually have a record written to HBase, where as once the record is written to the WAL, it should be consistent in the time it takes to be written to disk for access by other HBase apps.

Or what am I missing?

Thx

-Mike



 		 	   		  

RE: Client Side buffering vs WAL

[Michael Segel <mi...@hotmail.com>]

> From: tsunanet@gmail.com
> Date: Wed, 8 Sep 2010 10:08:52 -0700
> Subject: Re: Client Side buffering vs WAL
> To: user@hbase.apache.org
> 
> On Tue, Sep 7, 2010 at 7:44 AM, Michael Segel <mi...@hotmail.com> wrote:
> > So, my question is how long does it take for a record in the WAL to be written to Disk?
> 
> This happens before the RPC completes successfully.  If the RPC
> succeeds, you know the WAL was written to disk and your data is safe,
> even if there's a server failure.
> 

yes... talked about this last night on IRC. I wanted to confirm that this was the case. 

Buffering put()s on the client side may make sense for most batch applications but when you need to see it in the database, you want it to go in when you send a put() and not when you either exceed the buffer or flush the buffer.

I just wanted to make sure that the client connects, writes to HBase WAL and memstore (See JD, I was paying attention) it will be accessible by others. So the data isn't getting delayed in another buffer.

Thanks to all for confirming this.

-Mike
 		 	   		  

Re: Client Side buffering vs WAL

[tsuna <ts...@gmail.com>]

On Tue, Sep 7, 2010 at 7:44 AM, Michael Segel <mi...@hotmail.com> wrote:
> So, my question is how long does it take for a record in the WAL to be written to Disk?

This happens before the RPC completes successfully.  If the RPC
succeeds, you know the WAL was written to disk and your data is safe,
even if there's a server failure.

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

Re: Client Side buffering vs WAL

[Jean-Daniel Cryans <jd...@apache.org>]

I think Lars explains it best:
http://www.larsgeorge.com/2010/01/hbase-architecture-101-write-ahead-log.html

Short version: writing to the WAL is a backup solution if the region
server dies, because it's the MemStore that's being used for reads
(not the WAL). If you autoFlush, then everyone can read the data if
once your put() call returns without errors.

J-D

On Tue, Sep 7, 2010 at 7:44 AM, Michael Segel <mi...@hotmail.com> wrote:
>
> Hi,
>
> Came across a problem that I need to walk through.
>
> On the client side, when you instantiate an HTable object, you can specify HTable.setAutoFlush(true/false).  Setting the boolean value to true means that when you execute a put(), the write is not buffered on the client and will be written directly to HBase. This overrides the client side buffering that you can set in your configuration files.
>
>
> While for many applications its ok for the app to buffer up its writes, however there's a set of apps where you don't want to do this. That is when your app writes a record to HBase, you want it exposed ASAP.
>
> On the server side, you have the Write Ahead Log.
>
> If I understand the WAL, it abstracts the actual process of writing to disk so that as far as your application is concerned, when you write to the WAL, its in HBase.
>
> So, my question is how long does it take for a record in the WAL to be written to Disk?
>
> Also if a record is in the WAL, if I did a get() will the record be found?
>
> Its possible that in a m/r job that client side buffering could mean that it could take a relatively 'long' time to actually have a record written to HBase, where as once the record is written to the WAL, it should be consistent in the time it takes to be written to disk for access by other HBase apps.
>
> Or what am I missing?
>
> Thx
>
> -Mike
>
>
>
>