You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Upendra Yadav <up...@gmail.com> on 2014/02/25 19:00:52 UTC
Need some information over WAL
I have also doubt over WAL(write ahead log).
In hdfs we can write a new file or we can append to old file.
Is that correct :
1. WAL only logs operations not its data... just like disk journaling(ext4)
2. In case of Region Server failure... WAL replay will depends on Client to
get each operation's data that yet not committed/flushed to hdfs.
3. Hbase uses append operation in hdfs to store/log WAL.
Thanks...
Re: Need some information over WAL
Posted by Ted Yu <yu...@gmail.com>.
Minor addition:
bq. WAL is a single and common file for all regions on a region server
hbase:meta has its own WAL file.
On Tue, Feb 25, 2014 at 10:22 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:
> yes WAL is a single and common file for all regions on a region server.
> Yes to 1.
> Re 2: HBase will roll over WAL files and eventually delete them when they
> are no longer needed.
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Upendra Yadav <up...@gmail.com>
> To: user@hbase.apache.org; Dhaval Shah <pr...@yahoo.co.in>
> Sent: Tuesday, 25 February 2014 1:18 PM
> Subject: Re: Need some information over WAL
>
>
> Thanks Dhawal....
>
> Oh... whatever i assumed with reading documents (partially).... that was
> wrong...
>
> With ur answer... i have another questions...
> WAL is a single and common file for all region of a Region server.
>
> Is this correct:
> 1. So all operation(including data) will go to WAL... operation by
> operation....
>
> 2. Then on HDFS, HBase will have to perform delete operation on some set of
> WAL files always...
>
>
>
>
> On Tue, Feb 25, 2014 at 11:35 PM, Dhaval Shah
> <pr...@yahoo.co.in>wrote:
>
> > Inline
> >
> > Regards,
> > Dhaval
> >
> >
> > ________________________________
> > From: Upendra Yadav <up...@gmail.com>
> > To: user@hbase.apache.org
> > Sent: Tuesday, 25 February 2014 1:00 PM
> > Subject: Need some information over WAL
> >
> >
> > I have also doubt over WAL(write ahead log).
> >
> > In hdfs we can write a new file or we can append to old file.
> >
> > Is that correct :
> > 1. WAL only logs operations not its data... just like disk
> journaling(ext4)
> > - No WAL is a log of all new data, not just the operations
> >
> > 2. In case of Region Server failure... WAL replay will depends on Client
> to
> > get each operation's data that yet not committed/flushed to hdfs.
> > - No client will not generally know of a region server failure. It gets
> > the data from the WAL and replays it. The client may not even exist when
> a
> > region server crashes
> >
> > 3. Hbase uses append operation in hdfs to store/log WAL.
> > - Yes
> >
> > Thanks...
>
Re: Need some information over WAL
Posted by Upendra Yadav <up...@gmail.com>.
Thanks.... :)
On Tue, Feb 25, 2014 at 11:52 PM, Dhaval Shah
<pr...@yahoo.co.in>wrote:
> yes WAL is a single and common file for all regions on a region server.
> Yes to 1.
> Re 2: HBase will roll over WAL files and eventually delete them when they
> are no longer needed.
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Upendra Yadav <up...@gmail.com>
> To: user@hbase.apache.org; Dhaval Shah <pr...@yahoo.co.in>
> Sent: Tuesday, 25 February 2014 1:18 PM
> Subject: Re: Need some information over WAL
>
>
> Thanks Dhawal....
>
> Oh... whatever i assumed with reading documents (partially).... that was
> wrong...
>
> With ur answer... i have another questions...
> WAL is a single and common file for all region of a Region server.
>
> Is this correct:
> 1. So all operation(including data) will go to WAL... operation by
> operation....
>
> 2. Then on HDFS, HBase will have to perform delete operation on some set of
> WAL files always...
>
>
>
>
> On Tue, Feb 25, 2014 at 11:35 PM, Dhaval Shah
> <pr...@yahoo.co.in>wrote:
>
> > Inline
> >
> > Regards,
> > Dhaval
> >
> >
> > ________________________________
> > From: Upendra Yadav <up...@gmail.com>
> > To: user@hbase.apache.org
> > Sent: Tuesday, 25 February 2014 1:00 PM
> > Subject: Need some information over WAL
> >
> >
> > I have also doubt over WAL(write ahead log).
> >
> > In hdfs we can write a new file or we can append to old file.
> >
> > Is that correct :
> > 1. WAL only logs operations not its data... just like disk
> journaling(ext4)
> > - No WAL is a log of all new data, not just the operations
> >
> > 2. In case of Region Server failure... WAL replay will depends on Client
> to
> > get each operation's data that yet not committed/flushed to hdfs.
> > - No client will not generally know of a region server failure. It gets
> > the data from the WAL and replays it. The client may not even exist when
> a
> > region server crashes
> >
> > 3. Hbase uses append operation in hdfs to store/log WAL.
> > - Yes
> >
> > Thanks...
>
Re: Need some information over WAL
Posted by Dhaval Shah <pr...@yahoo.co.in>.
yes WAL is a single and common file for all regions on a region server.
Yes to 1.
Re 2: HBase will roll over WAL files and eventually delete them when they are no longer needed.
Regards,
Dhaval
________________________________
From: Upendra Yadav <up...@gmail.com>
To: user@hbase.apache.org; Dhaval Shah <pr...@yahoo.co.in>
Sent: Tuesday, 25 February 2014 1:18 PM
Subject: Re: Need some information over WAL
Thanks Dhawal....
Oh... whatever i assumed with reading documents (partially).... that was
wrong...
With ur answer... i have another questions...
WAL is a single and common file for all region of a Region server.
Is this correct:
1. So all operation(including data) will go to WAL... operation by
operation....
2. Then on HDFS, HBase will have to perform delete operation on some set of
WAL files always...
On Tue, Feb 25, 2014 at 11:35 PM, Dhaval Shah
<pr...@yahoo.co.in>wrote:
> Inline
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Upendra Yadav <up...@gmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, 25 February 2014 1:00 PM
> Subject: Need some information over WAL
>
>
> I have also doubt over WAL(write ahead log).
>
> In hdfs we can write a new file or we can append to old file.
>
> Is that correct :
> 1. WAL only logs operations not its data... just like disk journaling(ext4)
> - No WAL is a log of all new data, not just the operations
>
> 2. In case of Region Server failure... WAL replay will depends on Client to
> get each operation's data that yet not committed/flushed to hdfs.
> - No client will not generally know of a region server failure. It gets
> the data from the WAL and replays it. The client may not even exist when a
> region server crashes
>
> 3. Hbase uses append operation in hdfs to store/log WAL.
> - Yes
>
> Thanks...
Re: Need some information over WAL
Posted by Upendra Yadav <up...@gmail.com>.
Thanks Dhawal....
Oh... whatever i assumed with reading documents (partially).... that was
wrong...
With ur answer... i have another questions...
WAL is a single and common file for all region of a Region server.
Is this correct:
1. So all operation(including data) will go to WAL... operation by
operation....
2. Then on HDFS, HBase will have to perform delete operation on some set of
WAL files always...
On Tue, Feb 25, 2014 at 11:35 PM, Dhaval Shah
<pr...@yahoo.co.in>wrote:
> Inline
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Upendra Yadav <up...@gmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, 25 February 2014 1:00 PM
> Subject: Need some information over WAL
>
>
> I have also doubt over WAL(write ahead log).
>
> In hdfs we can write a new file or we can append to old file.
>
> Is that correct :
> 1. WAL only logs operations not its data... just like disk journaling(ext4)
> - No WAL is a log of all new data, not just the operations
>
> 2. In case of Region Server failure... WAL replay will depends on Client to
> get each operation's data that yet not committed/flushed to hdfs.
> - No client will not generally know of a region server failure. It gets
> the data from the WAL and replays it. The client may not even exist when a
> region server crashes
>
> 3. Hbase uses append operation in hdfs to store/log WAL.
> - Yes
>
> Thanks...
Re: Need some information over WAL
Posted by Upendra Yadav <up...@gmail.com>.
Thanks @Ted Yu...
On Tue, Feb 25, 2014 at 11:41 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. Hbase uses append operation in hdfs to store/log WAL.
>
> Following append operation, hdfs sync operation would be called for
> persistence.
>
> FYI
>
>
> On Tue, Feb 25, 2014 at 10:05 AM, Dhaval Shah
> <pr...@yahoo.co.in>wrote:
>
> > Inline
> >
> > Regards,
> > Dhaval
> >
> >
> > ________________________________
> > From: Upendra Yadav <up...@gmail.com>
> > To: user@hbase.apache.org
> > Sent: Tuesday, 25 February 2014 1:00 PM
> > Subject: Need some information over WAL
> >
> >
> > I have also doubt over WAL(write ahead log).
> >
> > In hdfs we can write a new file or we can append to old file.
> >
> > Is that correct :
> > 1. WAL only logs operations not its data... just like disk
> journaling(ext4)
> > - No WAL is a log of all new data, not just the operations
> >
> > 2. In case of Region Server failure... WAL replay will depends on Client
> to
> > get each operation's data that yet not committed/flushed to hdfs.
> > - No client will not generally know of a region server failure. It gets
> > the data from the WAL and replays it. The client may not even exist when
> a
> > region server crashes
> >
> > 3. Hbase uses append operation in hdfs to store/log WAL.
> > - Yes
> >
> > Thanks...
>
Re: Need some information over WAL
Posted by Ted Yu <yu...@gmail.com>.
bq. Hbase uses append operation in hdfs to store/log WAL.
Following append operation, hdfs sync operation would be called for
persistence.
FYI
On Tue, Feb 25, 2014 at 10:05 AM, Dhaval Shah
<pr...@yahoo.co.in>wrote:
> Inline
>
> Regards,
> Dhaval
>
>
> ________________________________
> From: Upendra Yadav <up...@gmail.com>
> To: user@hbase.apache.org
> Sent: Tuesday, 25 February 2014 1:00 PM
> Subject: Need some information over WAL
>
>
> I have also doubt over WAL(write ahead log).
>
> In hdfs we can write a new file or we can append to old file.
>
> Is that correct :
> 1. WAL only logs operations not its data... just like disk journaling(ext4)
> - No WAL is a log of all new data, not just the operations
>
> 2. In case of Region Server failure... WAL replay will depends on Client to
> get each operation's data that yet not committed/flushed to hdfs.
> - No client will not generally know of a region server failure. It gets
> the data from the WAL and replays it. The client may not even exist when a
> region server crashes
>
> 3. Hbase uses append operation in hdfs to store/log WAL.
> - Yes
>
> Thanks...
Re: Need some information over WAL
Posted by Dhaval Shah <pr...@yahoo.co.in>.
Inline
Regards,
Dhaval
________________________________
From: Upendra Yadav <up...@gmail.com>
To: user@hbase.apache.org
Sent: Tuesday, 25 February 2014 1:00 PM
Subject: Need some information over WAL
I have also doubt over WAL(write ahead log).
In hdfs we can write a new file or we can append to old file.
Is that correct :
1. WAL only logs operations not its data... just like disk journaling(ext4)
- No WAL is a log of all new data, not just the operations
2. In case of Region Server failure... WAL replay will depends on Client to
get each operation's data that yet not committed/flushed to hdfs.
- No client will not generally know of a region server failure. It gets the data from the WAL and replays it. The client may not even exist when a region server crashes
3. Hbase uses append operation in hdfs to store/log WAL.
- Yes
Thanks...