You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Lars George <la...@gmail.com> on 2011/07/05 08:39:06 UTC
.oldlogs in region dir
Hi,
Could someone confirm that we dropped the replay of the .oldlogs in the
region directory because it is not needed anymore? Before the master rewrite
we were, just as now, creating the region and for a brief period it had its
own log directory, the .oldlogs under the region directory. The
HRegion.initialize() would replay it in case an edit to the region arrived
in between. It would then also remove it:
public void initialize(Path initialFiles, final Progressable reporter)
throws IOException {
Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
...
// Play log if one. Delete when done.
doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId, reporter);
if (fs.exists(oldLogFile)) {
if (LOG.isDebugEnabled()) {
LOG.debug("Deleting old log file: " + oldLogFile);
}
fs.delete(oldLogFile, false);
}
...
Now we still create that directory, but the above code has been dropped.
Only the recovered.edits is replayed instead while the .oldlogs now hang
around. I am not even sure when they may contain an entry, but if they don't
then we can either do not use "<regiondir>/.oldlogs", but point it to the
new "<regiondir>/.tmp" - or we can add a unconditional delete in the
initialize as before.
Create a JIRA for this?
Lars
Re: .oldlogs in region dir
Posted by Stack <st...@duboce.net>.
It looks like left-over crud Lars. Thanks for noting it. I made
HBASE-4068 to clean it up.
St.Ack
On Tue, Jul 5, 2011 at 7:41 AM, Lars George <la...@gmail.com> wrote:
> Hi Dhruba,
>
> Yes, but that is the one on the root dir level. I am referring to the one on
> the region level.
>
> Cheers,
> Lars
>
> On Tue, Jul 5, 2011 at 6:25 AM, Dhruba Borthakur <dh...@gmail.com> wrote:
>
>> Doesn't the wide-area HBase replication use the .oldlogs to keep the slave
>> hBase cluster in sync?
>>
>> thanks
>> dhruba
>>
>> On Tue, Jul 5, 2011 at 7:27 AM, Lars George <la...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > Ah, I see Ted has that also questioned in HBASE-4010. Good.
>> >
>> > And I was slightly wrong below, as there is a .logs and .oldlogs for the
>> > newly created regions. I agree that when there is no chance for the
>> region
>> > to take edits before it gets the proper log, then we should drop this.
>> Those
>> > dangling dirs do not harm, but also do not help either.
>> >
>> > Lars
>> >
>> > On Jul 5, 2011, at 8:39 AM, Lars George wrote:
>> >
>> > > Hi,
>> > >
>> > > Could someone confirm that we dropped the replay of the .oldlogs in the
>> > region directory because it is not needed anymore? Before the master
>> rewrite
>> > we were, just as now, creating the region and for a brief period it had
>> its
>> > own log directory, the .oldlogs under the region directory. The
>> > HRegion.initialize() would replay it in case an edit to the region
>> arrived
>> > in between. It would then also remove it:
>> > >
>> > > public void initialize(Path initialFiles, final Progressable
>> reporter)
>> > > throws IOException {
>> > > Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
>> > > ...
>> > > // Play log if one. Delete when done.
>> > > doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
>> > reporter);
>> > > if (fs.exists(oldLogFile)) {
>> > > if (LOG.isDebugEnabled()) {
>> > > LOG.debug("Deleting old log file: " + oldLogFile);
>> > > }
>> > > fs.delete(oldLogFile, false);
>> > > }
>> > > ...
>> > >
>> > > Now we still create that directory, but the above code has been
>> dropped.
>> > Only the recovered.edits is replayed instead while the .oldlogs now hang
>> > around. I am not even sure when they may contain an entry, but if they
>> don't
>> > then we can either do not use "<regiondir>/.oldlogs", but point it to the
>> > new "<regiondir>/.tmp" - or we can add a unconditional delete in the
>> > initialize as before.
>> > >
>> > > Create a JIRA for this?
>> > >
>> > > Lars
>> >
>> >
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>
Re: .oldlogs in region dir
Posted by Dhruba Borthakur <dh...@gmail.com>.
Ok, got it, thanks Lars.
-dhruba
On Tue, Jul 5, 2011 at 2:41 PM, Lars George <la...@gmail.com> wrote:
> Hi Dhruba,
>
> Yes, but that is the one on the root dir level. I am referring to the one
> on
> the region level.
>
> Cheers,
> Lars
>
> On Tue, Jul 5, 2011 at 6:25 AM, Dhruba Borthakur <dh...@gmail.com> wrote:
>
> > Doesn't the wide-area HBase replication use the .oldlogs to keep the
> slave
> > hBase cluster in sync?
> >
> > thanks
> > dhruba
> >
> > On Tue, Jul 5, 2011 at 7:27 AM, Lars George <la...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Ah, I see Ted has that also questioned in HBASE-4010. Good.
> > >
> > > And I was slightly wrong below, as there is a .logs and .oldlogs for
> the
> > > newly created regions. I agree that when there is no chance for the
> > region
> > > to take edits before it gets the proper log, then we should drop this.
> > Those
> > > dangling dirs do not harm, but also do not help either.
> > >
> > > Lars
> > >
> > > On Jul 5, 2011, at 8:39 AM, Lars George wrote:
> > >
> > > > Hi,
> > > >
> > > > Could someone confirm that we dropped the replay of the .oldlogs in
> the
> > > region directory because it is not needed anymore? Before the master
> > rewrite
> > > we were, just as now, creating the region and for a brief period it had
> > its
> > > own log directory, the .oldlogs under the region directory. The
> > > HRegion.initialize() would replay it in case an edit to the region
> > arrived
> > > in between. It would then also remove it:
> > > >
> > > > public void initialize(Path initialFiles, final Progressable
> > reporter)
> > > > throws IOException {
> > > > Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> > > > ...
> > > > // Play log if one. Delete when done.
> > > > doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
> > > reporter);
> > > > if (fs.exists(oldLogFile)) {
> > > > if (LOG.isDebugEnabled()) {
> > > > LOG.debug("Deleting old log file: " + oldLogFile);
> > > > }
> > > > fs.delete(oldLogFile, false);
> > > > }
> > > > ...
> > > >
> > > > Now we still create that directory, but the above code has been
> > dropped.
> > > Only the recovered.edits is replayed instead while the .oldlogs now
> hang
> > > around. I am not even sure when they may contain an entry, but if they
> > don't
> > > then we can either do not use "<regiondir>/.oldlogs", but point it to
> the
> > > new "<regiondir>/.tmp" - or we can add a unconditional delete in the
> > > initialize as before.
> > > >
> > > > Create a JIRA for this?
> > > >
> > > > Lars
> > >
> > >
> >
> >
> > --
> > Connect to me at http://www.facebook.com/dhruba
> >
>
--
Connect to me at http://www.facebook.com/dhruba
Re: .oldlogs in region dir
Posted by Lars George <la...@gmail.com>.
Hi Dhruba,
Yes, but that is the one on the root dir level. I am referring to the one on
the region level.
Cheers,
Lars
On Tue, Jul 5, 2011 at 6:25 AM, Dhruba Borthakur <dh...@gmail.com> wrote:
> Doesn't the wide-area HBase replication use the .oldlogs to keep the slave
> hBase cluster in sync?
>
> thanks
> dhruba
>
> On Tue, Jul 5, 2011 at 7:27 AM, Lars George <la...@gmail.com> wrote:
>
> > Hi,
> >
> > Ah, I see Ted has that also questioned in HBASE-4010. Good.
> >
> > And I was slightly wrong below, as there is a .logs and .oldlogs for the
> > newly created regions. I agree that when there is no chance for the
> region
> > to take edits before it gets the proper log, then we should drop this.
> Those
> > dangling dirs do not harm, but also do not help either.
> >
> > Lars
> >
> > On Jul 5, 2011, at 8:39 AM, Lars George wrote:
> >
> > > Hi,
> > >
> > > Could someone confirm that we dropped the replay of the .oldlogs in the
> > region directory because it is not needed anymore? Before the master
> rewrite
> > we were, just as now, creating the region and for a brief period it had
> its
> > own log directory, the .oldlogs under the region directory. The
> > HRegion.initialize() would replay it in case an edit to the region
> arrived
> > in between. It would then also remove it:
> > >
> > > public void initialize(Path initialFiles, final Progressable
> reporter)
> > > throws IOException {
> > > Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> > > ...
> > > // Play log if one. Delete when done.
> > > doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
> > reporter);
> > > if (fs.exists(oldLogFile)) {
> > > if (LOG.isDebugEnabled()) {
> > > LOG.debug("Deleting old log file: " + oldLogFile);
> > > }
> > > fs.delete(oldLogFile, false);
> > > }
> > > ...
> > >
> > > Now we still create that directory, but the above code has been
> dropped.
> > Only the recovered.edits is replayed instead while the .oldlogs now hang
> > around. I am not even sure when they may contain an entry, but if they
> don't
> > then we can either do not use "<regiondir>/.oldlogs", but point it to the
> > new "<regiondir>/.tmp" - or we can add a unconditional delete in the
> > initialize as before.
> > >
> > > Create a JIRA for this?
> > >
> > > Lars
> >
> >
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Re: .oldlogs in region dir
Posted by Dhruba Borthakur <dh...@gmail.com>.
Doesn't the wide-area HBase replication use the .oldlogs to keep the slave
hBase cluster in sync?
thanks
dhruba
On Tue, Jul 5, 2011 at 7:27 AM, Lars George <la...@gmail.com> wrote:
> Hi,
>
> Ah, I see Ted has that also questioned in HBASE-4010. Good.
>
> And I was slightly wrong below, as there is a .logs and .oldlogs for the
> newly created regions. I agree that when there is no chance for the region
> to take edits before it gets the proper log, then we should drop this. Those
> dangling dirs do not harm, but also do not help either.
>
> Lars
>
> On Jul 5, 2011, at 8:39 AM, Lars George wrote:
>
> > Hi,
> >
> > Could someone confirm that we dropped the replay of the .oldlogs in the
> region directory because it is not needed anymore? Before the master rewrite
> we were, just as now, creating the region and for a brief period it had its
> own log directory, the .oldlogs under the region directory. The
> HRegion.initialize() would replay it in case an edit to the region arrived
> in between. It would then also remove it:
> >
> > public void initialize(Path initialFiles, final Progressable reporter)
> > throws IOException {
> > Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> > ...
> > // Play log if one. Delete when done.
> > doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId,
> reporter);
> > if (fs.exists(oldLogFile)) {
> > if (LOG.isDebugEnabled()) {
> > LOG.debug("Deleting old log file: " + oldLogFile);
> > }
> > fs.delete(oldLogFile, false);
> > }
> > ...
> >
> > Now we still create that directory, but the above code has been dropped.
> Only the recovered.edits is replayed instead while the .oldlogs now hang
> around. I am not even sure when they may contain an entry, but if they don't
> then we can either do not use "<regiondir>/.oldlogs", but point it to the
> new "<regiondir>/.tmp" - or we can add a unconditional delete in the
> initialize as before.
> >
> > Create a JIRA for this?
> >
> > Lars
>
>
--
Connect to me at http://www.facebook.com/dhruba
Re: .oldlogs in region dir
Posted by Lars George <la...@gmail.com>.
Hi,
Ah, I see Ted has that also questioned in HBASE-4010. Good.
And I was slightly wrong below, as there is a .logs and .oldlogs for the newly created regions. I agree that when there is no chance for the region to take edits before it gets the proper log, then we should drop this. Those dangling dirs do not harm, but also do not help either.
Lars
On Jul 5, 2011, at 8:39 AM, Lars George wrote:
> Hi,
>
> Could someone confirm that we dropped the replay of the .oldlogs in the region directory because it is not needed anymore? Before the master rewrite we were, just as now, creating the region and for a brief period it had its own log directory, the .oldlogs under the region directory. The HRegion.initialize() would replay it in case an edit to the region arrived in between. It would then also remove it:
>
> public void initialize(Path initialFiles, final Progressable reporter)
> throws IOException {
> Path oldLogFile = new Path(regiondir, HREGION_OLDLOGFILE_NAME);
> ...
> // Play log if one. Delete when done.
> doReconstructionLog(oldLogFile, minSeqIdToRecover, maxSeqId, reporter);
> if (fs.exists(oldLogFile)) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Deleting old log file: " + oldLogFile);
> }
> fs.delete(oldLogFile, false);
> }
> ...
>
> Now we still create that directory, but the above code has been dropped. Only the recovered.edits is replayed instead while the .oldlogs now hang around. I am not even sure when they may contain an entry, but if they don't then we can either do not use "<regiondir>/.oldlogs", but point it to the new "<regiondir>/.tmp" - or we can add a unconditional delete in the initialize as before.
>
> Create a JIRA for this?
>
> Lars