You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by Azuryy Yu <az...@gmail.com> on 2013/06/27 03:12:28 UTC

Question for edit log retain

Hi Dear all,

I have some confusion for edit log retain,

NNStorageRetentionManager.java:

1)
purgeCheckpointsOlderThan()
  What's mean of check point here?

2)purgeOldStorage()
  I cannot understand the calculation of minimum txid, I think I can
understand it if I know these keys indicates.
DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_KEY
DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY
DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_KEY

Can any developer give me a short detail explanation? Thanks much.

Re: Question for edit log retain

Posted by Azuryy Yu <az...@gmail.com>.

That's detailed. Thanks ATM.


On Tue, Jul 2, 2013 at 7:02 AM, Aaron T. Myers <at...@cloudera.com> wrote:

> Hi Azuryy,
>
> On Wed, Jun 26, 2013 at 6:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>
> > Hi Dear all,
> >
> > I have some confusion for edit log retain,
> >
> > NNStorageRetentionManager.java:
> >
> > 1)
> > purgeCheckpointsOlderThan()
> >   What's mean of check point here?
> >
>
> Here, "checkpoint" is referring to the fsimage files on the NN.
>
>
> >
> > 2)purgeOldStorage()
> >   I cannot understand the calculation of minimum txid, I think I can
> > understand it if I know these keys indicates.
> > DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_KEY
> >
>
> This is the number of old fsimage files to retain on the NN during purging.
>
>
> > DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY
> >
>
> This is the number of extra transactions to retain during purging. Here,
> "extra" is referring to transactions beyond what is strictly required to
> play back all FS changes since the last checkpoint (fsimage.)
>
>
> > DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_KEY
> >
>
> This is the maximum number of extra edit _segments_ to retain during
> purging. Segments refer to finalized edit log files, with a start and end
> transaction ID. A single edit file is a "segment."
>
>
> >
> > Can any developer give me a short detail explanation? Thanks much.
> >
>

Re: Question for edit log retain

Posted by "Aaron T. Myers" <at...@cloudera.com>.

Hi Azuryy,

On Wed, Jun 26, 2013 at 6:12 PM, Azuryy Yu <az...@gmail.com> wrote:

> Hi Dear all,
>
> I have some confusion for edit log retain,
>
> NNStorageRetentionManager.java:
>
> 1)
> purgeCheckpointsOlderThan()
>   What's mean of check point here?
>

Here, "checkpoint" is referring to the fsimage files on the NN.

>
> 2)purgeOldStorage()
>   I cannot understand the calculation of minimum txid, I think I can
> understand it if I know these keys indicates.
> DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_KEY
>

This is the number of old fsimage files to retain on the NN during purging.

> DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY
>

This is the number of extra transactions to retain during purging. Here,
"extra" is referring to transactions beyond what is strictly required to
play back all FS changes since the last checkpoint (fsimage.)

> DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_KEY
>

This is the maximum number of extra edit _segments_ to retain during
purging. Segments refer to finalized edit log files, with a start and end
transaction ID. A single edit file is a "segment."

>
> Can any developer give me a short detail explanation? Thanks much.
>