You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by lei liu <li...@gmail.com> on 2012/11/13 09:48:32 UTC
HA for hadoop-0.20.2
I want to implement HA function for hadoop-0.20.2.
When I learn the hadoop-2.0 code, I meet some question like this:
Thera are below code in FSEditLogLoader.loadEditRecords method.
if (op.hasTransactionId()) {
if (op.getTransactionId() > expectedTxId) {
MetaRecoveryContext.editLogLoaderPrompt("There appears " +
"to be a gap in the edit log. We expected txid " +
expectedTxId + ", but got txid " +
op.getTransactionId() + ".", recovery, "ignoring missing
" +
" transaction IDs");
} *else if (op.getTransactionId() < expectedTxId) {
MetaRecoveryContext.editLogLoaderPrompt("There appears " +
"to be an out-of-order edit in the edit log. We " +
"expected txid " + expectedTxId + ", but got txid " +
op.getTransactionId() + ".", recovery,
"skipping the out-of-order edit");
continue;*
}
}
The method use transaction id to guarantee same transaction log is not
applied to namespace more than once.
But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
log file. So I want to know if StandbyNN apply same transaction log to
namespace more than once, that will lead to the namespace of StandbyNN is
corrupt?
Please give me some advice，Thanks.
Best Regards
LiuLei
Re: HA for hadoop-0.20.2
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi, A question, is 2.x ready for production deployment? Thanks
On 13 Nov 2012, at 5:19 PM, Harsh J wrote:
> Hi,
>
> Why not just use the 2.x releases for HA-NNs? There is quite a wide
> delta between 0.20.x and 2.x, especially around the edit log areas
> after HDFS-1073.
>
> In any case, I think your question suits hdfs-dev@hadoop.apache.org
> more than the user lists, although I don't quite understand what
> you're attempting to do (or point).
>
> On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
>> I want to implement HA function for hadoop-0.20.2.
>>
>> When I learn the hadoop-2.0 code, I meet some question like this:
>>
>> Thera are below code in FSEditLogLoader.loadEditRecords method.
>>
>> if (op.hasTransactionId()) {
>> if (op.getTransactionId() > expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be a gap in the edit log. We expected txid " +
>> expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery, "ignoring missing "
>> +
>> " transaction IDs");
>> } else if (op.getTransactionId() < expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be an out-of-order edit in the edit log. We " +
>> "expected txid " + expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery,
>> "skipping the out-of-order edit");
>> continue;
>> }
>> }
>>
>> The method use transaction id to guarantee same transaction log is not
>> applied to namespace more than once.
>>
>> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
>> log file. So I want to know if StandbyNN apply same transaction log to
>> namespace more than once, that will lead to the namespace of StandbyNN is
>> corrupt?
>>
>> Please give me some advice，Thanks.
>>
>>
>>
>> Best Regards
>>
>> LiuLei
>
>
>
> --
> Harsh J
Re: HA for hadoop-0.20.2
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi, A question, is 2.x ready for production deployment? Thanks
On 13 Nov 2012, at 5:19 PM, Harsh J wrote:
> Hi,
>
> Why not just use the 2.x releases for HA-NNs? There is quite a wide
> delta between 0.20.x and 2.x, especially around the edit log areas
> after HDFS-1073.
>
> In any case, I think your question suits hdfs-dev@hadoop.apache.org
> more than the user lists, although I don't quite understand what
> you're attempting to do (or point).
>
> On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
>> I want to implement HA function for hadoop-0.20.2.
>>
>> When I learn the hadoop-2.0 code, I meet some question like this:
>>
>> Thera are below code in FSEditLogLoader.loadEditRecords method.
>>
>> if (op.hasTransactionId()) {
>> if (op.getTransactionId() > expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be a gap in the edit log. We expected txid " +
>> expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery, "ignoring missing "
>> +
>> " transaction IDs");
>> } else if (op.getTransactionId() < expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be an out-of-order edit in the edit log. We " +
>> "expected txid " + expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery,
>> "skipping the out-of-order edit");
>> continue;
>> }
>> }
>>
>> The method use transaction id to guarantee same transaction log is not
>> applied to namespace more than once.
>>
>> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
>> log file. So I want to know if StandbyNN apply same transaction log to
>> namespace more than once, that will lead to the namespace of StandbyNN is
>> corrupt?
>>
>> Please give me some advice，Thanks.
>>
>>
>>
>> Best Regards
>>
>> LiuLei
>
>
>
> --
> Harsh J
Re: HA for hadoop-0.20.2
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi, A question, is 2.x ready for production deployment? Thanks
On 13 Nov 2012, at 5:19 PM, Harsh J wrote:
> Hi,
>
> Why not just use the 2.x releases for HA-NNs? There is quite a wide
> delta between 0.20.x and 2.x, especially around the edit log areas
> after HDFS-1073.
>
> In any case, I think your question suits hdfs-dev@hadoop.apache.org
> more than the user lists, although I don't quite understand what
> you're attempting to do (or point).
>
> On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
>> I want to implement HA function for hadoop-0.20.2.
>>
>> When I learn the hadoop-2.0 code, I meet some question like this:
>>
>> Thera are below code in FSEditLogLoader.loadEditRecords method.
>>
>> if (op.hasTransactionId()) {
>> if (op.getTransactionId() > expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be a gap in the edit log. We expected txid " +
>> expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery, "ignoring missing "
>> +
>> " transaction IDs");
>> } else if (op.getTransactionId() < expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be an out-of-order edit in the edit log. We " +
>> "expected txid " + expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery,
>> "skipping the out-of-order edit");
>> continue;
>> }
>> }
>>
>> The method use transaction id to guarantee same transaction log is not
>> applied to namespace more than once.
>>
>> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
>> log file. So I want to know if StandbyNN apply same transaction log to
>> namespace more than once, that will lead to the namespace of StandbyNN is
>> corrupt?
>>
>> Please give me some advice，Thanks.
>>
>>
>>
>> Best Regards
>>
>> LiuLei
>
>
>
> --
> Harsh J
Re: HA for hadoop-0.20.2
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi, A question, is 2.x ready for production deployment? Thanks
On 13 Nov 2012, at 5:19 PM, Harsh J wrote:
> Hi,
>
> Why not just use the 2.x releases for HA-NNs? There is quite a wide
> delta between 0.20.x and 2.x, especially around the edit log areas
> after HDFS-1073.
>
> In any case, I think your question suits hdfs-dev@hadoop.apache.org
> more than the user lists, although I don't quite understand what
> you're attempting to do (or point).
>
> On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
>> I want to implement HA function for hadoop-0.20.2.
>>
>> When I learn the hadoop-2.0 code, I meet some question like this:
>>
>> Thera are below code in FSEditLogLoader.loadEditRecords method.
>>
>> if (op.hasTransactionId()) {
>> if (op.getTransactionId() > expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be a gap in the edit log. We expected txid " +
>> expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery, "ignoring missing "
>> +
>> " transaction IDs");
>> } else if (op.getTransactionId() < expectedTxId) {
>> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
>> "to be an out-of-order edit in the edit log. We " +
>> "expected txid " + expectedTxId + ", but got txid " +
>> op.getTransactionId() + ".", recovery,
>> "skipping the out-of-order edit");
>> continue;
>> }
>> }
>>
>> The method use transaction id to guarantee same transaction log is not
>> applied to namespace more than once.
>>
>> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
>> log file. So I want to know if StandbyNN apply same transaction log to
>> namespace more than once, that will lead to the namespace of StandbyNN is
>> corrupt?
>>
>> Please give me some advice，Thanks.
>>
>>
>>
>> Best Regards
>>
>> LiuLei
>
>
>
> --
> Harsh J
Re: HA for hadoop-0.20.2
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Why not just use the 2.x releases for HA-NNs? There is quite a wide
delta between 0.20.x and 2.x, especially around the edit log areas
after HDFS-1073.
In any case, I think your question suits hdfs-dev@hadoop.apache.org
more than the user lists, although I don't quite understand what
you're attempting to do (or point).
On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
> I want to implement HA function for hadoop-0.20.2.
>
> When I learn the hadoop-2.0 code, I meet some question like this:
>
> Thera are below code in FSEditLogLoader.loadEditRecords method.
>
> if (op.hasTransactionId()) {
> if (op.getTransactionId() > expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be a gap in the edit log. We expected txid " +
> expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery, "ignoring missing "
> +
> " transaction IDs");
> } else if (op.getTransactionId() < expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be an out-of-order edit in the edit log. We " +
> "expected txid " + expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery,
> "skipping the out-of-order edit");
> continue;
> }
> }
>
> The method use transaction id to guarantee same transaction log is not
> applied to namespace more than once.
>
> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
> log file. So I want to know if StandbyNN apply same transaction log to
> namespace more than once, that will lead to the namespace of StandbyNN is
> corrupt?
>
> Please give me some advice，Thanks.
>
>
>
> Best Regards
>
> LiuLei
--
Harsh J
Re: HA for hadoop-0.20.2
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Why not just use the 2.x releases for HA-NNs? There is quite a wide
delta between 0.20.x and 2.x, especially around the edit log areas
after HDFS-1073.
In any case, I think your question suits hdfs-dev@hadoop.apache.org
more than the user lists, although I don't quite understand what
you're attempting to do (or point).
On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
> I want to implement HA function for hadoop-0.20.2.
>
> When I learn the hadoop-2.0 code, I meet some question like this:
>
> Thera are below code in FSEditLogLoader.loadEditRecords method.
>
> if (op.hasTransactionId()) {
> if (op.getTransactionId() > expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be a gap in the edit log. We expected txid " +
> expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery, "ignoring missing "
> +
> " transaction IDs");
> } else if (op.getTransactionId() < expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be an out-of-order edit in the edit log. We " +
> "expected txid " + expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery,
> "skipping the out-of-order edit");
> continue;
> }
> }
>
> The method use transaction id to guarantee same transaction log is not
> applied to namespace more than once.
>
> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
> log file. So I want to know if StandbyNN apply same transaction log to
> namespace more than once, that will lead to the namespace of StandbyNN is
> corrupt?
>
> Please give me some advice，Thanks.
>
>
>
> Best Regards
>
> LiuLei
--
Harsh J
Re: HA for hadoop-0.20.2
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Why not just use the 2.x releases for HA-NNs? There is quite a wide
delta between 0.20.x and 2.x, especially around the edit log areas
after HDFS-1073.
In any case, I think your question suits hdfs-dev@hadoop.apache.org
more than the user lists, although I don't quite understand what
you're attempting to do (or point).
On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
> I want to implement HA function for hadoop-0.20.2.
>
> When I learn the hadoop-2.0 code, I meet some question like this:
>
> Thera are below code in FSEditLogLoader.loadEditRecords method.
>
> if (op.hasTransactionId()) {
> if (op.getTransactionId() > expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be a gap in the edit log. We expected txid " +
> expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery, "ignoring missing "
> +
> " transaction IDs");
> } else if (op.getTransactionId() < expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be an out-of-order edit in the edit log. We " +
> "expected txid " + expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery,
> "skipping the out-of-order edit");
> continue;
> }
> }
>
> The method use transaction id to guarantee same transaction log is not
> applied to namespace more than once.
>
> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
> log file. So I want to know if StandbyNN apply same transaction log to
> namespace more than once, that will lead to the namespace of StandbyNN is
> corrupt?
>
> Please give me some advice，Thanks.
>
>
>
> Best Regards
>
> LiuLei
--
Harsh J
Re: HA for hadoop-0.20.2
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Why not just use the 2.x releases for HA-NNs? There is quite a wide
delta between 0.20.x and 2.x, especially around the edit log areas
after HDFS-1073.
In any case, I think your question suits hdfs-dev@hadoop.apache.org
more than the user lists, although I don't quite understand what
you're attempting to do (or point).
On Tue, Nov 13, 2012 at 2:18 PM, lei liu <li...@gmail.com> wrote:
> I want to implement HA function for hadoop-0.20.2.
>
> When I learn the hadoop-2.0 code, I meet some question like this:
>
> Thera are below code in FSEditLogLoader.loadEditRecords method.
>
> if (op.hasTransactionId()) {
> if (op.getTransactionId() > expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be a gap in the edit log. We expected txid " +
> expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery, "ignoring missing "
> +
> " transaction IDs");
> } else if (op.getTransactionId() < expectedTxId) {
> MetaRecoveryContext.editLogLoaderPrompt("There appears " +
> "to be an out-of-order edit in the edit log. We " +
> "expected txid " + expectedTxId + ", but got txid " +
> op.getTransactionId() + ".", recovery,
> "skipping the out-of-order edit");
> continue;
> }
> }
>
> The method use transaction id to guarantee same transaction log is not
> applied to namespace more than once.
>
> But in hadoop-0.20.2, FSEditLog don't store the transaction id into edits
> log file. So I want to know if StandbyNN apply same transaction log to
> namespace more than once, that will lead to the namespace of StandbyNN is
> corrupt?
>
> Please give me some advice，Thanks.
>
>
>
> Best Regards
>
> LiuLei
--
Harsh J