You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ivan Kelly (Created) (JIRA)" <ji...@apache.org> on 2012/03/07 16:42:59 UTC
[jira] [Created] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
HA: Bring BookKeeperJournalManager up to date with HA changes
-------------------------------------------------------------
Key: HDFS-3058
URL: https://issues.apache.org/jira/browse/HDFS-3058
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 0.24.0
There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283231#comment-13283231 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Author: umamahesh
Date: Fri May 25 09:09:39 2012
New Revision: 1342538
URL: http://svn.apache.org/viewvc?rev=1342538&view=rev
Log:
svn merge -c -1342534 for HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. Contributed by Ivan Kelly.
Added:
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
- copied unchanged from r1342534, hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
- copied unchanged from r1342534, hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
- copied unchanged from r1342534, hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
Modified:
hadoop/common/branches/branch-2/hadoop-hdfs-project/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/native/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs/ (props changed)
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
Propchange: hadoop/common/branches/branch-2/hadoop-hdfs-project/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-hdfs-project:r1342534
Propchange: hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs:r1342534
Modified: hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt?rev=1342538&r1=1342537&r2=1342538&view=diff
==============================================================================
--- hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt (original)
+++ hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fri May 25 09:09:39 2012
@@ -113,6 +113,9 @@ Release 2.0.1-alpha - UNRELEASED
HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full
principal name. (tucu)
+ HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes.
+ (Ivan Kelly via umamahesh)
+
Release 2.0.0-alpha - UNRELEASED
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224532#comment-13224532 ]
jiraposter@reviews.apache.org commented on HDFS-3058:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4230/
-----------------------------------------------------------
Review request for hadoop-hdfs.
Summary
-------
There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
This addresses bug HDFS-3058.
http://issues.apache.org/jira/browse/HDFS-3058
Diffs
-----
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml 1fba846
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java 636471a
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java 047efd5
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java 5937fa8
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java 6557b96
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java 7c630d7
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java 8888bec
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java 5440c38
Diff: https://reviews.apache.org/r/4230/diff
Testing
-------
Thanks,
Ivan
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13273830#comment-13273830 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Hi Ivan, Thanks a lot for the patch. Few comments.
1) Patch does not apply cleanly. many Hunk failure while applying patch.
{noformat}
Hunk #13 FAILED at 264.
Hunk #14 FAILED at 281.
Hunk #15 FAILED at 293.
Hunk #16 FAILED at 309.
Hunk #17 FAILED at 332.
Hunk #18 FAILED at 345.
Hunk #19 FAILED at 362.
Hunk #20 FAILED at 372.
Hunk #21 FAILED at 395.
{noformat}
Could you please regenerate on the correct trunk code base?
2) TestBookKeeperHACheckpoints.java:
{code}
conf.set(DFSConfigKeys.DFS_NAMENODE_SHARED_EDITS_DIR_KEY,
+ BKJMUtil.createJournalURI("/checkpointing" + journalCount++)
+ .toString());
+ BKJMUtil.addJournalManagerDefinition(conf);
{code}
I can see setUpCluster code almost duplicates with TestStandbyCheckpoints. How about extracting configuration setup alone to protected method in TestStandbyCheckpoints and override in TestBookKeeperHACheckpoints.
Set the above configurations and give the call to super.
3) {code}
+
+ @AfterClass
+ public static void shutdownBK() throws Exception {
+ bkutil.teardown();
+ }
{code}
Could you please add null condition for bkutil? also correct the bkutil->bkUtil
Not completed nmy review, will continue review...
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283157#comment-13283157 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Actually It has conflicts with HDFS-2982. I just resolved them and updated the patch again which is same as Ivan patch + resolving conflicts.
Also verfied basic flows with BK clusters.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-3058:
--------------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HDFS-3399
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282583#comment-13282583 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Today, Ivan , Rekesh and Me had a chatt about this comment. Decided to handle in follow up JIRA because we have identified some more scenarios to handle. Since the JIRA is up and down most of the time today, we could not add the comments here.
So, I am +1 for committing this and opening a separate JIRA for above comment.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283233#comment-13283233 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
I have committed HDFS-3058.patch now for both trunk and branch-2.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283407#comment-13283407 ]
Hudson commented on HDFS-3058:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1090 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1090/])
Revert HDFS-3058 (Revision 1342493)
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. Contributed by Ivan Kelly. (Revision 1342483)
Result = ABORTED
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342493
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342483
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283205#comment-13283205 ]
Hadoop QA commented on HDFS-3058:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12529671/HDFS-3058.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2518//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2518//console
This message is automatically generated.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-3058:
--------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
3.0.0
2.0.1-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I have just committed this to trunk and branch-2. Thanks a lot, Ivan.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272383#comment-13272383 ]
jiraposter@reviews.apache.org commented on HDFS-3058:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4230/
-----------------------------------------------------------
(Updated 2012-05-10 14:44:08.706764)
Review request for hadoop-hdfs.
Summary
-------
There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
This addresses bug HDFS-3058.
http://issues.apache.org/jira/browse/HDFS-3058
Diffs (updated)
-----
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml 380ef62
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java 9d070d9
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java 7fa9026
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java b949bc2
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java 41f0292
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java c144906
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java 3810614
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java 5440c38
Diff: https://reviews.apache.org/r/4230/diff
Testing
-------
Thanks,
Ivan
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282253#comment-13282253 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Patch looks pretty good to me.
Here is the concern:
{code}
try {
+ long id = currentLedger.getId();
currentLedger.close();
+ bkc.deleteLedger(id);
} catch (Exception e2) {
{code}
Instead deleting ledger alone, con't we delete inprogressNode first and on success of deletion try to delete ledger? I think this is the point Rakesh mentioned above.
+1 On addressing the above comment.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283434#comment-13283434 ]
Hudson commented on HDFS-3058:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2361 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2361/])
Revert HDFS-3058 (Revision 1342493)
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. Contributed by Ivan Kelly. (Revision 1342483)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342493
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342483
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272392#comment-13272392 ]
Hadoop QA commented on HDFS-3058:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526350/HDFS-3058.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2404//console
This message is automatically generated.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283095#comment-13283095 ]
Eli Collins commented on HDFS-3058:
-----------------------------------
This change broke trunk and branch-2 compilation. I've reverted it from both branches.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reopened HDFS-3058:
-------------------------------
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278887#comment-13278887 ]
jiraposter@reviews.apache.org commented on HDFS-3058:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4230/
-----------------------------------------------------------
(Updated 2012-05-18 15:43:52.712478)
Review request for hadoop-hdfs.
Summary
-------
There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
This addresses bug HDFS-3058.
http://issues.apache.org/jira/browse/HDFS-3058
Diffs (updated)
-----
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml 380ef62
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java 9d070d9
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java 047efd5
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java PRE-CREATION
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java 5937fa8
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java 41f0292
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java c144906
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java b418fcf
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java 5440c38
Diff: https://reviews.apache.org/r/4230/diff
Testing
-------
Thanks,
Ivan
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-3058:
--------------------------------------
Status: Patch Available (was: Reopened)
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274530#comment-13274530 ]
Ivan Kelly commented on HDFS-3058:
----------------------------------
Regarding the patch applying, it depends on HDFS-2717, which fixes error handling on the bk output stream. This is necessary for the tests in HDFS-3058 to work.
Regarding 2) it not quite possible to do it that way, as we also need to tell the minicluster setup to allow the test to managed the shared storage dir, so we'd need an override to build the configuration and an override to build the minicluster, which I think is messier than whats there now.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280836#comment-13280836 ]
Hadoop QA commented on HDFS-3058:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528083/HDFS-3058.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2504//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2504//console
This message is automatically generated.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-3058:
-----------------------------
Attachment: HDFS-3058.diff
Rebased on trunk. Some fixups were required to tests, as the way Namenode exits from bad edit logs has changed.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283096#comment-13283096 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
I just realized it, patch did not apply cleanly, I will revert and reapply it again.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283100#comment-13283100 ]
Eli Collins commented on HDFS-3058:
-----------------------------------
Thanks Uma, please post a new patch rebased on trunk so jenkins can run again.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-3058:
-----------------------------
Status: Patch Available (was: Open)
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283101#comment-13283101 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Thanks Eli, patch command did not show any conflicts and committed based on above +1 but BookKeeperJournalManager has conflicts.I will resolve and update it.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-3058:
--------------------------------------
Attachment: HDFS-3058.patch
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283097#comment-13283097 ]
Eli Collins commented on HDFS-3058:
-----------------------------------
Here are the errors:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project hadoop-hdfs-bkjournal: Compilation failure: Compilation failure:
[ERROR] /home/eli/src/hadoop1/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java:[367,23] recoverLastTxId(org.apache.hadoop.contrib.bkjournal.EditLogLedgerMetadata,boolean) in org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager cannot be applied to (org.apache.hadoop.contrib.bkjournal.EditLogLedgerMetadata)
[ERROR] /home/eli/src/hadoop1/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java:[390,18] cannot find symbol
[ERROR] symbol : variable lastTxId
[ERROR] location: class org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager
[ERROR] /home/eli/src/hadoop1/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java:[391,24] cannot find symbol
[ERROR] symbol : variable lastTxId
[ERROR] location: class org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager
[ERROR] /home/eli/src/hadoop1/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java:[391,33] incompatible types
[ERROR] found : <nulltype>
[ERROR] required: long
[ERROR] -> [Help 1]
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283506#comment-13283506 ]
Hudson commented on HDFS-3058:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2307 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2307/])
Revert HDFS-3058 (Revision 1342493)
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. Contributed by Ivan Kelly. (Revision 1342483)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342493
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342483
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283319#comment-13283319 ]
Hudson commented on HDFS-3058:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1056 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1056/])
Revert HDFS-3058 (Revision 1342493)
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. Contributed by Ivan Kelly. (Revision 1342483)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342493
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342483
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3058.
-------------------------------
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.2-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-3058:
-----------------------------
Attachment: HDFS-3058.diff
Rebased onto trunk
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283160#comment-13283160 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Pending Jenkins report on latest patch for commit.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283446#comment-13283446 ]
Hudson commented on HDFS-3058:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2288 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2288/])
Revert HDFS-3058 (Revision 1342493)
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. Contributed by Ivan Kelly. (Revision 1342483)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342493
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1342483
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/BKJMUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280759#comment-13280759 ]
Rakesh R commented on HDFS-3058:
--------------------------------
Hi Ivan,
Its great work and good to see the hot standby flows. Just few thoughts over the patch.
# In BookKeeperJournalManager, startLogSegment() is deleting the corresponding 'inprogress_ledger' ledger on exception. Here leaving the 'inprogress_x' ledger metadata in ZooKeeper. When the other node becomes active, he will see the 'inprogress_x' znode and tries to recoverLastTxId() it would throw exception, since there is no 'inprogress_ledger' exists.
{noformat}
Caused by: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException
at org.apache.bookkeeper.client.BookKeeper.openLedger(BookKeeper.java:393)
at org.apache.hadoop.contrib.bkjournal.BookKeeperJournalManager.recoverLastTxId(BookKeeperJournalManager.java:493)
{noformat}
# Also, can we do maxTxId.store(txId); only when finalizeLogSegment(). I'm thinking to limit standby do tailing only from 'edit_logs' ?
-Rakesh
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283187#comment-13283187 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Here is the follow up JIRA for above discussed comments. HDFS-3464
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha, 3.0.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated HDFS-3058:
-----------------------------
Attachment: HDFS-3058.diff
Patch addresses TODOs. BookKeeperJournalManager now works with HA. This patch applies on top of the patch for HDFS-2717.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3058) HA: Bring BookKeeperJournalManager up
to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-3058:
--------------------------------------
Resolution: Fixed
Fix Version/s: (was: 3.0.0)
Status: Resolved (was: Patch Available)
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.patch
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278899#comment-13278899 ]
Hadoop QA commented on HDFS-3058:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528083/HDFS-3058.diff
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2476//console
This message is automatically generated.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3058) HA: Bring BookKeeperJournalManager
up to date with HA changes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274567#comment-13274567 ]
Uma Maheswara Rao G commented on HDFS-3058:
-------------------------------------------
Yep, Ivan I realized it, that we have to apply HDFS-2717 first.
I have seen it 'manageNameDfsSharedDirs(false)', it make sense to me for overriding whole.
Let me take a look on HDFS-2717 first, if I can review it.
> HA: Bring BookKeeperJournalManager up to date with HA changes
> -------------------------------------------------------------
>
> Key: HDFS-3058
> URL: https://issues.apache.org/jira/browse/HDFS-3058
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 0.24.0
>
> Attachments: HDFS-3058.diff, HDFS-3058.diff
>
>
> There's a couple of TODO(HA) comments in the BookKeeperJournalManager code. This JIRA is to address those.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira