You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hari Mankude (Created) (JIRA)" <ji...@apache.org> on 2012/04/03 22:44:28 UTC
[jira] [Created] (HDFS-3186) Sync lagging journal service from the
active journal service
Sync lagging journal service from the active journal service
------------------------------------------------------------
Key: HDFS-3186
URL: https://issues.apache.org/jira/browse/HDFS-3186
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Hari Mankude
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "Hari Mankude (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245874#comment-13245874 ]
Hari Mankude commented on HDFS-3186:
------------------------------------
We need to have a mechanism for the newly added journal service to get the missing edit logs from the active journal service daemons.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3186:
-----------------------------
Attachment: HDFS-3186.HDFS-3092.patch
New patch has the added descriptions for the tests.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3186:
-----------------------------
Attachment: HDFS-3186.HDFS-3092.patch
rebase the patch with HDFS-3092 head
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3186:
-----------------------------
Attachment: HDFS-3186.HDFS-3092.patch
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269291#comment-13269291 ]
Suresh Srinivas commented on HDFS-3186:
---------------------------------------
There are bunch of TODO comments in the patch, these should be addressed before merging this branch to trunk.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269290#comment-13269290 ]
Suresh Srinivas commented on HDFS-3186:
---------------------------------------
Thanks for adding more descriptions to the tests. +1 for the patch.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266965#comment-13266965 ]
Suresh Srinivas commented on HDFS-3186:
---------------------------------------
Preliminary comments:
# JournalService.java
#* JournalSync conf should be renamed as it hides outer class conf.
#* Remove unnecessary casting 3 to long
#* In startLogSegment the comment you have added is unnecessary
#* Why do you want to throw IllegalStateException in inSync() method?
#* Update the JournalService constructor javadoc for newly added parameters
#* Why should syncThread be a never ending thread?
#* There are bunch of TODO items
#* Why would syncThread be not null in run() method?
#* Please use space after // in comments. Also for method comments use /** */ comments.
#* runSync - startSync name better? valueSet may not be needed?
#* enableRunSync() - startSync() better name?
#* getRunSync() - waitForStartSync() better name?
#* Add javadoc to JouranalSync methods
#* rename getAllJournalSegments to syncAllJournalSegments?
#* "Sync trial failed for" to "Sync failed for"
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13267049#comment-13267049 ]
Brandon Li commented on HDFS-3186:
----------------------------------
{quote}Why do you want to throw IllegalStateException in inSync() method?{quote}
I don't think I get the question. do we want to fail in a different way here?
{quote}Why should syncThread be a never ending thread?{quote}
It is because sync may be required anytime due to NN failover or journal service failure. If it's not a never ending thread, journal service needs to spawn sync thread on demand and lets the sync thread exit when sync is done. Both approaches need to handle the same set of situations, such as, the journal service state changes before sync finishes. The communication between main thread and sync thread is strait forward in both cases, and the never-ending-thread approach can avoid the cost of repeated thread creation.
{quote}There are bunch of TODO items{quote}
One cause of the TODOs is that, the NN is not updated to cooperate with journal service.
{quote}Why would syncThread be not null in run() method?{quote}
removed this check. I was over cautious there.
{quote}runSync - startSync name better? valueSet may not be needed?{quote}
changed name. removed valueSet since it's redundent with another veriable.
Other comments are addressed.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3186:
-----------------------------
Attachment: HDFS-3186.HDFS-3092.patch
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "amith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269325#comment-13269325 ]
amith commented on HDFS-3186:
-----------------------------
Hi, sorry for jumping late
{code}
if (minTxid == -1 || minTxid == -1) {
{code}
I think one of the var should be maxTxid
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Fix For: Shared journals (HDFS-3092)
>
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3186) Sync lagging journal service from
the active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13269362#comment-13269362 ]
Brandon Li commented on HDFS-3186:
----------------------------------
Thank you Amith! I will fix it in my next patch.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Fix For: Shared journals (HDFS-3092)
>
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Hari Mankude (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude reassigned HDFS-3186:
----------------------------------
Assignee: Brandon Li (was: Hari Mankude)
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3186:
-----------------------------
Attachment: HDFS-3186.HDFS-3092.patch
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Hari Mankude (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Mankude reassigned HDFS-3186:
----------------------------------
Assignee: Hari Mankude
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Hari Mankude
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas resolved HDFS-3186.
-----------------------------------
Resolution: Fixed
Fix Version/s: Shared journals (HDFS-3092)
Hadoop Flags: Reviewed
I committed the patch. Thank you Brandon.
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Fix For: Shared journals (HDFS-3092)
>
> Attachments: HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch, HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3186) Sync lagging journal service from the
active journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3186:
-----------------------------
Attachment: HDFS-3186.HDFS-3092.patch
The basic mechanism is like the following:
The journal service spawns a sync thread to download journal segments form other journal services. The sync thread contacts other journal services one by one until it can sync all the missed journal segments.
After the synchronization is done, the sync thread changes the journal service state from SYNCING to IN_SYNC, and then blocks itself. The journal service wakes up the sync thread to do another round of synchronization when its state transits from WAITING_FOR_ROLL to SYNCING.
Here is the journal service state auto-machine https://issues.apache.org/jira/secure/attachment/12524775/JNStates.png
> Sync lagging journal service from the active journal service
> ------------------------------------------------------------
>
> Key: HDFS-3186
> URL: https://issues.apache.org/jira/browse/HDFS-3186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Hari Mankude
> Assignee: Brandon Li
> Attachments: HDFS-3186.HDFS-3092.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira