You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Rakesh R (JIRA)" <ji...@apache.org> on 2012/06/26 10:27:44 UTC
[jira] [Created] (BOOKKEEPER-319) Manage auditing and replication
processes
Rakesh R created BOOKKEEPER-319:
-----------------------------------
Summary: Manage auditing and replication processes
Key: BOOKKEEPER-319
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
Project: Bookkeeper
Issue Type: Sub-task
Reporter: Rakesh R
This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469606#comment-13469606 ]
Hudson commented on BOOKKEEPER-319:
-----------------------------------
Integrated in bookkeeper-trunk #740 (See [https://builds.apache.org/job/bookkeeper-trunk/740/])
BOOKKEEPER-319: Manage auditing and replication processes (Vinay via ivank) (Revision 1394145)
Result = UNSTABLE
ivank :
Files :
* /zookeeper/bookkeeper/trunk/CHANGES.txt
* /zookeeper/bookkeeper/trunk/bookkeeper-server/bin/bookkeeper
* /zookeeper/bookkeeper/trunk/bookkeeper-server/bin/bookkeeper-daemon.sh
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/AuditorElector.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/AutoRecoveryMain.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/ReplicationWorker.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/replication/AutoRecoveryMainTest.java
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch, BOOKKEEPER-319.patch, BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-319:
----------------------------------
Fix Version/s: 4.2.0
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated BOOKKEEPER-319:
-------------------------------------------
Component/s: bookkeeper-auto-recovery
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443074#comment-13443074 ]
Vinay commented on BOOKKEEPER-319:
----------------------------------
Hi All,
I want to take up this Jira.
I assume the following from various discussions under BOOKKEEPER-237.
# Auditor will be a separate daemon, which also contains ReplicationWorker, which should be started in all bookie machines.
# One of them will be elected as Auditor and start publishing the bookie failures and under-replicated ledgers. Replication workers start replicating to local bookie.
So here are my thoughts about current Jira,
# Create a class do start Auditor service and replication worker. How about having class name as "AuditorPeer", similar to zookeeper "QuorumPeer"..?
# Include this in the bookkeeper scripts.
# Assuming that this daemon will be started from the same installation and configurations as of bookie.
Any thoughts, suggestions, inputs are welcome.
Thanks.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G reassigned BOOKKEEPER-319:
----------------------------------------------
Assignee: Vinay
Sounds great Vinay.
Coming to naming of the process, I wanted get the opinions from Ivan/Flavio as Ivan might have already thought about some good name :-)
I can see why you are asking for other name rather than simply auditor. It's because only one process can act as Auditor and others are just workers and participants for next elections.
Anyway, that can be updated once name decided. You can start work on other areas of this patch and once name finalized you can update it. So, that should not be a blocking for you.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460505#comment-13460505 ]
Ivan Kelly commented on BOOKKEEPER-319:
---------------------------------------
The new patch is good except for AutoRecoveryMain#join(). Why not just join the deathWatcher thread?
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch, BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462367#comment-13462367 ]
Uma Maheswara Rao G commented on BOOKKEEPER-319:
------------------------------------------------
+1 , I am fine to push this in. Thanks Vinay.
@Ivan, Could you please push this in, if you like the latest one. As this is the startUp for recovery process,
Surendra wanted to start verifying the scenarios with recoevry Node after this patch in.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch, BOOKKEEPER-319.patch, BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459659#comment-13459659 ]
Ivan Kelly commented on BOOKKEEPER-319:
---------------------------------------
Patch looks good, but needs a rebase against trunk. I have the following comments.
For naming I would suggest calling the command (for the bash script), "autorec" or "autorecovery", and the main class "AutoRecoveryMain". We shouldn't expose the actual implementation to the user, we just want to them to know that a auto recovery service is available.
Typo in the name of the death watcher thread.
The help for "conf" should be "Bookie server configuration".
A ParseException in parseArgs will cause the same error to be printed twice.
In printUsage, there should be no "or". It should be
{code}
hf.printHelp("AutoRecoveryMain [options]\n", opts);
{code}
getMyBookieAddress can be static.
getNewZookeeper exists in AuditorPeer and in ZkUtils. There is already a version of it in ZkUtils now as well.
Also, you're catching Exception e in main(). Could you run "mvn install findbugs:check" once you've rebased to ensure this isn't causing a findbugs error.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Rakesh R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401240#comment-13401240 ]
Rakesh R commented on BOOKKEEPER-319:
-------------------------------------
Following are some of the discussion threads from BOOKKEEPER-272
- As a bookie process
https://issues.apache.org/jira/browse/BOOKKEEPER-272?focusedCommentId=13400581&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13400581
- As a separate process
https://issues.apache.org/jira/browse/BOOKKEEPER-272?focusedCommentId=13400628&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13400628
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Reporter: Rakesh R
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated BOOKKEEPER-319:
-----------------------------
Attachment: BOOKKEEPER-319.patch
Attaching the latest patch addressing both Uma and Ivan's comments.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch, BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Uma Maheswara Rao G (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457201#comment-13457201 ]
Uma Maheswara Rao G commented on BOOKKEEPER-319:
------------------------------------------------
Hi Vinay, thanks for the patch.
{code}
private ZooKeeper getNewZookeeper(ServerConfiguration conf)
+ throws IOException, InterruptedException, KeeperException {
+ ZooKeeper newZk = ZkUtils.getNewZookeeperClient(conf.getZkServers(),
+ conf.getZkTimeout(), new Watcher() {
{code}
Please use ZKUtils#getNewConnectedClient instead, as you added it recently in format patch.
{code}
*/
+ public void join() throws InterruptedException {
+ synchronized (this) {
+ while (auditorElector.isRunning()) {
+ wait(100);
+ }
+ while (replicationWorker.isRunning()) {
+ wait(100);
+ }
+ }
{code}
Could you please give me a hint why you added this log in join, did not strikes it to me on first look.
anyway shutdown is calling auditor and rw shutdown, so they will be interrupted and joined there?
Simply why can't we add add join in this method?
{code}
private static class AuditorPeerDeathWatcher extends Thread {
+ int watchInterval;
+ AuditorPeer peer;
{code}
why can't we make these variable private?
{code}
new File(confFile).toURI().toURL()
{code}
directly also, you can convert to url right? any issue?
{code}
} catch (Exception e) {
+ LOG.error("Exception running bookie server : ", e);
{code}
Why bookie server?
{code}
"Shut down AuditorPeer
{code}
typo?
isRunning --> add small doc for this
These are some comments on first look. Did not take a look into tests much
I would request, Ivan to take a look for better naming here for the process.
Also please add your comments
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated BOOKKEEPER-319:
-----------------------------
Attachment: BOOKKEEPER-319.patch
Attaching the patch for review. Here I have named the class as 'AuditorPeer' itself.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Attachments: BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinay updated BOOKKEEPER-319:
-----------------------------
Attachment: BOOKKEEPER-319.patch
Thanks Ivan for the hint.. I just changed that and uploading updated patch.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-319.patch, BOOKKEEPER-319.patch, BOOKKEEPER-319.patch
>
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-319) Manage auditing and replication
processes
Posted by "Vinay (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443799#comment-13443799 ]
Vinay commented on BOOKKEEPER-319:
----------------------------------
Thanks Uma,
Yeah, I will start working with other things. Name I will update once its finalized.
> Manage auditing and replication processes
> -----------------------------------------
>
> Key: BOOKKEEPER-319
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-319
> Project: Bookkeeper
> Issue Type: Sub-task
> Components: bookkeeper-auto-recovery
> Reporter: Rakesh R
> Assignee: Vinay
>
> This subtask discusses, how we will manage the whole rereplication processes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira