You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2011/07/24 03:04:09 UTC
[jira] [Created] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Make the Replication Service pluggable via a standard interface definition
--------------------------------------------------------------------------
Key: HBASE-4131
URL: https://issues.apache.org/jira/browse/HBASE-4131
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jeff Whiting (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191408#comment-13191408 ]
Jeff Whiting commented on HBASE-4131:
-------------------------------------
This work is great. However we need this in 0.92 (and maybe 0.90). I'm thinking it shouldn't be too big of a deal to backport this as it doesn't change any replication functionality but just makes it pluggable.
I'll do the footwork of making the patches for the older versions and creating a new jira for the backport. Do you think it is this feasible to get this back ported?
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114985#comment-13114985 ]
Jonathan Gray commented on HBASE-4131:
--------------------------------------
@Ted, I don't think this diff is related to TestRollingRestart in any way. You might want to open a separate JIRA and put in your log file from the failed TestRollingRestart. Assign to me if you'd like me to take a look.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115000#comment-13115000 ]
Ted Yu commented on HBASE-4131:
-------------------------------
@Jonathan:
Thanks for taking Apache build seriously.
I created HBASE-4492 with reference to a failure on Jenkins.
I am +1 on patch v4 with the minor modification as mentioned @ 26/Sep/11 20:52
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113674#comment-13113674 ]
Jonathan Gray commented on HBASE-4131:
--------------------------------------
Thanks stack!
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reopened HBASE-4131:
--------------------------
Backed out this patch. Was causing TestReplication failures. See https://builds.apache.org/job/HBase-TRUNK/2246/testReport/
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070083#comment-13070083 ]
dhruba borthakur commented on HBASE-4131:
-----------------------------------------
To keep compatibility with the currently existing Replication Service, my initial proposal is to do something like this:
{code}
public interface ReplicationService {
/**
* Start replication services.
* @throws IOException
*/
public void startReplicationService() throws IOException;
/**
* Stops replication service.
*/
public void stopReplicationService();
/**
* Returns a WALObserver for the service. This is needed to
* observe log rolls and log archival events.
*/
public WALObserver getWALObserver();
/**
* Carry on the list of log entries down to the sink
* @param entries list of entries to replicate
* @throws IOException
*/
public void replicateLogEntries(HLog.Entry[] entries) throws IOException;
}
{code}
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4131:
------------------------------------
Attachment: replicationInterface3.txt
I removed redundant imports, rearranged the order of the constructors in Replication.java .
I left the const in HBaseConstants.java because they are needed by HBase even if no replication moduleis configured. But please feel free to put them somewhere else (at time of commit) if you so desire.
I also left the FileSystem object as a parameter into the initialize call, otherwise more code changes will be required in the Replication module. I can do that as a separate jira if you so desire (keeps this jira easier to review).
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4131:
-------------------------
Attachment: 4131-backedout.txt
Here is what I backed out.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114697#comment-13114697 ]
dhruba borthakur commented on HBASE-4131:
-----------------------------------------
Two unit tests failed (all other passed):
{code}
grep FAILURE target/surefire-reports/*.txt
target/surefire-reports/org.apache.hadoop.hbase.client.TestHTablePool.txt:Tests run: 18, Failures: 0, Errors: 8, Skipped: 0, Time elapsed: 147.044 sec <<< FAILURE!
target/surefire-reports/org.apache.hadoop.hbase.master.TestDistributedLogSplitting.txt:Tests run: 7, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 31.396 sec <<< FAILURE!
{code}
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4131:
------------------------------------
Attachment: replicationInterface4.txt
The ReplicationTests were failing because line 3121 was erroneously written as
{code}
server.replicationSinkHandler = (ReplicationSinkService)
server.replicationSinkHandler;
{code}
It should have been
{code}
server.replicationSinkHandler = (ReplicationSinkService)
server.replicationSourceHandler;
{code}
I fixed this one and verified that the Replication test suite passes now. I am still in the process of running the entire test suite and will post the results here tomorrow.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115039#comment-13115039 ]
Jonathan Gray commented on HBASE-4131:
--------------------------------------
Thanks Ted. Will commit with your suggestion.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4131:
------------------------------------
Attachment: replicationInterface2.txt
Addressed Stacks' comments. Split up the Replication Interface into two parts, the ReplicationSource and ReplicationSink Interfaces.
{code}
public interface ReplicationService {
public void initialize(Server rs, FileSystem fs, Path logdir,
Path oldLogDir) throws IOException;
public void startReplicationService() throws IOException;
public void stopReplicationService();
}
public interface ReplicationSourceService extends ReplicationService {
public WALActionsListener getWALActionsListener();
}
public interface ReplicationSinkService extends ReplicationService {
public void replicateLogEntries(HLog.Entry[] entries) throws IOException;
}
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112987#comment-13112987 ]
Jonathan Gray commented on HBASE-4131:
--------------------------------------
@JD, everything compiles for me. What are you seeing?
Running test suite now.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112849#comment-13112849 ]
Jonathan Gray commented on HBASE-4131:
--------------------------------------
Talking with Dhruba I think it doesn't make sense to have the replicateLogEntries() method in the ReplicationService API. There are really two replication service types, a source and a sink. HBase replication uses the same service for both, but other services may only want to be a source or a sink. I will let Dhruba propose the specific interfaces.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114791#comment-13114791 ]
dhruba borthakur commented on HBASE-4131:
-----------------------------------------
I checked out a fresh workspace from Apache trunk and ran the entire unit tests. The above two tests (TestDistributedLogSplitting and TestHTablePool) fail on that workspace too. So, the two unit test failure are not related to the patch in this JIRA. Please review and commit if found satisfactory.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113023#comment-13113023 ]
stack commented on HBASE-4131:
------------------------------
nit: Have replication constants in the replication package rather than up in HConstants (do they need to be there?)
nit: Looks like you are importing a bunch of crud you are not using in Interfaces at least.
nit: Do you need to pass the filesystem on initialize of ReplicationService? You can do:
FileSystem fs = FileSystem.get(server.getConfiguration());
Else +1 on patch if tests pass (Above nits can be fixed on commit)
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4131:
-------------------------
Resolution: Fixed
Fix Version/s: 0.94.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to TRUNK
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112965#comment-13112965 ]
Jean-Daniel Cryans commented on HBASE-4131:
-------------------------------------------
Looks good to me, but even patch v2 is incomplete right? I'm pretty sure some tests don't even compile.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4131:
------------------------------------
Release Note: The Replication Interface can be used to plug in external software for the purpose of cluster-to-cluster HBase replication.
Status: Patch Available (was: Open)
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112990#comment-13112990 ]
Jean-Daniel Cryans commented on HBASE-4131:
-------------------------------------------
TestReplicationSourceManager instantiates a Replication object and that patch changes its constructor.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt, replicationInterface2.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-4131.
----------------------------------
Resolution: Fixed
Committed to trunk. Good work Dhruba and thanks for review Ted!
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113850#comment-13113850 ]
dhruba borthakur commented on HBASE-4131:
-----------------------------------------
Stack: thanks for taking the time to review this patch. I had not yet run the full test suite and was expecting the submitPatch request to run it (similar to Hadoop). But extremely sorry that it got committed much earlier.
I am working on this one now and will verify and post what I find.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070096#comment-13070096 ]
stack commented on HBASE-4131:
------------------------------
This looks fine. How will replicateLogEntries work? The current replication reading the edits source would invoke it? The current replication would then need to change so that on invocation of replicateLogEntries, if it was the configured sink, it would pass the edits to the remote cluster as it does now?
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113774#comment-13113774 ]
Hudson commented on HBASE-4131:
-------------------------------
Integrated in HBase-TRUNK #2246 (See [https://builds.apache.org/job/HBase-TRUNK/2246/])
HBASE-4131 Make the Replication Service pluggable via a standard interface definition
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114900#comment-13114900 ]
Ted Yu commented on HBASE-4131:
-------------------------------
TestHTablePool took 4 minutes on Linux with latest patch and it passed.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Jonathan Gray (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114987#comment-13114987 ]
Jonathan Gray commented on HBASE-4131:
--------------------------------------
@Ted, so are you +1 to commit patch v4 now?
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114977#comment-13114977 ]
Ted Yu commented on HBASE-4131:
-------------------------------
I got one test failure in the suite:
{code}
testBasicRollingRestart(org.apache.hadoop.hbase.master.TestRollingRestart) Time elapsed: 300.28 sec <<< ERROR!
java.lang.Exception: test timed out after 300000 milliseconds
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.master.TestRollingRestart.waitForRSShutdownToStartAndFinish(TestRollingRestart.java:313)
at org.apache.hadoop.hbase.master.TestRollingRestart.testBasicRollingRestart(TestRollingRestart.java:210)
{code}
It passed when run standalone.
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Ted Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114914#comment-13114914 ]
Ted Yu commented on HBASE-4131:
-------------------------------
Minor comment for Replication.java:
{code}
+ this.zkHelper = new ReplicationZookeeper(server, this.replicating);
+ } catch (KeeperException ke) {
+ throw new IOException("Failed replication handler create", ke);
+ }
{code}
I think the exception message should include this.replicating
This can be done at time of integration.
Nice work Dhruba - looking forward to the follow-on JIRA(s)
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116762#comment-13116762 ]
Hudson commented on HBASE-4131:
-------------------------------
Integrated in HBase-TRUNK #2264 (See [https://builds.apache.org/job/HBase-TRUNK/2264/])
HBASE-4131 Make the Replication Service pluggable via a standard interface definition (dhruba via jgray)
jgray :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HBASE-4131:
------------------------------------
Attachment: replicationInterface1.txt
This patch creates an interface called ReplicationInterface. This is the interface that should be implemented by any Replication Module.
The classname shouldbe specified via a config variable hbase.replication.service. If this is not set, then the default name of the Replication class is org.apache.hadoop.hbase.replication.regionserver.Replication to maintain backward compatibility.
Stack: does this answer the question you asked earlier?
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: replicationInterface1.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4131) Make the Replication Service
pluggable via a standard interface definition
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113902#comment-13113902 ]
Hudson commented on HBASE-4131:
-------------------------------
Integrated in HBase-TRUNK #2247 (See [https://builds.apache.org/job/HBase-TRUNK/2247/])
HBASE-4131 Make the Replication Service pluggable via a standard interface definition; BACKED IT OUT -- WAS CAUSING TestReplication failures
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSourceService.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
> Make the Replication Service pluggable via a standard interface definition
> --------------------------------------------------------------------------
>
> Key: HBASE-4131
> URL: https://issues.apache.org/jira/browse/HBASE-4131
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0
>
> Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt
>
>
> The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira