You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/09/17 20:43:34 UTC
[jira] Created: (HBASE-3013) Tool to verify data in two clusters
Tool to verify data in two clusters
-----------------------------------
Key: HBASE-3013
URL: https://issues.apache.org/jira/browse/HBASE-3013
Project: HBase
Issue Type: New Feature
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.90.0
It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926682#action_12926682 ]
stack commented on HBASE-3013:
------------------------------
Sorry J-D, I must have had a stale view.
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
> Attachments: HBASE-3013-v2.patch
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3013) Tool to verify data in two clusters
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-3013.
---------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Committed to trunk.
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
> Attachments: HBASE-3013-v2.patch
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926659#action_12926659 ]
Jean-Daniel Cryans commented on HBASE-3013:
-------------------------------------------
@Stack, I resolved it 2 days ago, am I missing something?
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
> Attachments: HBASE-3013-v2.patch
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918300#action_12918300 ]
Jean-Daniel Cryans commented on HBASE-3013:
-------------------------------------------
No, I have to the tool here I just need to post the patch.
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3013) Tool to verify data in two clusters
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-3013:
--------------------------------------
Attachment: HBASE-3013-v2.patch
Final patch I'm about to commit.
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
> Attachments: HBASE-3013-v2.patch
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918220#action_12918220 ]
Jonathan Gray commented on HBASE-3013:
--------------------------------------
Punt to 0.92?
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925996#action_12925996 ]
HBase Review Board commented on HBASE-3013:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1111/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
This new mapreduce job called VerifyReplication compares the data between two clusters that are replication-enabled. Its usage is relatively simple when you already use replication, and even let's you pass the peer id instead of the cluster key for the target cluster.
This addresses bug HBASE-3013.
http://issues.apache.org/jira/browse/HBASE-3013
Diffs
-----
/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java 1028470
/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java PRE-CREATION
/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java 1028470
/trunk/src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 1028470
Diff: http://review.cloudera.org/r/1111/diff
Testing
-------
Unit test (one new included) and this has been running for a month here.
Thanks,
Jean-Daniel
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926628#action_12926628 ]
stack commented on HBASE-3013:
------------------------------
Can we close this now?
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
> Attachments: HBASE-3013-v2.patch
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3013) Tool to verify data in two clusters
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926010#action_12926010 ]
HBase Review Board commented on HBASE-3013:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1111/#review1693
-----------------------------------------------------------
Ship it!
Excellent. Minor items to address on commit. So, is this new fancy tool doc'd in replication notes? I think it kinda critical. Would suggest opening a doc issue against 0.90 to mention presence of this tool in the replication chapter.... if only a sentence.
/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
<http://review.cloudera.org/r/1111/#comment5603>
Missing '<p>' here?
/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
<http://review.cloudera.org/r/1111/#comment5602>
Space around the '+'
/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
<http://review.cloudera.org/r/1111/#comment5604>
Put this in Result?
- stack
> Tool to verify data in two clusters
> -----------------------------------
>
> Key: HBASE-3013
> URL: https://issues.apache.org/jira/browse/HBASE-3013
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> It would be useful to have a tool to easily compare the data between tables in different clusters, at least to make sure that replication is working correctly. I'm thinking of building that inside CopyTable, kind of an option à là --verify that could be run independently or after the copy (or not at all). The fact that we can already pass start/stop times is pretty useful too when you don't want to check the whole tables, do incremental verifications, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.