You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/07/21 02:24:50 UTC
[jira] Created: (HBASE-2858) TestReplication.queueFailover fails
half the time
TestReplication.queueFailover fails half the time
-------------------------------------------------
Key: HBASE-2858
URL: https://issues.apache.org/jira/browse/HBASE-2858
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.90.0
TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2858) TestReplication.queueFailover fails
half the time
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-2858.
---------------------------------------
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed to trunk, thanks for the review Stack!
> TestReplication.queueFailover fails half the time
> -------------------------------------------------
>
> Key: HBASE-2858
> URL: https://issues.apache.org/jira/browse/HBASE-2858
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2858) TestReplication.queueFailover fails
half the time
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890518#action_12890518 ]
HBase Review Board commented on HBASE-2858:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/349/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
This patch fixes the ZKW.listZNodes issue and clears up a bit the path situation in ReplicationSource by removing a lock and adding wits to figure where the log is moved. The test now passes 100% of the time for me (up from 50%).
There's one open issues as outlined by the two TODOS, what happens if a log is missing from HDFS? When the queue is recovered, it could mean that HDFS was cleared but not ZK, but during normal operations it would point to a bug? Report and continue?
This addresses bug HBASE-2858.
http://issues.apache.org/jira/browse/HBASE-2858
Diffs
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e6b365e
src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java a037aae
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 6b9dcb5
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java 2e13a0a
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java e8dd268
src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 163671f
src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java bb09bc3
Diff: http://review.hbase.org/r/349/diff
Testing
-------
Unit testing.
Thanks,
Jean-Daniel
> TestReplication.queueFailover fails half the time
> -------------------------------------------------
>
> Key: HBASE-2858
> URL: https://issues.apache.org/jira/browse/HBASE-2858
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2858) TestReplication.queueFailover fails
half the time
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890548#action_12890548 ]
HBase Review Board commented on HBASE-2858:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/349/#review435
-----------------------------------------------------------
Looks good. I'm not clear on some of below. Can you clarify before I +1 it?
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
<http://review.hbase.org/r/349/#comment1834>
Just remove this method altogether?
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
<http://review.hbase.org/r/349/#comment1835>
Is this right? You are getting a reader here. If file is not in expected location, we fall into FNFE area. These just seem to be testing existance, not getting a reader on the new location if it exists.
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
<http://review.hbase.org/r/349/#comment1836>
See above comment on this same method.
- stack
> TestReplication.queueFailover fails half the time
> -------------------------------------------------
>
> Key: HBASE-2858
> URL: https://issues.apache.org/jira/browse/HBASE-2858
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2858) TestReplication.queueFailover fails
half the time
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890955#action_12890955 ]
HBase Review Board commented on HBASE-2858:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/349/#review444
-----------------------------------------------------------
Ship it!
+1
- stack
> TestReplication.queueFailover fails half the time
> -------------------------------------------------
>
> Key: HBASE-2858
> URL: https://issues.apache.org/jira/browse/HBASE-2858
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2858) TestReplication.queueFailover fails
half the time
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890817#action_12890817 ]
HBase Review Board commented on HBASE-2858:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
bq. On 2010-07-20 20:34:23, stack wrote:
bq. > src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java, line 221
bq. > <http://review.hbase.org/r/349/diff/1/?file=2968#file2968line221>
bq. >
bq. > Just remove this method altogether?
I was thinking of keeping it around in the interface for other uses... but currently there's none. Deleting.
bq. On 2010-07-20 20:34:23, stack wrote:
bq. > src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java, line 437
bq. > <http://review.hbase.org/r/349/diff/1/?file=2968#file2968line437>
bq. >
bq. > Is this right? You are getting a reader here. If file is not in expected location, we fall into FNFE area. These just seem to be testing existance, not getting a reader on the new location if it exists.
If the queue was recovered, we're waiting for the master to finish splitting it to start reading from it. Then it sleeps and retries.
If it's a normal queue, the location is updated. Then it sleeps and retries with the new location. I guess it could get the reader right away.
- Jean-Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/349/#review435
-----------------------------------------------------------
> TestReplication.queueFailover fails half the time
> -------------------------------------------------
>
> Key: HBASE-2858
> URL: https://issues.apache.org/jira/browse/HBASE-2858
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2858) TestReplication.queueFailover fails
half the time
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890878#action_12890878 ]
HBase Review Board commented on HBASE-2858:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/349/
-----------------------------------------------------------
(Updated 2010-07-21 13:33:51.105842)
Review request for hbase.
Changes
-------
Updated diff, removed the logArchived method and, when a log is archived, we start reading from the new location right away.
Summary
-------
This patch fixes the ZKW.listZNodes issue and clears up a bit the path situation in ReplicationSource by removing a lock and adding wits to figure where the log is moved. The test now passes 100% of the time for me (up from 50%).
There's one open issues as outlined by the two TODOS, what happens if a log is missing from HDFS? When the queue is recovered, it could mean that HDFS was cleared but not ZK, but during normal operations it would point to a bug? Report and continue?
This addresses bug HBASE-2858.
http://issues.apache.org/jira/browse/HBASE-2858
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e6b365e
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java e1b4077
src/main/java/org/apache/hadoop/hbase/regionserver/wal/LogActionsListener.java 9b18992
src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java a037aae
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 6b9dcb5
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java 1d7ae7e
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java 2e13a0a
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java e8dd268
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogActionsListener.java d03809c
src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java ad9e8fb
src/test/java/org/apache/hadoop/hbase/replication/TestReplication.java 163671f
src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java bb09bc3
Diff: http://review.hbase.org/r/349/diff
Testing
-------
Unit testing.
Thanks,
Jean-Daniel
> TestReplication.queueFailover fails half the time
> -------------------------------------------------
>
> Key: HBASE-2858
> URL: https://issues.apache.org/jira/browse/HBASE-2858
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> TestReplication.queueFailover fails 50% of the time, it's because ZooKeeperWrapper.listZnodes (introduced in HBASE-2694 and missed by HBASE-2735) doesn't use the Watcher it's passed so sometimes ReplicationSource misses hlogs to replicate for the region server we kill. Also it uncovered an issue (while I was fixing the first one) that RepSource ignores log files too quickly when the master is a bit too slow to split logs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.