You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Enis Soztutar (JIRA)" <ji...@apache.org> on 2012/06/04 22:22:23 UTC
[jira] [Created] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Enis Soztutar created HBASE-6160:
------------------------------------
Summary: META entries from daughters can be deleted before parent entries
Key: HBASE-6160
URL: https://issues.apache.org/jira/browse/HBASE-6160
Project: HBase
Issue Type: Bug
Components: client, regionserver
Affects Versions: 0.94.0, 0.92.2, 0.96.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
Region A is split to -> B, C
Region B is split to -> D, E
After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289696#comment-13289696 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.92 #450 (See [https://builds.apache.org/job/HBase-0.92/450/])
HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346506)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6160:
-------------------------
Attachment: HBASE-6160_v2.patch
What I applied to trunk (has Ted's suggested fixes)
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289125#comment-13289125 ]
Hadoop QA commented on HBASE-6160:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530889/HBASE-6160_v1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2105//console
This message is automatically generated.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6160:
-------------------------
Attachment: HBASE-6160v2092.txt
Here is patch I applied to 0.92 and 0.94 branches.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated HBASE-6160:
---------------------------------
Attachment: HBASE-6160_v2.patch
v2 patch addressing Ted's comments.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289131#comment-13289131 ]
Zhihong Ted Yu commented on HBASE-6160:
---------------------------------------
Minor comments:
{code}
+ * Scans META ane returns a pair of number of scanned rows, and
{code}
'ane' -> 'and'
I think 'a pair of' is not needed above.
{code}
+ //we could not clean the parent, so it's daughters should not be cleaned as well (HBASE-6160)
{code}
'as well' -> 'either'
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl closed HBASE-6160.
--------------------------------
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: Client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440147#comment-13440147 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.92-security #116 (See [https://builds.apache.org/job/HBase-0.92-security/116/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375159)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6160) META entries from daughters
can be deleted before parent entries
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289277#comment-13289277 ]
ramkrishna.s.vasudevan edited comment on HBASE-6160 at 6/5/12 9:21 AM:
-----------------------------------------------------------------------
@Enis
The great grandfather should have ideally been compacted right, so that it does not have reference files, because he was the one who got splitted first. But the father's reference files got cleared/compacted thus leading to this problem. Correct me if am wrong?
@Stack
Do we still need to prioritize the reference files so that compaction of ref files surely happen? So i think after this patch still the reference file of the great grand father may not get compacted and we may never allow the CJ to remove the entry from META? Correct me if am wrong Stack?
was (Author: ram_krish):
+1 on patch.
@Enis
The great grandfather should have ideally been compacted right, so that it does not have reference files, because he was the one who got splitted first. But the father's reference files got cleared/compacted thus leading to this problem. Correct me if am wrong?
@Stack
Do we still need to prioritize the reference files so that compaction of ref files surely happen? So i think after this patch still the reference file of the great grand father may not get compacted and we may never allow the CJ to remove the entry from META? Correct me if am wrong Stack?
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289584#comment-13289584 ]
ramkrishna.s.vasudevan commented on HBASE-6160:
-----------------------------------------------
HBASE-5161 is already there for prioritizing reference files. I had reopened it already some time back.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289614#comment-13289614 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-TRUNK #2990 (See [https://builds.apache.org/job/HBase-TRUNK/2990/])
HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346496)
Result = FAILURE
stack :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289664#comment-13289664 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.94 #249 (See [https://builds.apache.org/job/HBase-0.94/249/])
HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346503)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288875#comment-13288875 ]
Enis Soztutar commented on HBASE-6160:
--------------------------------------
One option for a fix is to ensure that META entry for the parent region is deleted before deleting the META entry, and do the META entry deletion recursively.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6160:
-------------------------
Resolution: Fixed
Fix Version/s: 0.94.1
0.92.2
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk, 0.92, and 0.94 branches. Thanks for the patch Enis.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289618#comment-13289618 ]
Enis Soztutar commented on HBASE-6160:
--------------------------------------
Thanks Stack, you beat me to the 92, and 94 patches.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289610#comment-13289610 ]
ramkrishna.s.vasudevan commented on HBASE-6160:
-----------------------------------------------
Thanks Enis and Stack for your reply.
bq.Yes, we still need to prioritize references files
I just wanted to say we need to do the prioritizing(along with this). In some cases that we saw the region kept on growing.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated HBASE-6160:
---------------------------------
Status: Patch Available (was: Open)
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.94.0, 0.92.2, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289843#comment-13289843 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #42 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/42/])
HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346496)
Result = FAILURE
stack :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437499#comment-13437499 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #136 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/136/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6160) META entries from daughters
can be deleted before parent entries
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288892#comment-13288892 ]
Zhihong Ted Yu edited comment on HBASE-6160 at 6/4/12 8:47 PM:
---------------------------------------------------------------
@Enis:
Can you attach log snippets to show the problem ?
e.g. what was the interval between the two splits.
was (Author: zhihyu@ebaysf.com):
@Enis:
Can you attach log snippets to show the problem ?
e.g. what was the duration between the two splits.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438163#comment-13438163 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.94 #407 (See [https://builds.apache.org/job/HBase-0.94/407/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158)
Result = SUCCESS
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288941#comment-13288941 ]
Enis Soztutar commented on HBASE-6160:
--------------------------------------
The exception:
{code}
12/06/04 06:50:41 ERROR security.UserGroupInformation: PriviledgedActionException as: cause:org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META.
Exception in thread "main" org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. cannot be found in META.
at org.apache.hadoop.hbase.client.MetaScanner$BlockingMetaScannerVisitor.processRow(MetaScanner.java:433)
at org.apache.hadoop.hbase.client.MetaScanner$TableMetaScannerVisitor.processRow(MetaScanner.java:490)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:227)
at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:57)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:136)
at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:361)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:108)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:86)
at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:326)
at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:499)
at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:452)
at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:132)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
{code}
So the region in question is
{code}
50a4617eead34cad335a8dfa727d177d
{code}
and from the logs we see that {{25d9c4ff574a37bd95bf5e5be6d618dd}} is split into {{1dc74065583c67b3916c4ed158cb53fa}} and {{50a4617eead34cad335a8dfa727d177d}}
{code}
./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 04:56:02,855 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd., new regions: TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa., TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.. Split took 4sec
{code}
After some time, {{50a4617eead34cad335a8dfa727d177d}} is further split into two:
{code}
./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 05:41:13,488 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META updated, and report to master. Parent=TestLoadAndVerify_1
338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d., new regions: TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338802866393.16288
65d7fa8e9eec3a7d8073465296e., TestLoadAndVerify_1338798130970,]y\x04\x00\x00\x00\x00\x00/000047_0,1338802866393.413cafe6c61426e26254c197e8c0a6ba.. Split took 7sec
{code}
Further time passes, and CatalogJanitor deletes the META entry for that region:
{code}
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:16,688 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. because daughter splits no longer hold references
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughters references, qualifier=splitA and qualifier=splitB, from parent TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,145 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d. from META
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:52:12,891 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
{code}
However, there is no log for the grantfather,25d9c4ff574a37bd95bf5e5be6d618dd, being deleted from META. In fact, META scan also confirms that the grandfather is in META, but not the father:
{code}
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:regioninfo, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 25d9c4ff574a37bd95bf5e5be6d618dd, OFFLINE => true, SPLIT => true,}
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:server, timestamp=1338799025521, value=ip-10-226-65-102.ec2.internal:60020
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:serverstartcode, timestamp=1338799025521, value=1338780117340
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitA, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.', STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENCODED => 1dc74065583c67b3916c4ed158cb53fa,}
TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd. column=info:splitB, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.', STARTKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl', ENCODED => 50a4617eead34cad335a8dfa727d177d,}
{code}
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289127#comment-13289127 ]
stack commented on HBASE-6160:
------------------------------
Weird. I thought this had been fixed previous. Patch looks good though Enis and nice test. Will commit unless objection in morning.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437443#comment-13437443 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-TRUNK #3239 (See [https://builds.apache.org/job/HBase-TRUNK/3239/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676)
Result = SUCCESS
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448327#comment-13448327 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158)
Result = FAILURE
Tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288892#comment-13288892 ]
Zhihong Ted Yu commented on HBASE-6160:
---------------------------------------
@Enis:
Can you attach log snippets to show the problem ?
e.g. what was the duration between the two splits.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289801#comment-13289801 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.94-security #35 (See [https://builds.apache.org/job/HBase-0.94-security/35/])
HBASE-6160 META entries from daughters can be deleted before parent entries (Revision 1346503)
Result = SUCCESS
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289615#comment-13289615 ]
stack commented on HBASE-6160:
------------------------------
bq. In some cases that we saw the region kept on growing.
Yeah, that can happen. Usually though because of other issues; a write rate in excess of compaction rate, etc.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160_v2.patch, HBASE-6160_v2.patch, HBASE-6160v2092.txt
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289572#comment-13289572 ]
Enis Soztutar commented on HBASE-6160:
--------------------------------------
@Ramkrishna
yes, ideally that is the case. But we may end up with this, if for example, the regions are non-uniform. I think we still have to prioritize the ref files in compaction, since they also prevent splitting further.
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enis Soztutar updated HBASE-6160:
---------------------------------
Attachment: HBASE-6160_v1.patch
Attaching a patch for trunk.
- Changes CatalogJanitor to not delete split parents, whose parents are still in META.
- Adds a test case
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "ramkrishna.s.vasudevan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289277#comment-13289277 ]
ramkrishna.s.vasudevan commented on HBASE-6160:
-----------------------------------------------
+1 on patch.
@Enis
The great grandfather should have ideally been compacted right, so that it does not have reference files, because he was the one who got splitted first. But the father's reference files got cleared/compacted thus leading to this problem. Correct me if am wrong?
@Stack
Do we still need to prioritize the reference files so that compaction of ref files surely happen? So i think after this patch still the reference file of the great grand father may not get compacted and we may never allow the CJ to remove the entry from META? Correct me if am wrong Stack?
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439501#comment-13439501 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.94-security #48 (See [https://builds.apache.org/job/HBase-0.94-security/48/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375158)
Result = FAILURE
tedyu :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438241#comment-13438241 ]
Hudson commented on HBASE-6160:
-------------------------------
Integrated in HBase-0.92 #508 (See [https://builds.apache.org/job/HBase-0.92/508/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1375159)
Result = SUCCESS
tedyu :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Fix For: 0.92.2, 0.94.1
>
> Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be
deleted before parent entries
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289585#comment-13289585 ]
stack commented on HBASE-6160:
------------------------------
@Ram Yes, the father was split and his children subsequently compacted and let go of references so he was removed but his no-good brother was still around... with references probably so the ancestor could not yet be cleared. The RegionOfflineException is because of the fix over in HBASE-5986.
Yes, we still need to prioritize references files. I'd think that the no-good brother would eventually compact away his references making the great grandfather removable. Why would this patch change compactions or the rate at which regions are removed from .META. Ram? (I can see removals perhaps going a little slower, more lumpy, but otherwise, should be as it is now)?
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
> Key: HBASE-6160
> URL: https://issues.apache.org/jira/browse/HBASE-6160
> Project: HBase
> Issue Type: Bug
> Components: client, regionserver
> Affects Versions: 0.92.2, 0.94.0, 0.96.0
> Reporter: Enis Soztutar
> Assignee: Enis Soztutar
> Attachments: HBASE-6160_v1.patch
>
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests:
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira