You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/08/13 21:20:16 UTC
[jira] Created: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Deadlock between HRegion.ICV and HRegion.close
----------------------------------------------
Key: HBASE-2915
URL: https://issues.apache.org/jira/browse/HBASE-2915
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.90.0
HRegion.ICV gets a row lock then gets a newScanner lock.
HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-2915:
-----------------------------------------
Assignee: Jean-Daniel Cryans
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900510#action_12900510 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Ryan Rawson" <ry...@gmail.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review966
-----------------------------------------------------------
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3144>
oh wow i cant believe this was ever here
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3145>
i thought we agreed that the closing flag had to be set BEFORE the write lock was acquired to prevent race conditions?
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3146>
lets just excise this and break compile time compatibility. Also remove internalPreFlushcacheCommit too
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3147>
ditto remove this whole try/finally bit
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3148>
maybe we shouldnt call this 'transaction' might confuse people into thinking we support real transactions... not sure what to call it at this moment tho
- Ryan
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900912#action_12900912 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review975
-----------------------------------------------------------
oh, one other thing, in discussions, we talked of no longer needing to wait on row locks to expire... I don't see this being excised from the close method. Should that be in here?
- stack
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2915:
--------------------------------------
Attachment: HBASE-2915-v4.patch
This patch is the same as the last one I posted on rb, with the addition of the unlocking in case the region is closed but the thread was able to get a readLock. I tested it on a 13 nodes cluster with 200 YCSB threads hitting it in different ways, didn't see any deadlock. Going to commit.
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: HBASE-2915-v4.patch
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900925#action_12900925 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 507
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line507>
bq. >
bq. > I suppose this order is ok if the first thing we do on entrance to HRegion is get the read lock before check of closing.
bq.
bq. Jean-Daniel Cryans wrote:
bq. So I just redid that part. setClosing is first taken so that when the client threads arrive they can fast fail by looking at closing.get before trying to get the readLock.
bq.
bq. stack wrote:
bq. Don't you have to check again the setClosing after you get the read lock?
I'm about to post a new patch, but it looks like this and it has to be called just before "try" instead of inside:
if (this.closing.get()) {
throw new NotServingRegionException(regionInfo.getRegionNameAsString() +
" is closing");
}
lock.readLock().lock();
if (this.closed.get()) {
throw new NotServingRegionException(regionInfo.getRegionNameAsString() +
" is closed");
}
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 712
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line712>
bq. >
bq. > Seems like you could use your opentransaction/closetransaction methods here and in flush too to be consistent?
bq.
bq. Jean-Daniel Cryans wrote:
bq. Yeah the issue with compact and flush is that the callers don't expect to see NSRE, the want null values.
bq.
bq. stack wrote:
bq. OK. Not important. This is deep internal stuff or make a version that takes a flag on whether to throw exception (default throws exception .. might get messy though... not important).
I'm afraid those little methods could be clogged fast.
- Jean-Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review974
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900928#action_12900928 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 712
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line712>
bq. >
bq. > Seems like you could use your opentransaction/closetransaction methods here and in flush too to be consistent?
bq.
bq. Jean-Daniel Cryans wrote:
bq. Yeah the issue with compact and flush is that the callers don't expect to see NSRE, the want null values.
bq.
bq. stack wrote:
bq. OK. Not important. This is deep internal stuff or make a version that takes a flag on whether to throw exception (default throws exception .. might get messy though... not important).
bq.
bq. Jean-Daniel Cryans wrote:
bq. I'm afraid those little methods could be clogged fast.
bq.
bq. stack wrote:
bq. Yeah. Not important.
Moreover, and this should be added to the documentation, none of the user-facing methods now directly use the lock, it has to go through those new methods. Means that flush, compact, split and close are the only ones with direct access.
- Jean-Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review974
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900914#action_12900914 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review974
-----------------------------------------------------------
Ship it!
+1 Nice fix. A few comments below.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3163>
I suppose this order is ok if the first thing we do on entrance to HRegion is get the read lock before check of closing.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3164>
Seems like you could use your opentransaction/closetransaction methods here and in flush too to be consistent?
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3165>
Yeah, just remove.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3166>
Aren't these lines unnecessary? openRegionTransaction does it?
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3167>
So, this javadoc is good but do you think we need some more doc? Does there need to be more detail on new locking regime? Maybe there is no more to be said that what is here in this paragraph. You've done all the work unravelling our lock mess. With time your nice unravelling will rot unless its clear what the pattern is. I'm just trying to think of ways of preventing that happening.
- stack
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900527#action_12900527 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Ted Yu" <te...@yahoo.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review969
-----------------------------------------------------------
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3155>
Or regionOperationProlog()
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3156>
and regionOperationEpilog()
- Ted
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900492#action_12900492 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
This patch removes newScannerLock and renames splitAndClose lock to just "lock". Every operation is now required to obtain the read lock on "lock" before doing anything (including getting a row lock). This is done by calling openRegionTransaction inside a try statement and by calling closeRegionTransaction in finally.
flushcache got refactored some more in order to do the locking in the proper order; first get the read lock, then do the writestate handling.
Finally, it removes the need to have a writeLock when flushing when subclassers give atomic work do to via internalPreFlushcacheCommit. This means that this patch breaks external contribs. This is required to keep our whole locking mechanism simpler.
This addresses bug HBASE-2915.
http://issues.apache.org/jira/browse/HBASE-2915
Diffs
-----
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 987300
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 987300
/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 987300
Diff: http://review.cloudera.org/r/691/diff
Testing
-------
5 concurrent ICV threads + randomWrite 3 + scans on a single RS. I'm also in the process of deploying it on a cluster.
Thanks,
Jean-Daniel
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900515#action_12900515 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Ted Yu" <te...@yahoo.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review967
-----------------------------------------------------------
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3151>
How about naming this method openRegionProlog ?
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3149>
Please add:
It has to be called inside the corresponding finally block
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3150>
How about naming this method closeRegionEpilog ?
- Ted
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900893#action_12900893 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review973
-----------------------------------------------------------
Further testing shows that I completely forgot to properly lock incrementColumnValue, thus missing the point of the title of the jira :S Uploading a new patch soon that addresses that, along with comments from the reviews.
- Jean-Daniel
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900916#action_12900916 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 507
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line507>
bq. >
bq. > I suppose this order is ok if the first thing we do on entrance to HRegion is get the read lock before check of closing.
So I just redid that part. setClosing is first taken so that when the client threads arrive they can fast fail by looking at closing.get before trying to get the readLock.
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 712
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line712>
bq. >
bq. > Seems like you could use your opentransaction/closetransaction methods here and in flush too to be consistent?
Yeah the issue with compact and flush is that the callers don't expect to see NSRE, the want null values.
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 1115
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line1115>
bq. >
bq. > Aren't these lines unnecessary? openRegionTransaction does it?
Good catch.
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 3142
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line3142>
bq. >
bq. > So, this javadoc is good but do you think we need some more doc? Does there need to be more detail on new locking regime? Maybe there is no more to be said that what is here in this paragraph. You've done all the work unravelling our lock mess. With time your nice unravelling will rot unless its clear what the pattern is. I'm just trying to think of ways of preventing that happening.
Yeah I'll included some more javadoc, maybe with code examples?
- Jean-Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review974
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900922#action_12900922 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: stack@duboce.net
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 507
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line507>
bq. >
bq. > I suppose this order is ok if the first thing we do on entrance to HRegion is get the read lock before check of closing.
bq.
bq. Jean-Daniel Cryans wrote:
bq. So I just redid that part. setClosing is first taken so that when the client threads arrive they can fast fail by looking at closing.get before trying to get the readLock.
bq.
bq. stack wrote:
bq. Don't you have to check again the setClosing after you get the read lock?
bq.
bq. Jean-Daniel Cryans wrote:
bq. I'm about to post a new patch, but it looks like this and it has to be called just before "try" instead of inside:
bq. if (this.closing.get()) {
bq. throw new NotServingRegionException(regionInfo.getRegionNameAsString() +
bq. " is closing");
bq. }
bq. lock.readLock().lock();
bq. if (this.closed.get()) {
bq. throw new NotServingRegionException(regionInfo.getRegionNameAsString() +
bq. " is closed");
bq. }
That looks right.
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 712
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line712>
bq. >
bq. > Seems like you could use your opentransaction/closetransaction methods here and in flush too to be consistent?
bq.
bq. Jean-Daniel Cryans wrote:
bq. Yeah the issue with compact and flush is that the callers don't expect to see NSRE, the want null values.
bq.
bq. stack wrote:
bq. OK. Not important. This is deep internal stuff or make a version that takes a flag on whether to throw exception (default throws exception .. might get messy though... not important).
bq.
bq. Jean-Daniel Cryans wrote:
bq. I'm afraid those little methods could be clogged fast.
Yeah. Not important.
- stack
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review974
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900951#action_12900951 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/
-----------------------------------------------------------
(Updated 2010-08-20 17:43:52.668799)
Review request for hbase.
Changes
-------
Updated patch with "closing" handling and ICV locking, still needs more javadoc.
Summary
-------
This patch removes newScannerLock and renames splitAndClose lock to just "lock". Every operation is now required to obtain the read lock on "lock" before doing anything (including getting a row lock). This is done by calling openRegionTransaction inside a try statement and by calling closeRegionTransaction in finally.
flushcache got refactored some more in order to do the locking in the proper order; first get the read lock, then do the writestate handling.
Finally, it removes the need to have a writeLock when flushing when subclassers give atomic work do to via internalPreFlushcacheCommit. This means that this patch breaks external contribs. This is required to keep our whole locking mechanism simpler.
This addresses bug HBASE-2915.
http://issues.apache.org/jira/browse/HBASE-2915
Diffs (updated)
-----
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 987355
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 987355
/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 987355
Diff: http://review.cloudera.org/r/691/diff
Testing
-------
5 concurrent ICV threads + randomWrite 3 + scans on a single RS. I'm also in the process of deploying it on a cluster.
Thanks,
Jean-Daniel
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901482#action_12901482 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
bq. On 2010-08-20 20:49:15, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 716
bq. > <http://review.cloudera.org/r/691/diff/2/?file=7680#file7680line716>
bq. >
bq. > Any reason this does not follow the pattern used elsewhere? You are not testing closing before getting the read lock?
mmm didn't want to copy the same chunk of code over there, but yeah it will also give us a fail-fast behavior which will speed up flushing.
bq. On 2010-08-20 20:49:15, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 785
bq. > <http://review.cloudera.org/r/691/diff/2/?file=7680#file7680line785>
bq. >
bq. > ditto
ditto
bq. On 2010-08-20 20:49:15, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 2960
bq. > <http://review.cloudera.org/r/691/diff/2/?file=7680#file7680line2960>
bq. >
bq. > Is this supposed to be inside the try?
Per that method's javadoc, no. The reason is that if it's all in a try block, and that closing is true, then you won't hold a readLock and you can't do a isHeldByCurrent thread on that lock. So I thought we could catch IllegalStateException in closeRegionOperation, but that would be really ugly. This leaves us to the current situation where if an exception is thrown when we are on this line:
if (this.closed.get()) {
that it would leave the readLock locked by the thread, although in this case the region is closing so we're getting rid of it anyways. Although, thinking about it, I should probably do this instead:
if (this.closed.get()) {
lock.readLock().unlock();
throw new NotServingRegionException(regionInfo.getRegionNameAsString() +
" is closed");
}
- Jean-Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review988
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900919#action_12900919 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: stack@duboce.net
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 507
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line507>
bq. >
bq. > I suppose this order is ok if the first thing we do on entrance to HRegion is get the read lock before check of closing.
bq.
bq. Jean-Daniel Cryans wrote:
bq. So I just redid that part. setClosing is first taken so that when the client threads arrive they can fast fail by looking at closing.get before trying to get the readLock.
Don't you have to check again the setClosing after you get the read lock?
bq. On 2010-08-20 15:57:34, stack wrote:
bq. > /trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 712
bq. > <http://review.cloudera.org/r/691/diff/1/?file=7612#file7612line712>
bq. >
bq. > Seems like you could use your opentransaction/closetransaction methods here and in flush too to be consistent?
bq.
bq. Jean-Daniel Cryans wrote:
bq. Yeah the issue with compact and flush is that the callers don't expect to see NSRE, the want null values.
OK. Not important. This is deep internal stuff or make a version that takes a flag on whether to throw exception (default throws exception .. might get messy though... not important).
- stack
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review974
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900917#action_12900917 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: "Jean-Daniel Cryans" <jd...@apache.org>
bq. On 2010-08-20 15:58:23, stack wrote:
bq. > oh, one other thing, in discussions, we talked of no longer needing to wait on row locks to expire... I don't see this being excised from the close method. Should that be in here?
Yep, it needs to go.
- Jean-Daniel
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review975
-----------------------------------------------------------
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900971#action_12900971 ]
HBase Review Board commented on HBASE-2915:
-------------------------------------------
Message from: stack@duboce.net
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/691/#review988
-----------------------------------------------------------
Ship it!
+1
if below are issues address on commit.
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3194>
Any reason this does not follow the pattern used elsewhere? You are not testing closing before getting the read lock?
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3195>
ditto
/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<http://review.cloudera.org/r/691/#comment3196>
Is this supposed to be inside the try?
- stack
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900367#action_12900367 ]
Jean-Daniel Cryans commented on HBASE-2915:
-------------------------------------------
There is another deadlock that needs fixing in the scope of this jira. Since the split code was redone, there's a deadlock when SplitTransaction acquires the splitAndCloses writelock while a flush is running for it. It looks like:
{noformat}
"regionserver60021.compactor" daemon prio=10 tid=0x00007fc31845b800 nid=0x5f62 in Object.wait() [0x00007fc31e9e7000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:493)
- locked <0x00007fc336877998> (a org.apache.hadoop.hbase.regionserver.HRegion$WriteState)
at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:213)
at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:186)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:157)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
"regionserver60021.cacheFlusher" daemon prio=10 tid=0x00007fc31845a000 nid=0x5f61 waiting on condition [0x00007fc31eae8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fc336561750> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:877)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:793)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:249)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:223)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:146)
{noformat}
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2915) Deadlock between HRegion.ICV and
HRegion.close
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-2915.
---------------------------------------
Hadoop Flags: [Reviewed]
Resolution: Fixed
Committed to trunk, thanks for the review Stack!
> Deadlock between HRegion.ICV and HRegion.close
> ----------------------------------------------
>
> Key: HBASE-2915
> URL: https://issues.apache.org/jira/browse/HBASE-2915
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.90.0
>
> Attachments: HBASE-2915-v4.patch
>
>
> HRegion.ICV gets a row lock then gets a newScanner lock.
> HRegion.close gets a newScanner lock, slitCloseLock and finally waits for all row locks to finish.
> If the ICV got the row lock and then close got the newScannerLock, both end up waiting on the other. This was introduced when Get became a Scan.
> Stack thinks we can get rid of the newScannerLock in close since we setClosing to true.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.