You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/07/03 19:59:47 UTC
[jira] Created: (HBASE-1609) [part of hbase-1583] We wait on leases
to expire before regionserver goes down. Rather, just let client fail
[part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
-------------------------------------------------------------------------------------------------------------
Key: HBASE-1609
URL: https://issues.apache.org/jira/browse/HBASE-1609
Project: Hadoop HBase
Issue Type: Improvement
Reporter: stack
Fix For: 0.20.0
Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727263#action_12727263 ]
ryan rawson commented on HBASE-1609:
------------------------------------
We should shut down a regionserver (when asked via sigkill or whatever) by first waiting for existing clients to finish their current operation, then refusing new connections, then shutting down. By shutting down faster we can reassign regions faster during a rolling restart or other partial shutdown scenario.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727110#action_12727110 ]
Jean-Daniel Cryans commented on HBASE-1609:
-------------------------------------------
Yes, and the client can just ask ZK to know if the cluster is really going down... could even set a watch on it (ZKW.setClusterStateWatch)
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1609) [part of hbase-1583] We wait on leases
to expire before regionserver goes down. Rather, just let client fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1609:
-------------------------
Attachment: 1609-v2.patch
This patch cancels leases when next is called and we're going down.
It also just closes all leases in HRS before we run close of all regions.
I don't see why we have to pussy-foot around with Scanners. They are reading. If they get exception in middle of reading, then no big deal. It looks like Scanners will get UnknownScannerException if cluster is shutting down on them. I added javadoc saying this could be one of the reasons for an USE.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1609) [part of hbase-1583] We wait on leases
to expire before regionserver goes down. Rather, just let client fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1609:
-------------------------
Attachment: 1609.patch
Here is a patch for the serverside. It removes all the fancy scanner counting we used do.
In testing, serverside logs seem fine when outstanding scanner..... scanners start to get notserveringregionexception as the regions go down.
Client-side though goes crazy.... retries of zk and retries of scanner stuff. Need to clean up client handling of this case before can commit, i'd suggest.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727262#action_12727262 ]
Dave Latham commented on HBASE-1609:
------------------------------------
+1 for this on 0.19
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-1609:
----------------------------
Assignee: stack
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1609.
--------------------------
Resolution: Fixed
Release Note: No more trying to be sweet to clients. If admin wants cluster to go down, we go down and let the clients each EOFExceptions (because RPC was cutoff midways) and failed connect type exceptions. Puts take out a row lock so should make it in before their Region closes (though if part of a batch, all in batch are not guaranteed to make it in)
Hadoop Flags: [Reviewed]
Committed to branch and trunk.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732259#action_12732259 ]
stack commented on HBASE-1609:
------------------------------
Testing...
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732264#action_12732264 ]
Jean-Daniel Cryans commented on HBASE-1609:
-------------------------------------------
+1 on patch. Good idea adding this to the javadoc.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727105#action_12727105 ]
stack commented on HBASE-1609:
------------------------------
You mean don't do anything fancy serverside? (I'll sign that contract)
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727275#action_12727275 ]
stack commented on HBASE-1609:
------------------------------
I see that in HRegionServer#next, we do a checkOpen and will do the below:
{code}
protected void checkOpen() throws IOException {
if (this.stopRequested.get() || this.abortRequested) {
throw new IOException("Server not running" +
(this.abortRequested? ", aborting": ""));
}
if (!fsOk) {
throw new IOException("File system not available");
}
}
{code}
... sending an IOE back to the scanner -- but we're not cancelling its lease. Cancelling lease would help some at least in case where Scanners are active.
But for inactives scanners -- e.g. a killed client -- we'd still be waiting around.
Regards Ryan's refusing new requests but finishing the outstanding, a get takes out a row lock which also takes a splitsAndClosesLock.readLock().lock();.... so you can't close region while a get is going on (it needs to get the write lock here).
Individual put does the same.
A scanner next does not take out the splitsAndCloses lock. I suppose it never had too because scanner can ride over splits and we'd never close while outstanding lease.
Lets make it so next takes at least a close lock. Then in actual shutdown, cancel all leases and not wait on any. This should make it so at least the current scanner next completes but all subsequent next's will be rejected.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732334#action_12732334 ]
stack commented on HBASE-1609:
------------------------------
Testing puts, I see the below in client when we shut down in middle of an upload:
{code}
2009-07-17 04:10:04,645 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=0 of max=10, waiting=2000ms
2009-07-17 04:10:06,904 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=1 of max=10, waiting=2000ms
2009-07-17 04:10:09,015 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=2 of max=10, waiting=2000ms
2009-07-17 04:10:11,068 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=3 of max=10, waiting=4000ms
2009-07-17 04:10:15,107 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=4 of max=10, waiting=4000ms
2009-07-17 04:10:19,216 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=5 of max=10, waiting=8000ms
2009-07-17 04:10:27,490 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=6 of max=10, waiting=8000ms
2009-07-17 04:10:35,534 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=7 of max=10, waiting=16000ms
2009-07-17 04:10:52,446 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Reloading region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 location because regionserver didn't accept updates; tries=8 of max=10, waiting=32000ms
2009-07-17 04:11:24,514 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server, retryOnlyOne=true, index=0, islastrow=false, tries=9, numtries=10, i=0, listsize=8643, location=address: X.X.X.141:60020, regioninfo: REGION => {NAME => 'TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498', STARTKEY => '\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03', ENDKEY => '\x00\x03\x02\x04\x06\x07\x07\x00\x06\x06', ENCODED => 1615573, TABLE => {{NAME => 'TestTable', FAMILIES => [{NAME => 'info', COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, region=TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498 for region TestTable,\x00\x03\x02\x04\x05\x07\x09\x04\x00\x03,1247725178498, row '\x00\x03\x02\x04\x06\x03\x09\x01\x02\x06', but failed after 10 attempts.
{code}
I think I see connection refused too.
That ain't bad I'd say.
This is with zk not managed by hbase. If I shut down a cluster where hbase is managing the zk quorum -- i.e. its shutdown as part of hbase shutdown -- then I see client log filled with zk complaints with above intermixed.
Scanning, I see EOFException because server went down returning result it looks like.
Exceptions ain't pretty but I don't see anything inherently wrong. Will go ahead and commit.
With this new commit, our new philosophy is no more trying to be mr. nice guy regards clients if admin wants cluster to go down.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732284#action_12732284 ]
Jonathan Gray commented on HBASE-1609:
--------------------------------------
+1 as well. Patch looks good, like the javadoc, we get lots of users asking what those exceptions mean.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1609-v2.patch, 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1609) [part of hbase-1583] We wait on
leases to expire before regionserver goes down. Rather, just let client
fail
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727070#action_12727070 ]
Jean-Daniel Cryans commented on HBASE-1609:
-------------------------------------------
I might also add that even if a client is still doing it's stuff with a lease opened, some other regions will already be down so you get an inconsistent view of the state of the cluster. Let's have a contract here: if the cluster goes down, don't try anything fancy.
> [part of hbase-1583] We wait on leases to expire before regionserver goes down. Rather, just let client fail
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1609
> URL: https://issues.apache.org/jira/browse/HBASE-1609
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1609.patch
>
>
> Addressing this issue will help hbase-1583. We should do for 0.20.0 and perhaps for 0.19.x even.
> Currently, if outstanding leases, in HRegion close, we'll hang until lease expires. Could be a minute. Could be worse, the client might come in and renew the lease a few times at least till it finishes out the region. This gets in way of regionserver shutting down fast.
> J-D suggests that regionserver should just go down and outstanding clients should fail rather than try and be nice to outstanding clients (in his case, his MR job had failed so no clients... but we insist on lease expiring).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.