You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "dhruba borthakur (Created) (JIRA)" <ji...@apache.org> on 2012/03/10 02:34:57 UTC
[jira] [Created] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Avoid RegionServer GC caused by timed-out calls
-----------------------------------------------
Key: HBASE-5560
URL: https://issues.apache.org/jira/browse/HBASE-5560
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235372#comment-13235372 ]
Hudson commented on HBASE-5560:
-------------------------------
Integrated in HBase-TRUNK-security #145 (See [https://builds.apache.org/job/HBase-TRUNK-security/145/])
HBASE-5560 Avoid RegionServer GC caused by timed-out calls (Revision 1303512)
Result = SUCCESS
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227297#comment-13227297 ]
Lars Hofhansl commented on HBASE-5560:
--------------------------------------
easiest is to just leave the 15m default
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227311#comment-13227311 ]
dhruba borthakur commented on HBASE-5560:
-----------------------------------------
The problem with the 5 min default is that it gives a bad-user experience out-of-the-box. If people are ok with it, I would prefer to change it to 90 seconds as Stack proposed. Is that ok with you lars? I know it breaks backward compatibility, but I will update the release-notes in the JIRA
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227015#comment-13227015 ]
dhruba borthakur commented on HBASE-5560:
-----------------------------------------
Lars is asking to change the default value to 5 min.
Stack is asking the default value to 90 sec.
any consensus?
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5560:
---------------------------------
Fix Version/s: 0.96.0
0.94.0
Let's get this into 0.94 as well.
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5560:
-------------------------------
Attachment: D2241.1.patch
dhruba requested code review of "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
Reviewers: stack, tedyu, sc, JIRA
A slow client is not consuming rpc responses from the client. But the server caches the call responses uptil 15 minutes (hardcoded). This caused the regionserver to run out of old-gen and trigger a full GC.
This patch makes the 15 minute value to be settable by a configurable parameter "ipc.client.call.purge.timeout". The default is still 15 minutes to maintain backward compatibility.
TEST PLAN
Run all unit tests
REVISION DETAIL
https://reviews.facebook.net/D2241
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4923/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: D2241.1.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226952#comment-13226952 ]
Phabricator commented on HBASE-5560:
------------------------------------
stack has commented on the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
Excellent lads. Good one. +1 . And +1 on 90seconds rather than 15 minutes (though this patch seems to have 15minutes still in spite of your comment Dhruba).
Reminds me of another ugly one; the queue of incoming rpc requests which was hard-coded at 100 outstanding requests * the number of handlers (So, if batch puts, could be 2MB*100*100 of unaccounted for memory when server was working hard).
REVISION DETAIL
https://reviews.facebook.net/D2241
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272569#comment-13272569 ]
Phabricator commented on HBASE-5560:
------------------------------------
dhruba has closed the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
REVISION DETAIL
https://reviews.facebook.net/D2241
To: stack, tedyu, sc, JIRA, lhofhansl, dhruba
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227319#comment-13227319 ]
stack commented on HBASE-5560:
------------------------------
Pardon me lads. Did not mean to confuse. Lars I was telling a story. It used to be a queue of 100 times the number of handlers. After we figured that, it was changed to be configurable and ten by default. On 15 minutes, that seems way too long. I'm good w/ 5 minutes but would prefer 90 seconds (which is 1.5 times the rpcTimeout -- the client won't be there after 60 seconds pass... should we hold on for some factor times rpcTimeout?).
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226732#comment-13226732 ]
Lars Hofhansl commented on HBASE-5560:
--------------------------------------
+1 on patch.
Isn't 15mins awfully long? Should we lower the default to (say) 5mins?
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226740#comment-13226740 ]
Phabricator commented on HBASE-5560:
------------------------------------
sc has commented on the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
+1
Only have one minor comment.
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java:1461 Can you put
15 * 60 * 1000L
to make it easier to read?
REVISION DETAIL
https://reviews.facebook.net/D2241
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "stack (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-5560.
--------------------------
Resolution: Fixed
Fix Version/s: (was: 0.96.0)
0.92.2
Hadoop Flags: Reviewed
Applied trunk, 0.92 and 0.94 branches. Thanks for patch Dhruba.
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234981#comment-13234981 ]
Hudson commented on HBASE-5560:
-------------------------------
Integrated in HBase-TRUNK #2691 (See [https://builds.apache.org/job/HBase-TRUNK/2691/])
HBASE-5560 Avoid RegionServer GC caused by timed-out calls (Revision 1303512)
Result = SUCCESS
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234882#comment-13234882 ]
Lars Hofhansl commented on HBASE-5560:
--------------------------------------
This just needs to be committed, no?
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235051#comment-13235051 ]
Hudson commented on HBASE-5560:
-------------------------------
Integrated in HBase-0.94 #45 (See [https://builds.apache.org/job/HBase-0.94/45/])
HBASE-5560 Avoid RegionServer GC caused by timed-out calls (Revision 1303513)
Result = SUCCESS
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227322#comment-13227322 ]
Lars Hofhansl commented on HBASE-5560:
--------------------------------------
Heh...
+1 on 90s timeout.
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229862#comment-13229862 ]
Phabricator commented on HBASE-5560:
------------------------------------
lhofhansl has accepted the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
lgtm
REVISION DETAIL
https://reviews.facebook.net/D2241
BRANCH
svn
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235113#comment-13235113 ]
Hudson commented on HBASE-5560:
-------------------------------
Integrated in HBase-0.92 #333 (See [https://builds.apache.org/job/HBase-0.92/333/])
HBASE-5560 Avoid RegionServer GC caused by timed-out calls (Revision 1303515)
HBASE-5560 Avoid RegionServer GC caused by timed-out calls (Revision 1303514)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226963#comment-13226963 ]
Lars Hofhansl commented on HBASE-5560:
--------------------------------------
I think Dhruba meant 15 * 60 * 1000 and 900000 :)
Should we file a separate issue for the 2nd issue you mentioned, seems bad, and would be good to get into a performance release.
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226980#comment-13226980 ]
Lars Hofhansl commented on HBASE-5560:
--------------------------------------
@Stack: The ipc queue size is already configurable (ipc.server.max.queue.size) and defaults to 10. HandlerCount also defaults to 10. So should be OK? Or did you want to put a limit on the total memory used by the queue?
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5560:
-------------------------------
Attachment: D2241.2.patch
dhruba updated the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
Reviewers: stack, tedyu, sc, JIRA
Changed the constant to 15 * 6 * 1000 (insted of 9000).
I did not change it to 5 minutes because of backward compatibility reasons.
Lars: if u fel strongly that we should change it to 5 min, please let me know
and I will make the change.
REVISION DETAIL
https://reviews.facebook.net/D2241
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227803#comment-13227803 ]
stack commented on HBASE-5560:
------------------------------
@Dhruba
In trunk:
{code}
src/main/java/org/apache/hadoop/hbase/HConstants.java: public static int DEFAULT_HBASE_RPC_TIMEOUT = 60000;
...
In HConnectionManager.....
this.rpcTimeout = conf.getInt(
HConstants.HBASE_RPC_TIMEOUT_KEY,
HConstants.DEFAULT_HBASE_RPC_TIMEOUT);
{code}
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5560:
---------------------------------
Fix Version/s: 0.96.0
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.92.2, 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "dhruba borthakur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227323#comment-13227323 ]
dhruba borthakur commented on HBASE-5560:
-----------------------------------------
The default rpc client timeout is Max_Value, see HConstants.DEFAULT_HBASE_CLIENT_OPERATION_TIMEOUT. where do you see that rpcTimeout is ti be 60 seconds?
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5560:
-------------------------------
Attachment: D2241.4.patch
dhruba updated the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
Reviewers: stack, tedyu, sc, JIRA
Changed the default timeout to be 2 times the HConstants.DEFAULT_HBASE_RPC_TIMEOUT
REVISION DETAIL
https://reviews.facebook.net/D2241
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch, D2241.4.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5560) Avoid RegionServer GC caused by
timed-out calls
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HBASE-5560:
-------------------------------
Attachment: D2241.3.patch
dhruba updated the revision "[jira][HBASE-5560] Avoid RegionServer GC caused by timed-out calls".
Reviewers: stack, tedyu, sc, JIRA
Changed default purge-timeout to 90 seconds
REVISION DETAIL
https://reviews.facebook.net/D2241
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
> Avoid RegionServer GC caused by timed-out calls
> -----------------------------------------------
>
> Key: HBASE-5560
> URL: https://issues.apache.org/jira/browse/HBASE-5560
> Project: HBase
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.94.0, 0.96.0
>
> Attachments: D2241.1.patch, D2241.2.patch, D2241.3.patch
>
>
> The HBaseRpcServer queues up rpc responses if the socket connection to the client is not yet ready to receive data. Calls are queued here until a 15 minute timeout occurs. I am able to generate a full GC when I artificially make a client read rpc-responses very slowly. This jira is to make this 15 minute time configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira