You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2007/12/17 19:02:43 UTC
[jira] Created: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
[hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
--------------------------------------------------------------------------------
Key: HADOOP-2443
URL: https://issues.apache.org/jira/browse/HADOOP-2443
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Reporter: stack
Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reopened HADOOP-2443:
-----------------------------------
Now that this works and has been committed, can we reduce the 'chattiness' of the debug level logging?
Thanks.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552523 ]
stack commented on HADOOP-2443:
-------------------------------
Supposition is that currently the client will not work if it does not have all regions for a table. If so, client should work even in the case where a region is completely missing serving all regions but the absent.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury resolved HADOOP-2443.
-----------------------------------
Resolution: Fixed
Resolving issue. I have opened new issues for dealing with the problems mentioned in comments above. See HADOOP-2598 and HADOOP-2599.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Patch Available (was: In Progress)
Trying on hudson.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Open (was: Patch Available)
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558594#action_12558594 ]
Hudson commented on HADOOP-2443:
--------------------------------
Integrated in Hadoop-Nightly #365 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/365/])
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12552529 ]
Bryan Duxbury commented on HADOOP-2443:
---------------------------------------
So, this issue indicates that clients keep a dense list of regions and locations. Is there a reason why they should keep a dense list instead of a sparse one? In production clusters, there could be hundreds or thousands of regions, and caching them all seems to be pretty inefficient.
Instead, couldn't we just cache the regions we've found through trying to use them? That is, when you're going to do some operation on a key, you ask the master/META regionserver to resolve where your key belongs, and just remember the location and bounds of that region. That way, you first check the cache to see if you already know where a key lives, and if it is a hit, you go out and talk to the region server. If the server hasn't got that region anymore, you get an NSRE, invalidate just the single cache item you used to get there. Isn't this how Bigtable does it?
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558482#action_12558482 ]
Hadoop QA commented on HADOOP-2443:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373068/2443-v10.patch
against trunk revision r611681.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1575/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1575/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1575/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1575/console
This message is automatically generated.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v8.patch
Fixed javadoc warning.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559759#action_12559759 ]
Bryan Duxbury commented on HADOOP-2443:
---------------------------------------
I'm not sure I understand your logic. if I replaced line 231 with a continue, it would just continue the for loop that's iterating over the columns of the current result we pulled back. The reason that we break all the way out of the scanner iteration loop is because it is impossible for this scanner to ever yield another HRegionInfo that might match the current table. Continuing would only waste time traversing the rest of the scanner.
This did, however, bring to my attention that if there are multiple .META. regions, and the table we're searching for is in the first .META. region, this code will actually search at least the first row of the second region too. So in reality, the break SCANNER_LOOP should be break REGION_LOOP. Does that make sense?
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v7.patch
Another version, more cleaned up.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v9.patch
Merged in Jim's patch in this version.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Patch Available (was: Open)
Trying Hudson.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Patch Available (was: Open)
Failure was in TestTableIndex in a place I'd never seen problems before. It was down in Flusher. I'm going to resubmit it and see if it passes again.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v3.patch
Not quite done yet - a few tests don't pass, and there is no caching at all right now.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Open (was: Patch Available)
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v5.patch
The latest version of the patch removes getTableServers and related methods entirely and switches HTable$ClientScanner to use getRegionLocation.
Unfortunately, I now have TestTableIndex and TestTableMapReduce failing all the time. The error seem to be related to scanners acting strangely with rows in the wrong regions, but I can't be sure of what's going on yet.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558358#action_12558358 ]
Hadoop QA commented on HADOOP-2443:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373038/2443-v8.patch
against trunk revision r611537.
@author +1. The patch does not contain any @author tags.
patch -1. The patch command could not apply the patch.
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1565/console
This message is automatically generated.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v4.patch
All tests are passing locally. This patch includes caching, so performance should be comparable to what you see with the other region location lookup approach.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury reassigned HADOOP-2443:
-------------------------------------
Assignee: Bryan Duxbury
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-2443) [hbase] Keep lazy cache of
regions in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-2443 started by Bryan Duxbury.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Patch Available (was: Open)
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Patch Available (was: Open)
Try the latest again.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Open (was: Patch Available)
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2443:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed. Had some minor questions -- see below -- but it ain't enough to hold up patch and I'm afraid patch would rot if left hang (It touches lots of classes). Resolving. Thanks for the patch Bryan.
Here's the couple of items:
Here in HConnectionManager:
{code}
+ scannerId = server.openScanner(
+ metaLocation.getRegionInfo().getRegionName(),
+ COLUMN_FAMILY_ARRAY, EMPTY_START_ROW, LATEST_TIMESTAMP,
+ null);
{code}
... EMPTY_START_ROW will work but 'startRow' would be easier to read (It'd work too -- right?).
Here:
{code}
+ } while (startRow.compareTo(EMPTY_START_ROW) != 0);
{code}
.. EMPTY_ROW or a new define, LAST_ROW, would have been more readable, don't you think?
This 999... 'trick' should be done as a method -- in HTableInfo? -- since you do it twice:
{code}
+ Text tableKey = new Text(tableName.toString() + ",,99999999999999");
{code}
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Status: Open (was: Patch Available)
Tests failed by TestGet2 timing out. No idea what happened there, so I'll resubmit.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury resolved HADOOP-2443.
-----------------------------------
Resolution: Fixed
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559769#action_12559769 ]
Bryan Duxbury commented on HADOOP-2443:
---------------------------------------
You know what, I think you're right. I must have only tested it in scenarios when there are single tables.
I've opened a new issue to track this problem, HADOOP-2631. I am resolving this issue again, because it's already been committed.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12555151 ]
Bryan Duxbury commented on HADOOP-2443:
---------------------------------------
I think I have a pretty good strategy in mind, but I've sort of hit a snag.
The problem is that when you're trying to determine which region a key lives in, the list of regions is keyed by the region's start key, and you're looking for the region that is closest to (without exceeding) the search key. That is, if you have regions [0, 10, 20] and you're trying to figure out where key 15 lives, you want the answer to be 10. Then data is all there for this sort of thing, but the interface of MapFile, which is where the .META. table is ultimately stored, doesn't have anything like this.
What we need essentially is a method just like getClosest, but that returns the last key found after the one the goes over the search key. The alternative is doing a linear search of some sort manually, which would be pretty inefficient.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clint Morgan reopened HADOOP-2443:
----------------------------------
Looks to me like this introduced a bug in HTable.getStartKeys():
Line 231 breaks from the while loop if the HRegionInfo is not for this.tableName. Instead it should continue.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558300#action_12558300 ]
stack commented on HADOOP-2443:
-------------------------------
Patch looks great. Patch generates a javadoc warning... will fail on hudson: Run 'ant javadoc' to see. +1 after you fix the javadoc warning.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Clint Morgan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12559768#action_12559768 ]
Clint Morgan commented on HADOOP-2443:
--------------------------------------
Well, I don't fully understand the code, but had a problem where HTable.getStartKeys was returning empty array.
I stepped through the execution and the first HRegionInfo it got was not for this table, so it would hit that break out of the loop, and so return empty array.
I changed break to continue, then it hits this.tableName, and correctly returns my startKey.
"The reason that we break all the way out of the scanner iteration loop is because it is impossible for this scanner to ever yield another HRegionInfo that might match the current table."
I don't understand this logic, but as I said I don't fully understand the code. Its seems to me you just break on the first HRegionInfo, and there is no reason to assume the one we want is not latter in the scanner...
The way I read it, this code only works if the *first* HRegionInfo you get is for this.tableName.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Fix Version/s: 0.16.0
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v6.patch
All the tests pass locally at this point, big improvement over the last version.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558327#action_12558327 ]
Hadoop QA commented on HADOOP-2443:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373038/2443-v8.patch
against trunk revision r611488.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1564/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1564/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1564/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1564/console
This message is automatically generated.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2443) [hbase] Keep lazy cache of regions in
client rather than an 'authoritative' list
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HADOOP-2443:
----------------------------------
Attachment: 2443-v10.patch
Ok, here's another go. The TestGet2 failure I had previously was actually correct, as I wasn't shutting down the test correctly. I fixed that, and in the process, merged in the changes from other recent patches.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v10.patch, 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558454#action_12558454 ]
Hadoop QA commented on HADOOP-2443:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373058/2443-v9.patch
against trunk revision r611553.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1570/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1570/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1570/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1570/console
This message is automatically generated.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2443) [hbase] Keep lazy cache of regions
in client rather than an 'authoritative' list
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558466#action_12558466 ]
Hadoop QA commented on HADOOP-2443:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373058/2443-v9.patch
against trunk revision r611629.
@author +1. The patch does not contain any @author tags.
patch -1. The patch command could not apply the patch.
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1573/console
This message is automatically generated.
> [hbase] Keep lazy cache of regions in client rather than an 'authoritative' list
> --------------------------------------------------------------------------------
>
> Key: HADOOP-2443
> URL: https://issues.apache.org/jira/browse/HADOOP-2443
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: Bryan Duxbury
> Fix For: 0.16.0
>
> Attachments: 2443-v3.patch, 2443-v4.patch, 2443-v5.patch, 2443-v6.patch, 2443-v7.patch, 2443-v8.patch, 2443-v9.patch
>
>
> Currently, when the client gets a NotServingRegionException -- usually because its in middle of being split or there has been a regionserver crash and region is being moved elsewhere -- the client does a complete refresh of its cache of region locations for a table.
> Chatting with Jim about a Paul Saab upload issue from Saturday night, when tables are big comprised of regions that are splitting fast (because of bulk upload), its unlikely a client will ever be able to obtain a stable list of all region locations. Given that any update or scan requires that the list of all regions be in place before it proceeds, this can get in the way of the client succeeding when the cluster is under load.
> Chatting, we figure that it better the client holds a lazy region cache: on NSRE, figure out where that region has gone only and update the client-side cache for that entry only rather than throw out all we know of a table every time.
> Hopefully this will fix the issue PS was experiencing where during intense upload, he was unable to get/scan/hql the same table.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.