You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jb Lee (JIRA)" <ji...@apache.org> on 2008/07/07 04:19:31 UTC
[jira] Created: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
TableServers's cachedRegionLocation doesn't have size limit.
------------------------------------------------------------
Key: HBASE-723
URL: https://issues.apache.org/jira/browse/HBASE-723
Project: Hadoop HBase
Issue Type: Bug
Components: client
Affects Versions: 0.1.3
Environment: hbase 0.3.0
Reporter: Jb Lee
Priority: Minor
cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611365#action_12611365 ]
stack commented on HBASE-723:
-----------------------------
Jb Lee:
Thanks for filing the issue.
Its correct to mark it minor I think. Here's why. This is the declaration for cachedRegionLocation:
{code}
private final Map<Integer, SoftSortedMap<byte [], HRegionLocation>>
cachedRegionLocations = ....
{code}
Its a map of integer keys to a value that is a map of soft references. If pressure on memory in the client, while the keys are hard references, the map of the much bigger soft references will release its hold on region locations.
It could be a problem when we hit extremely large tables.
Andrew: Good find. Make another issue for it?
> TableServers's cachedRegionLocation doesn't have size limit.
> ------------------------------------------------------------
>
> Key: HBASE-723
> URL: https://issues.apache.org/jira/browse/HBASE-723
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.1.3
> Environment: hbase 0.3.0
> Reporter: Jb Lee
> Assignee: Andrew Purtell
> Priority: Minor
>
> cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
Posted by "Jb Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611553#action_12611553 ]
Jb Lee commented on HBASE-723:
------------------------------
stack:
I confused with old source file (0.1.3?), where the cachedRegionLocations are declared as Map<Text, SortedMap<Text, HRegionLocation>>. Sorry.
I agree with you. With soft references, it may not be the problem.
I checked another cache in hbase. HBASE-288 added block cache implementation in HStoreFile using BlockFSInputStream.
"Map<Long, byte[]> blocks" stores blocks, but I can't find the part that remove it before the BlockFSInputStream object is GCed.
I think it can be significant problem if we use block cached table because the block cache is in regionserver and block data is not so small.
Do we need to fix it?
Andrew Purtell:
Hypothetically. Sorry again.
> TableServers's cachedRegionLocation doesn't have size limit.
> ------------------------------------------------------------
>
> Key: HBASE-723
> URL: https://issues.apache.org/jira/browse/HBASE-723
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.1.3
> Environment: hbase 0.3.0
> Reporter: Jb Lee
> Assignee: Andrew Purtell
> Priority: Minor
>
> cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610879#action_12610879 ]
Andrew Purtell commented on HBASE-723:
--------------------------------------
Also while on the subject, there is no means for other parts of the client API to invalidate cache entries on demand. While writing a testcase for HBASE-62, I observed that table metadata is cached as part of the cached region information and this is not invalidated by disable/enable table, so to get up to date metadata the client has to use a scanner over .META. directly using the meta visitor. Ideally my client-side support for metadata updates (HBaseAdmin.updateTableMeta(byte[] table, HTableDescriptor htd)) could call into this cache to invalidate entries related to the table, so then the next HTable.getTableDescriptor() would go to meta to return up to date information.
> TableServers's cachedRegionLocation doesn't have size limit.
> ------------------------------------------------------------
>
> Key: HBASE-723
> URL: https://issues.apache.org/jira/browse/HBASE-723
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.1.3
> Environment: hbase 0.3.0
> Reporter: Jb Lee
> Priority: Minor
>
> cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-723:
------------------------------------
Assignee: Andrew Purtell
> TableServers's cachedRegionLocation doesn't have size limit.
> ------------------------------------------------------------
>
> Key: HBASE-723
> URL: https://issues.apache.org/jira/browse/HBASE-723
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.1.3
> Environment: hbase 0.3.0
> Reporter: Jb Lee
> Assignee: Andrew Purtell
> Priority: Minor
>
> cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611494#action_12611494 ]
Andrew Purtell commented on HBASE-723:
--------------------------------------
Opened new JIRA as HBASE-729.
Regarding this issue, JB -- Did your client actually die with an OOME, or were you speaking hypothetically?
> TableServers's cachedRegionLocation doesn't have size limit.
> ------------------------------------------------------------
>
> Key: HBASE-723
> URL: https://issues.apache.org/jira/browse/HBASE-723
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.1.3
> Environment: hbase 0.3.0
> Reporter: Jb Lee
> Assignee: Andrew Purtell
> Priority: Minor
>
> cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-723) TableServers's cachedRegionLocation
doesn't have size limit.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-723.
----------------------------------
Resolution: Won't Fix
For this specific issue a limit on TableServers cachedRegionLocation as stack says is probably not necessary because a soft map is used.
> TableServers's cachedRegionLocation doesn't have size limit.
> ------------------------------------------------------------
>
> Key: HBASE-723
> URL: https://issues.apache.org/jira/browse/HBASE-723
> Project: Hadoop HBase
> Issue Type: Bug
> Components: client
> Affects Versions: 0.1.3
> Environment: hbase 0.3.0
> Reporter: Jb Lee
> Assignee: Andrew Purtell
> Priority: Minor
>
> cachedRegionLocation stores region locations of tables whenever new region is looked up. However, the enties are deleted only when TableServers object is closed or locateRegion is called with false useCache argument. Therefore, it seems to grow without limit and cause out of memory exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.