You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2009/02/05 18:57:59 UTC
[jira] Created: (HBASE-1186) Memory-aware Maps with LRU eviction
Memory-aware Maps with LRU eviction
-----------------------------------
Key: HBASE-1186
URL: https://issues.apache.org/jira/browse/HBASE-1186
Project: Hadoop HBase
Issue Type: New Feature
Reporter: Jonathan Gray
Priority: Critical
Fix For: 0.20.0
Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
I propose two initial classes: LruHashMap and LruBlockMap
*LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
- Memory-aware
- Fixed size
- LRU eviction
*LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
- Memory-aware
- Fully resizable
- LRU eviction (with some additions)
- High priority blocks
- _Optional: Scan resistant algorithm_
Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: hbase-1186-v2.patch
A few small changes. Now includes two functions for debugging/unit test purposes. One returns a sorted List of entries in the LRU in order of their access time (oldest/first to evict is first). The other returns a Set of all elements found in the hash table.
The two of these data structures can be used to ensure the LRU is being updated properly and that the linked list does not become out of sync with the hash table.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: hbase-1186-v2.patch, HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: (was: LruHashMap.java)
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: hbase-1186-v3.patch
Working version. Fixes a few small bugs.
This patch does not contain the unit test. Will post another patch when that's cleaned up.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: hbase-1186-v2.patch, hbase-1186-v3.patch, HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: LruHashMap.java
Stripped down, partially tested.
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: LruHashMap.java
Apache-friendly. Thread-safe. Heavily commented as this will serve as the base to other memory-aware structures.
Still partially untested but would like a general code review before moving forward.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Status: Patch Available (was: Open)
Tested and ready for review/commit.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: hbase-1186-v2.patch, hbase-1186-v3.patch, hbase-1186-v4.patch, HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1186) Memory-aware Maps with LRU eviction
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671521#action_12671521 ]
stack commented on HBASE-1186:
------------------------------
You think we should go to bother of trying to elicit if 32-bit vs. 64-bit? (There is probably a canonical way but perhaps looking at bytes in an Integer would be a flag)? When HeapSize is classloaded, it could run static methods to figure 32-bit vs. 64-bit and then adjust the defines in this class accordingly? Then have methods on HeapSize that implementers call to size themselves for the different architectures? Maybe thats something for later, for the embedded hbase (smile)? Lets just go w/ 64-bit for now.
Let me know if you want me to run some tests on profiler to check your sizings.
For next version of LruHashMap, fix spacing -- its two spaces by default and no tabs -- and add apache license.
Why do you have transient datamembers? Are your Maps meant to be (java) serializable?
Make most of the data members private rather than default access?
So, maxMemUsage is passed in? It'll be some percentage of the heap?
We're not soft referencing it because of your testing showing soft references are not LRU?
It'll be really interesting watching this thing at core of a concurrent app. There are no synchronizes anywhere. Whats the plan for that?
Good stuff
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1186:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Passes all local tests, including the new test case addition for this feature.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: hbase-1186-v2.patch, hbase-1186-v3.patch, hbase-1186-v4.patch, HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1186) Memory-aware Maps with LRU eviction
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671535#action_12671535 ]
Jonathan Gray commented on HBASE-1186:
--------------------------------------
I didn't have time to finish work on this yesterday. I just posted up what I had finished before I left, it's definitely not done yet.
Will properly format it, etc well before it's ready to be included.
Re: 32-bit vs 64-bit there are some tests that can be used. My current plan is to go forward with assuming 64bit and in the future we can easily add something to detect like you said. It would then just change the size of REFERENCE which would bubble up into anything implementing HeapSize which should be using that static to compute its size.
There is some stuff left over from other code in there (transient datamembers and such) that I will be taking out. I see no need for serialization. Same goes for making things private, have been working on the lru and heapsize elements and not so much the rest of the code. Will clean it up next week.
Currently, maxMemUsage is passed in. This LRU is intended for cell cache which I think should be a per-regionserver setting. This can be adapted to work for the block cache as well in any number of ways. There are a few reasons I don't think softrefs are the way to go for block caching, definitely not the way to go for cell caching. Next week I'll put together the proposal for why I think we should use our own LRU mechanism rather than relying on softrefs across the board. The primary issues are how this will interact with existing memcache/heap monitoring, unpredictability of softref eviction, and the ability to implement in-memory tables very simply if we have our own lru mechanism (implemented in the way described in the bigtable paper).
Same goes for synchronization. This is an early version not ready for prime time. Not sure what my plan is yet will post thoughts here. Will be working on this early in the week.
Thanks for the review stack.
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688063#action_12688063 ]
Andrew Purtell commented on HBASE-1186:
---------------------------------------
I've reviewed this and except for a minor collision with the patch now it looks good. I'm just waiting for HBASE-1274 to be resolved first so I can claim that all local tests pass before committing this.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: hbase-1186-v2.patch, hbase-1186-v3.patch, hbase-1186-v4.patch, HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1186) Memory-aware Maps with LRU eviction
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-1186:
------------------------------------
Assignee: Erik Holstad
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: HeapSize.java
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Attachment: hbase-1186-v4.patch
Contains unit test and changes to HeapSize interface.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: hbase-1186-v2.patch, hbase-1186-v3.patch, hbase-1186-v4.patch, HeapSize.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Erik Holstad updated HBASE-1186:
--------------------------------
Attachment: LruHashMap.java
Adding the code for the current implementation of the LruHashMap.
Same code as HBASE-80 right now.
> Memory-aware Maps with LRU eviction
> -----------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Erik Holstad
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1186) Memory-aware Maps with LRU eviction
for Cell Cache
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1186:
---------------------------------
Assignee: Jonathan Gray (was: Erik Holstad)
Summary: Memory-aware Maps with LRU eviction for Cell Cache (was: Memory-aware Maps with LRU eviction)
There's a discussion that needs to happen around how we'll implement block caching. I'm going to create a new issue for that, this one will be about the LruHashMap used for the cell cache.
> Memory-aware Maps with LRU eviction for Cell Cache
> --------------------------------------------------
>
> Key: HBASE-1186
> URL: https://issues.apache.org/jira/browse/HBASE-1186
> Project: Hadoop HBase
> Issue Type: New Feature
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.20.0
>
> Attachments: HeapSize.java, LruHashMap.java, LruHashMap.java
>
>
> Caching is key for 0.20. We need a set of memory-aware data structures to manage our caches.
> I propose two initial classes: LruHashMap and LruBlockMap
> *LruHashMap* is currently being used over in HBASE-80 for the Cell cache. Erik Holstad has done extensive testing and benchmarking and will post results over in this issue.
> - Memory-aware
> - Fixed size
> - LRU eviction
> *LruBlockMap* can be used for the block caching of the new file format in HBASE-61. It should try to use all available memory, but must contend with Memcaches so is resizable to deal with heap pressure. Adding high priority blocks (evicted last) gives us in-memory functionality as described in bigtable paper.
> - Memory-aware
> - Fully resizable
> - LRU eviction (with some additions)
> - High priority blocks
> - _Optional: Scan resistant algorithm_
> Part of this issue is also solving how we will determine the size of cached objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.