You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2008/11/23 23:41:44 UTC
[jira] Created: (HBASE-1024) Rearchitect regionserver I/O
Rearchitect regionserver I/O
----------------------------
Key: HBASE-1024
URL: https://issues.apache.org/jira/browse/HBASE-1024
Project: Hadoop HBase
Issue Type: Improvement
Components: io, regionserver
Reporter: Andrew Purtell
Consider supporting:
* 2GB store files
* 1TB per node (500 store files)
* Cell values up to ~100MB
* Typical use case of RS running with 1GB of heap only
Some ideas:
* Drop MapFile and make a custom store file format with (competing) design goals:
** heap efficiency
** fast lookups
** minimize I/O operations
** optimize for typical DFS blocksizes (8MB, 64MB)
* MRU cache for filehandles and store file indexes
* Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS blockcache for performance
* "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1024) Rearchitect regionserver I/O
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-1024.
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
0.20.0 did enough of this.
> Rearchitect regionserver I/O
> ----------------------------
>
> Key: HBASE-1024
> URL: https://issues.apache.org/jira/browse/HBASE-1024
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: io, regionserver
> Reporter: Andrew Purtell
> Fix For: 0.20.0
>
>
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design goals:
> ** heap efficiency
> ** fast lookups
> ** minimize I/O operations
> ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1024) Rearchitect regionserver I/O
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650413#action_12650413 ]
stack commented on HBASE-1024:
------------------------------
Good stuff Andrew. I concur. Lets take on these goals. Shall we put up a wiki page pointing to this issue that talks of rearchitecting project? and maybe belong in a working document of their own. Perhaps a page up on wiki -- like http://wiki.apache.org/hadoop/Hbase/NewFileFormat -- with perhaps a link under roadmap or maybe we need an architectural goals section where we stable these targets (and X-them out as we knock them off)?
> Rearchitect regionserver I/O
> ----------------------------
>
> Key: HBASE-1024
> URL: https://issues.apache.org/jira/browse/HBASE-1024
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: io, regionserver
> Reporter: Andrew Purtell
>
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design goals:
> ** heap efficiency
> ** fast lookups
> ** minimize I/O operations
> ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1024) Rearchitect regionserver I/O
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678622#action_12678622 ]
stack commented on HBASE-1024:
------------------------------
Bunch of us discussed the zero-copy above this evening at HUG6. In particular remove HStoreKey and just do byte array all the ways into the HFile and then on the way out, carry the HFile key and value all the ways out to IPC. Will make an issue once I have better handle on it.
> Rearchitect regionserver I/O
> ----------------------------
>
> Key: HBASE-1024
> URL: https://issues.apache.org/jira/browse/HBASE-1024
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: io, regionserver
> Reporter: Andrew Purtell
>
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design goals:
> ** heap efficiency
> ** fast lookups
> ** minimize I/O operations
> ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1024) Rearchitect regionserver I/O
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650417#action_12650417 ]
Andrew Purtell commented on HBASE-1024:
---------------------------------------
Hi Stack. I was thinking of setting up subtasks under this issue in part. A page on the wiki would be good also. I'll set one up if someone doesn't get to it first.
> Rearchitect regionserver I/O
> ----------------------------
>
> Key: HBASE-1024
> URL: https://issues.apache.org/jira/browse/HBASE-1024
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: io, regionserver
> Reporter: Andrew Purtell
>
> Consider supporting:
> * 2GB store files
> * 1TB per node (500 store files)
> * Cell values up to ~100MB
> * Typical use case of RS running with 1GB of heap only
> Some ideas:
> * Drop MapFile and make a custom store file format with (competing) design goals:
> ** heap efficiency
> ** fast lookups
> ** minimize I/O operations
> ** optimize for typical DFS blocksizes (8MB, 64MB)
> * MRU cache for filehandles and store file indexes
> * Memory mapped store file indexes -- don't hold the indexes in heap; rely on the OS blockcache for performance
> * "Zero copy" I/O from IPC to store file and vice versa, like NIO buffers
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.