You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Bryan Duxbury (JIRA)" <ji...@apache.org> on 2008/01/04 02:54:34 UTC
[jira] Created: (HADOOP-2521) [hbase] HStoreFiles needlessly store
the column family name in every entry
[hbase] HStoreFiles needlessly store the column family name in every entry
--------------------------------------------------------------------------
Key: HADOOP-2521
URL: https://issues.apache.org/jira/browse/HADOOP-2521
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Reporter: Bryan Duxbury
Priority: Minor
Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2521) [hbase] HStoreFiles needlessly
store the column family name in every entry
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555778#action_12555778 ]
Jim Kellerman commented on HADOOP-2521:
---------------------------------------
Remember that an HStoreKey contains both the family name and the member name. You could have entries for 'contents:' (just the family name), 'contents:member1', 'contents:member2', etc., and they all get stored in the same HStoreFile.
So unless you want to create a new object type to be the key, and then add the necessary logic to transform to/from HStoreKeys, I'd say that trading off a little space for time is a benefit, not a fault.
-1 on this proposal.
> [hbase] HStoreFiles needlessly store the column family name in every entry
> --------------------------------------------------------------------------
>
> Key: HADOOP-2521
> URL: https://issues.apache.org/jira/browse/HADOOP-2521
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: Bryan Duxbury
> Priority: Minor
>
> Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2521) [hbase] HStoreFiles needlessly
store the column family name in every entry
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556258#action_12556258 ]
Jim Kellerman commented on HADOOP-2521:
---------------------------------------
Point taken. However since we have done little to no performance analysis to date, I would say that this would be a premature optimization. Let's see where the hot spots are first and address them.
> [hbase] HStoreFiles needlessly store the column family name in every entry
> --------------------------------------------------------------------------
>
> Key: HADOOP-2521
> URL: https://issues.apache.org/jira/browse/HADOOP-2521
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: Bryan Duxbury
> Priority: Minor
>
> Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2521) [hbase] HStoreFiles needlessly
store the column family name in every entry
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556302#action_12556302 ]
stack commented on HADOOP-2521:
-------------------------------
Jim: I've been profiling while you've been on holidays. Looks like most of the low-hanging fruit has been picked: e.g. RPC serializations and gratuitous object creations in hbase. Apart from updates to memcache -- SortedMaps are 'expensive' -- the bulk of our time/resources are now in appending and nexting over MapFiles/SequenceFiles whether updating, reading, compacting or flushing (The latter two take up the bulk of CPU during writes at least). Anything we can do to improve our i/o story here will make for a win.
As to dropping family name when we go to the fs, I like the idea, especially as its making keys (slightly) smaller... but yeah, lets measure first to see if these seemingly small savings even show up on the size/speed register.
> [hbase] HStoreFiles needlessly store the column family name in every entry
> --------------------------------------------------------------------------
>
> Key: HADOOP-2521
> URL: https://issues.apache.org/jira/browse/HADOOP-2521
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: Bryan Duxbury
> Priority: Minor
>
> Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2521) [hbase] HStoreFiles needlessly
store the column family name in every entry
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555997#action_12555997 ]
Bryan Duxbury commented on HADOOP-2521:
---------------------------------------
Jim, I am aware that multiple qualified cells show up in the same store file per row in the same HStoreFile. I'm just suggesting that the part that comes before the qualified name is unnecessary.
I understand that changing this would necessitate adding a new key type and transform logic, but I'm not convinced that the translation would actually cost that much more time. You have to recognize that even though the data is precomputed, it is probably coming off of a disk on another computer through the network in 64MB blocks. I have to think that the added transmission time of all the redundant data in aggregate is at least as much as the added time it would take to do translation, and possibly more.
> [hbase] HStoreFiles needlessly store the column family name in every entry
> --------------------------------------------------------------------------
>
> Key: HADOOP-2521
> URL: https://issues.apache.org/jira/browse/HADOOP-2521
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: Bryan Duxbury
> Priority: Minor
>
> Today, HStoreFiles keep the entire serialized HStoreKey objects around for every cell in the HStore. Since HStores are 1-1 with column families, this is really unnecessary - you can always surmise the column family by looking at the HStore it belongs to. (This information would ostensibly come from the file name or a header section.) This means that we could remove the column family part of the HStoreKeys we put into the HStoreFile, reducing the size of data stored. This would be a space-saving benefit, removing redundant data, and could be a speed benefit, as you have to scan over less data in memory and transfer less data over the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.