You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Dave Latham (JIRA)" <ji...@apache.org> on 2012/05/23 18:16:41 UTC

[jira] [Created] (HBASE-6072) Make TableRecordReaderImpl more easily extended

Dave Latham created HBASE-6072:
----------------------------------

             Summary: Make TableRecordReaderImpl more easily extended
                 Key: HBASE-6072
                 URL: https://issues.apache.org/jira/browse/HBASE-6072
             Project: HBase
          Issue Type: Improvement
          Components: mapreduce
            Reporter: Dave Latham
            Priority: Minor


We have a MR job that is very memory bound.  It reads a potentially large row from hbase, then deserializes it into an (even larger) object representation, then does a fair amount of computation requiring memory.  After converting the Result into our object representation we want to free the memory holding the Result to be available for the actual computation of output values.

Currently we have our own custom modified copy of TableRecordReaderImpl to be able to set the Result value to null after reading it, but it's almost entirely a duplicate of hbase's TableRecordReaderImpl so we have to manually keep it up to date with changes to the hbase version.  If the value field of TableRecordReaderImpl were protected instead of private we could use a very simple subclass instead.

Are there any philosophical guidelines about what parts of HBase should or should not be easily extensible?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6072) Make TableRecordReaderImpl more easily extended

Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281723#comment-13281723 ] 

Zhihong Yu commented on HBASE-6072:
-----------------------------------

Instead of making value field protected, we can also add method resetValue() which clears the field.

@Dave:
What do you think ?
                
> Make TableRecordReaderImpl more easily extended
> -----------------------------------------------
>
>                 Key: HBASE-6072
>                 URL: https://issues.apache.org/jira/browse/HBASE-6072
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>            Reporter: Dave Latham
>            Priority: Minor
>
> We have a MR job that is very memory bound.  It reads a potentially large row from hbase, then deserializes it into an (even larger) object representation, then does a fair amount of computation requiring memory.  After converting the Result into our object representation we want to free the memory holding the Result to be available for the actual computation of output values.
> Currently we have our own custom modified copy of TableRecordReaderImpl to be able to set the Result value to null after reading it, but it's almost entirely a duplicate of hbase's TableRecordReaderImpl so we have to manually keep it up to date with changes to the hbase version.  If the value field of TableRecordReaderImpl were protected instead of private we could use a very simple subclass instead.
> Are there any philosophical guidelines about what parts of HBase should or should not be easily extensible?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-6072) Make TableRecordReaderImpl more easily extended

Posted by "Dave Latham (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281740#comment-13281740 ] 

Dave Latham commented on HBASE-6072:
------------------------------------

That works great for my case.

I am still curious about the more general question of what should be extensible, but am content with a specific fix for me if people feel its appropriate to belong in the shared code.
                
> Make TableRecordReaderImpl more easily extended
> -----------------------------------------------
>
>                 Key: HBASE-6072
>                 URL: https://issues.apache.org/jira/browse/HBASE-6072
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>            Reporter: Dave Latham
>            Priority: Minor
>
> We have a MR job that is very memory bound.  It reads a potentially large row from hbase, then deserializes it into an (even larger) object representation, then does a fair amount of computation requiring memory.  After converting the Result into our object representation we want to free the memory holding the Result to be available for the actual computation of output values.
> Currently we have our own custom modified copy of TableRecordReaderImpl to be able to set the Result value to null after reading it, but it's almost entirely a duplicate of hbase's TableRecordReaderImpl so we have to manually keep it up to date with changes to the hbase version.  If the value field of TableRecordReaderImpl were protected instead of private we could use a very simple subclass instead.
> Are there any philosophical guidelines about what parts of HBase should or should not be easily extensible?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira