Re: [jira] Commented: (HBASE-1197) IPC of large cells should transfer in chunks not via naive full copy

[Ryan Rawson <ry...@gmail.com>]

I doubt we could chunk values straight into HFile - you'd have to have 1
file for 1 value.  If your value is that large (more than hundreds of megs),
maybe you shouldn't be storing that in HBase - store directly in HDFS and
use HBase to index the content and provide filename pointers.

As it stands, right now a key/value has to live in memcache for some period
of time (seconds? minutes?), so storing an entire key/value has to be
feasable.  Not supporting chunking/streaming doesn't seem to be a major
deficiency.

I think of hbase as a way of making it possible to effiency store smallish
values on HDFS.  I think we should support reasonably large values, but
right now there is a 2gb value max size (int size for value).  With enough
RAM thrown at HBase it should be possible to support nearly all of that
size.

On Wed, Feb 11, 2009 at 9:51 PM, stack (JIRA) <ji...@apache.org> wrote:

>
>    [
> https://issues.apache.org/jira/browse/HBASE-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672889#action_12672889]
>
> stack commented on HBASE-1197:
> ------------------------------
>
> Its almost as though we should chunk end-to-end; from hfile all the ways on
> out to the client.
>
> Tough one is on the way in.  The big cell is going to have to sit in the
> mecache for some -- probably small if its a big cell value -- period of
> time.  Can we chunk on the way in?  From client to hfile?
>
>
>
> > IPC of large cells should transfer in chunks not via naive full copy
> > --------------------------------------------------------------------
> >
> >                 Key: HBASE-1197
> >                 URL: https://issues.apache.org/jira/browse/HBASE-1197
> >             Project: Hadoop HBase
> >          Issue Type: Improvement
> >            Reporter: Andrew Purtell
> >             Fix For: 0.20.0
> >
> >
> > Several instances of OOME when trying to serve up large cells to clients
> have been observed. IPC should send large cell content in chunks instead of
> as one large naive copy.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>