[jira] Commented: (HBASE-1234) Change HBase StoreKey format

["stack (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678592#action_12678592 ] 

stack commented on HBASE-1234:
------------------------------

Chatting w/ lads at HUG6 -- in particular jgray, holstad, jimk, ryan, travis -- we came up w/ following proposed format:

{code}<int1><int2><int3><row><columnfamily><columnqualifier><timestamp><type>{code}

int1 is the length of row,  2 bytes
int2 is length of columnfamily, 1 byte
int3 is length of columnqualifier, 2 bytes

The ints above are plain ints, not vints

The type is a single byte.  Types are update, cell delete, column delete, column family delete, row delete and possibly counter.

 

> Change HBase StoreKey format
> ----------------------------
>
>                 Key: HBASE-1234
>                 URL: https://issues.apache.org/jira/browse/HBASE-1234
>             Project: Hadoop HBase
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.20.0
>
>
> HBASE-859 cleaned up keys removing the need of HRegionInfo being in the context comparing keys.  This issue is about changing the format.  Work done in HBASE-859 means changes have been localized to HStoreKey, in particular to comparators and parse routines.  We should do this now since 0.20.0 will require rewriting all data.
> Things to consider:
> <row> <columnfamily> <columnqualifier> <timestamp> <keytype>
> Or leave off columnfamily altogether and just write it once into the hfile metadata (All key compares are done in the Store context so columnfamily can be safely left out of the equation; its only when the key rises above Store that the columnfamily needs appending).
> keytype is probably a byte. Types are delete cell, delete row, delete family, delete column?  What else?  Where should we put it?  At the end?  How should type sort?  Or should it not be part of sort so its just the order at which we encounter the key?
> How are we going to support keys that go in out of chronological order?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [jira] Commented: (HBASE-1234) Change HBase StoreKey format

[Ryan Rawson <ry...@gmail.com>]

+1 I like it.

HFile would only require modest modifications to support this.

-ryan

On Wed, Mar 4, 2009 at 11:12 AM, stack <st...@duboce.net> wrote:

> I just chatted with Erik and I missed what he was saying altogether.  His
> point was that we can drop the columnqualifier length IF we know the key
> overall length which I think is going to be true in near all cases.
>
> Chatting further (Jon and Erik just came by the house), they argue that the
> native regionserver entity should be a KeyValue blob whose format is:
>
> keylength
> valuelength
> key
> value
>
> ..where the key is then further decomposable as suggested below
> (rowlength-int, familylength-int, row, family, qualifier, timestamp, type).
> The blob would be carried in a ByteBuffer.
>
> On the way in, we'd make one of these out of the proffered row, column,
> etc.
> and shove it into the Memcache (Memcache would change from TreeMap to
> TreeSet).  Flushing would be append of this KeyValue to hfile.
>
> On way out, we'd pick the KeyValue blob from hfile and move this through
> the
> system out to the RPC.
>
> (One day we might put the KeyValue blob on nio if we use something other
> than hadoop's RPC).
>
> St.Ack
>
> On Wed, Mar 4, 2009 at 9:06 AM, stack <st...@duboce.net> wrote:
>
> > On Wed, Mar 4, 2009 at 8:19 AM, Erik Holstad <erikholstad@gmail.com
> >wrote:
> >
> >> Was thinking this morning that me might have to do some adjustments in
> the
> >> format,
> >> we wanted <int><int><int><row><fam><qf><ts><type> for the key and
> >> <int><value> or
> >> something like that, right?
> >> But what is stored in HFile right now is if I'm not mistaken
> >> <int><int><key><val>, so if
> >> we want to match that I think we need to do some small adjustments
> >> probably
> >> to something
> >> like:
> >> <keyLen><valLen><rowLen><famLent><row><fam><qf><ts><type><val>
> >
> >
> >
> > I think we are saying the same thing (if your omission of columnqualifier
> > length was not intentional).
> >
> > In hfile currently its as you say:
> >
> > keylength
> > vallength
> > key
> > value
> >
> > where key expands to
> >
> > vint // Length of the row as vint
> > row
> > vint // Length of the column -- family + qualifier -- as vint
> > column
> > timestamp
> >
> > The proposal is that hfile is as it was only the key now exands to:
> >
> > int // rowlength as a short
> > int // column family length in a byte
> > int // column qualifier length in a short
> > row
> > columnfamily
> > columnqualifier
> > timestamp
> > type
> >
> > St.Ack
> >
> >
> >
>

Re: [jira] Commented: (HBASE-1234) Change HBase StoreKey format

[stack <st...@duboce.net>]

I just chatted with Erik and I missed what he was saying altogether.  His
point was that we can drop the columnqualifier length IF we know the key
overall length which I think is going to be true in near all cases.

Chatting further (Jon and Erik just came by the house), they argue that the
native regionserver entity should be a KeyValue blob whose format is:

keylength
valuelength
key
value

..where the key is then further decomposable as suggested below
(rowlength-int, familylength-int, row, family, qualifier, timestamp, type).
The blob would be carried in a ByteBuffer.

On the way in, we'd make one of these out of the proffered row, column, etc.
and shove it into the Memcache (Memcache would change from TreeMap to
TreeSet).  Flushing would be append of this KeyValue to hfile.

On way out, we'd pick the KeyValue blob from hfile and move this through the
system out to the RPC.

(One day we might put the KeyValue blob on nio if we use something other
than hadoop's RPC).

St.Ack

On Wed, Mar 4, 2009 at 9:06 AM, stack <st...@duboce.net> wrote:

> On Wed, Mar 4, 2009 at 8:19 AM, Erik Holstad <er...@gmail.com>wrote:
>
>> Was thinking this morning that me might have to do some adjustments in the
>> format,
>> we wanted <int><int><int><row><fam><qf><ts><type> for the key and
>> <int><value> or
>> something like that, right?
>> But what is stored in HFile right now is if I'm not mistaken
>> <int><int><key><val>, so if
>> we want to match that I think we need to do some small adjustments
>> probably
>> to something
>> like:
>> <keyLen><valLen><rowLen><famLent><row><fam><qf><ts><type><val>
>
>
>
> I think we are saying the same thing (if your omission of columnqualifier
> length was not intentional).
>
> In hfile currently its as you say:
>
> keylength
> vallength
> key
> value
>
> where key expands to
>
> vint // Length of the row as vint
> row
> vint // Length of the column -- family + qualifier -- as vint
> column
> timestamp
>
> The proposal is that hfile is as it was only the key now exands to:
>
> int // rowlength as a short
> int // column family length in a byte
> int // column qualifier length in a short
> row
> columnfamily
> columnqualifier
> timestamp
> type
>
> St.Ack
>
>
>

Re: [jira] Commented: (HBASE-1234) Change HBase StoreKey format

[stack <st...@duboce.net>]

On Wed, Mar 4, 2009 at 8:19 AM, Erik Holstad <er...@gmail.com> wrote:

> Was thinking this morning that me might have to do some adjustments in the
> format,
> we wanted <int><int><int><row><fam><qf><ts><type> for the key and
> <int><value> or
> something like that, right?
> But what is stored in HFile right now is if I'm not mistaken
> <int><int><key><val>, so if
> we want to match that I think we need to do some small adjustments probably
> to something
> like:
> <keyLen><valLen><rowLen><famLent><row><fam><qf><ts><type><val>



I think we are saying the same thing (if your omission of columnqualifier
length was not intentional).

In hfile currently its as you say:

keylength
vallength
key
value

where key expands to

vint // Length of the row as vint
row
vint // Length of the column -- family + qualifier -- as vint
column
timestamp

The proposal is that hfile is as it was only the key now exands to:

int // rowlength as a short
int // column family length in a byte
int // column qualifier length in a short
row
columnfamily
columnqualifier
timestamp
type

St.Ack

Re: [jira] Commented: (HBASE-1234) Change HBase StoreKey format

[Erik Holstad <er...@gmail.com>]

Was thinking this morning that me might have to do some adjustments in the
format,
we wanted <int><int><int><row><fam><qf><ts><type> for the key and
<int><value> or
something like that, right?
But what is stored in HFile right now is if I'm not mistaken
<int><int><key><val>, so if
we want to match that I think we need to do some small adjustments probably
to something
like:
<keyLen><valLen><rowLen><famLent><row><fam><qf><ts><type><val>

Erik