You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Nick Dimiduk <nd...@gmail.com> on 2013/07/11 21:16:31 UTC

Data types stage 1 is ready for reivew

Heya devs,

I have two 90k patches I'd like to get some more eyes on. The first is the
OrderedBytes patch, HBASE-8201 [0]. It is lightly analogous to the existing
Bytes.toXXX methods, but implements order-preserving serialization
strategies. Its encoding format can be inspected to determine the encoding
used. The patch is nearing completion, so now's the time to add your
concerns to RB [1]. The major outstanding issue with this patch is its use
of ByteBuffers; there's concern about the amount of garbage that will be
generated in tight loops. Our ByteRange was proposed as a mutable
alternative. The other concern is performance of the encoding
implementations. While performance is a consideration in this
implementation, I've created a separate ticket [2] for evaluating and
improving runtime performance. Please keep your attention on the on-disk
formats and APIs this patch implements.

The second is the Data Types API patch, HBASE-8693 [3]. It provides a
user-extensible data type API and includes implementations of a number of
simple types. Many of those types depend on the encoding provided by
OrderedByes. It also provides a collection of "legacy" types, which all use
the existing Bytes.toXXX methods. These are intended to ease the burden of
transitioning an existing HBase application over to the new data types
APIs. This patch is in an earlier stage, but it's conceptually quite simple
so it should be easier to review. It's also available on RB [4].

Thanks for your time and attention,
Nick

[0]: https://issues.apache.org/jira/browse/HBASE-8201
 [1]: https://reviews.apache.org/r/11633/
[2]: https://issues.apache.org/jira/browse/HBASE-8694
[3]: https://issues.apache.org/jira/browse/HBASE-8693
[4]: https://reviews.apache.org/r/12069/

Re: Data types stage 1 is ready for reivew

Posted by Nick Dimiduk <nd...@gmail.com>.
An extra week of reviews and comments and updates is behind us. Here's a
summary of the high-level changes.

 - Matt's ByteRange class has been refactored into an interface. That
interface is extended to PositionedByteRange that provides the API goodness
many of you desired. HBASE-9091.
 - The encoding OrderedBytes methods all operate over PositionedByteRange
now. Implementations are largely unchanged. Docstrings have been tightened
up a fair bit. HBASE-8201.
 - The data type API also operates over PositionedByteRange. For all the
data type implementations built on o.a.h.h.u.Bytes, the name "Legacy" is
replaced with "Raw." The FixedLengthWrapper class now provides a method to
inspect its length. HBASE-8693.
 - All remaining issues are outside of the critical scope of on-disk
formats and user-level APIs, and are deferred to follow-up JIRAs.

Thanks,
Nick

On Sat, Aug 3, 2013 at 11:24 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> This is my preferred approach. Many users don't want to have any types on
> the data because they might already have their own format, or just because
> they don't need it.
>
> So we should not enforce clients to use it.
>
> JM
>
> 2013/8/3 Nick Dimiduk <nd...@gmail.com>
>
> > On Sat, Aug 3, 2013 at 10:33 AM, Michael Segel <
> msegel_hadoop@hotmail.com
> > >wrote:
> >
> > > How do you plan on enforcing data types within the engine?
> > >
> >
> > Data types, at this point, are a client-side feature, not enforced by the
> > engine in anyway.
> >
> >  On Aug 1, 2013, at 10:57 PM, Matt Corgan <mc...@hotpads.com> wrote:
> > >
> > > > Looks great to me.  Without the strict dependencies on hadoop or
> hbase
> > > > it'll be easy to pull into its own standalone module or new project
> if
> > > > there's demand.
> > > >
> > > >
> > > > On Thu, Aug 1, 2013 at 6:30 PM, Nick Dimiduk <nd...@gmail.com>
> > wrote:
> > > >
> > > >> Finally-for-real-this-time patches posted. I'll take your +1's any
> > time
> > > now
> > > >> ;)
> > > >>
> > > >> Thanks,
> > > >> Nick
> > > >>
> > > >> On Wed, Jul 24, 2013 at 11:27 AM, Nick Dimiduk <nd...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> As of yesterday, I've posted "final" patched on both HBASE-8201<
> > > >> https://issues.apache.org/jira/browse/HBASE-8201>and
> > > >>> HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The
> > > >> former
> > > >>> specifies on-disk format and the latter is the user-facing API. If
> > > you've
> > > >>> already left me a review, thank you; please have another look at
> > these
> > > >>> patches. If you have an opinion here and haven't voiced it, we're
> > > >>> approaching the "forever hold your peace" part of the ceremony.
> > > >>>
> > > >>> Thanks,
> > > >>> Nick
> > > >>>
> > > >>>
> > > >>> On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com>
> > > >> wrote:
> > > >>>
> > > >>>> Thanks for having a look. If you don't mind terribly, I responded
> to
> > > >> your
> > > >>>> comments on JIRA [0].
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Nick
> > > >>>>
> > > >>>> [0]:
> > > https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
> > > >>>>
> > > >>>>
> > > >>>> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <
> > > >> theo.bertozzi@gmail.com
> > > >>>>> wrote:
> > > >>>>
> > > >>>>> I was looking at the HBASE-8693 patch, and looks good to me for
> the
> > > >>>>> primitive types.
> > > >>>>> but I can't see how do you plan to evolve stuff like the struct.
> > > >>>>> By "evolve" I mean add/remove fields, or just query it with a
> > subset
> > > of
> > > >>>>> fields.
> > > >>>>> the fields don't have an id, and on read you must specify all of
> > them
> > > >> in
> > > >>>>> the same order as you've used for write.
> > > >>>>> (but maybe is just an immutable/fixed list of fields, and I'm ok
> > with
> > > >>>>> just
> > > >>>>> adding that info to the comment on top of the class)
> > > >>>>>
> > > >>>>>
> > > >>>>> Matteo
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <
> ndimiduk@gmail.com
> > >
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> New patch posted. What do you think about the new isSkippable()
> > and
> > > >> the
> > > >>>>>> associated limitation in Struct?
> > > >>>>>>
> > > >>>>>> I also posted some "dogfeed" per Enis's suggestion.
> > > >>>>>>
> > > >>>>>> -n
> > > >>>>>>
> > > >>>>>> On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net>
> wrote:
> > > >>>>>>
> > > >>>>>>> On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <
> enis@apache.org>
> > > >>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Did some chatting with Nick today.
> > > >>>>>>>>
> > > >>>>>>>> I think it is really important to get this right, and for that
> > we
> > > >>>>> would
> > > >>>>>>>> definitely need more eyes towards it. The current patch set is
> > > >> in a
> > > >>>>>> good
> > > >>>>>>>> state to bolster the discussion.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>> I'll do another pass (Kicking others to give it a looksee too).
> > > >>>>>>> St.Ack
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: Data types stage 1 is ready for reivew

Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
This is my preferred approach. Many users don't want to have any types on
the data because they might already have their own format, or just because
they don't need it.

So we should not enforce clients to use it.

JM

2013/8/3 Nick Dimiduk <nd...@gmail.com>

> On Sat, Aug 3, 2013 at 10:33 AM, Michael Segel <msegel_hadoop@hotmail.com
> >wrote:
>
> > How do you plan on enforcing data types within the engine?
> >
>
> Data types, at this point, are a client-side feature, not enforced by the
> engine in anyway.
>
>  On Aug 1, 2013, at 10:57 PM, Matt Corgan <mc...@hotpads.com> wrote:
> >
> > > Looks great to me.  Without the strict dependencies on hadoop or hbase
> > > it'll be easy to pull into its own standalone module or new project if
> > > there's demand.
> > >
> > >
> > > On Thu, Aug 1, 2013 at 6:30 PM, Nick Dimiduk <nd...@gmail.com>
> wrote:
> > >
> > >> Finally-for-real-this-time patches posted. I'll take your +1's any
> time
> > now
> > >> ;)
> > >>
> > >> Thanks,
> > >> Nick
> > >>
> > >> On Wed, Jul 24, 2013 at 11:27 AM, Nick Dimiduk <nd...@gmail.com>
> > wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> As of yesterday, I've posted "final" patched on both HBASE-8201<
> > >> https://issues.apache.org/jira/browse/HBASE-8201>and
> > >>> HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The
> > >> former
> > >>> specifies on-disk format and the latter is the user-facing API. If
> > you've
> > >>> already left me a review, thank you; please have another look at
> these
> > >>> patches. If you have an opinion here and haven't voiced it, we're
> > >>> approaching the "forever hold your peace" part of the ceremony.
> > >>>
> > >>> Thanks,
> > >>> Nick
> > >>>
> > >>>
> > >>> On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com>
> > >> wrote:
> > >>>
> > >>>> Thanks for having a look. If you don't mind terribly, I responded to
> > >> your
> > >>>> comments on JIRA [0].
> > >>>>
> > >>>> Thanks,
> > >>>> Nick
> > >>>>
> > >>>> [0]:
> > https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
> > >>>>
> > >>>>
> > >>>> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <
> > >> theo.bertozzi@gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>> I was looking at the HBASE-8693 patch, and looks good to me for the
> > >>>>> primitive types.
> > >>>>> but I can't see how do you plan to evolve stuff like the struct.
> > >>>>> By "evolve" I mean add/remove fields, or just query it with a
> subset
> > of
> > >>>>> fields.
> > >>>>> the fields don't have an id, and on read you must specify all of
> them
> > >> in
> > >>>>> the same order as you've used for write.
> > >>>>> (but maybe is just an immutable/fixed list of fields, and I'm ok
> with
> > >>>>> just
> > >>>>> adding that info to the comment on top of the class)
> > >>>>>
> > >>>>>
> > >>>>> Matteo
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <ndimiduk@gmail.com
> >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> New patch posted. What do you think about the new isSkippable()
> and
> > >> the
> > >>>>>> associated limitation in Struct?
> > >>>>>>
> > >>>>>> I also posted some "dogfeed" per Enis's suggestion.
> > >>>>>>
> > >>>>>> -n
> > >>>>>>
> > >>>>>> On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
> > >>>>>>
> > >>>>>>> On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Did some chatting with Nick today.
> > >>>>>>>>
> > >>>>>>>> I think it is really important to get this right, and for that
> we
> > >>>>> would
> > >>>>>>>> definitely need more eyes towards it. The current patch set is
> > >> in a
> > >>>>>> good
> > >>>>>>>> state to bolster the discussion.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> I'll do another pass (Kicking others to give it a looksee too).
> > >>>>>>> St.Ack
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: Data types stage 1 is ready for reivew

Posted by Nick Dimiduk <nd...@gmail.com>.
On Sat, Aug 3, 2013 at 10:33 AM, Michael Segel <ms...@hotmail.com>wrote:

> How do you plan on enforcing data types within the engine?
>

Data types, at this point, are a client-side feature, not enforced by the
engine in anyway.

 On Aug 1, 2013, at 10:57 PM, Matt Corgan <mc...@hotpads.com> wrote:
>
> > Looks great to me.  Without the strict dependencies on hadoop or hbase
> > it'll be easy to pull into its own standalone module or new project if
> > there's demand.
> >
> >
> > On Thu, Aug 1, 2013 at 6:30 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> >
> >> Finally-for-real-this-time patches posted. I'll take your +1's any time
> now
> >> ;)
> >>
> >> Thanks,
> >> Nick
> >>
> >> On Wed, Jul 24, 2013 at 11:27 AM, Nick Dimiduk <nd...@gmail.com>
> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> As of yesterday, I've posted "final" patched on both HBASE-8201<
> >> https://issues.apache.org/jira/browse/HBASE-8201>and
> >>> HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The
> >> former
> >>> specifies on-disk format and the latter is the user-facing API. If
> you've
> >>> already left me a review, thank you; please have another look at these
> >>> patches. If you have an opinion here and haven't voiced it, we're
> >>> approaching the "forever hold your peace" part of the ceremony.
> >>>
> >>> Thanks,
> >>> Nick
> >>>
> >>>
> >>> On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com>
> >> wrote:
> >>>
> >>>> Thanks for having a look. If you don't mind terribly, I responded to
> >> your
> >>>> comments on JIRA [0].
> >>>>
> >>>> Thanks,
> >>>> Nick
> >>>>
> >>>> [0]:
> https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
> >>>>
> >>>>
> >>>> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <
> >> theo.bertozzi@gmail.com
> >>>>> wrote:
> >>>>
> >>>>> I was looking at the HBASE-8693 patch, and looks good to me for the
> >>>>> primitive types.
> >>>>> but I can't see how do you plan to evolve stuff like the struct.
> >>>>> By "evolve" I mean add/remove fields, or just query it with a subset
> of
> >>>>> fields.
> >>>>> the fields don't have an id, and on read you must specify all of them
> >> in
> >>>>> the same order as you've used for write.
> >>>>> (but maybe is just an immutable/fixed list of fields, and I'm ok with
> >>>>> just
> >>>>> adding that info to the comment on top of the class)
> >>>>>
> >>>>>
> >>>>> Matteo
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> New patch posted. What do you think about the new isSkippable() and
> >> the
> >>>>>> associated limitation in Struct?
> >>>>>>
> >>>>>> I also posted some "dogfeed" per Enis's suggestion.
> >>>>>>
> >>>>>> -n
> >>>>>>
> >>>>>> On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
> >>>>>>
> >>>>>>> On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Did some chatting with Nick today.
> >>>>>>>>
> >>>>>>>> I think it is really important to get this right, and for that we
> >>>>> would
> >>>>>>>> definitely need more eyes towards it. The current patch set is
> >> in a
> >>>>>> good
> >>>>>>>> state to bolster the discussion.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> I'll do another pass (Kicking others to give it a looksee too).
> >>>>>>> St.Ack
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: Data types stage 1 is ready for reivew

Posted by Michael Segel <ms...@hotmail.com>.
Uhm...

Ok.

Silly question. 

How do you plan on enforcing data types within the engine? 

I did a quick read of the Jira, and there is a question of design philosophy that should be discussed. 


On Aug 1, 2013, at 10:57 PM, Matt Corgan <mc...@hotpads.com> wrote:

> Looks great to me.  Without the strict dependencies on hadoop or hbase
> it'll be easy to pull into its own standalone module or new project if
> there's demand.
> 
> 
> On Thu, Aug 1, 2013 at 6:30 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> 
>> Finally-for-real-this-time patches posted. I'll take your +1's any time now
>> ;)
>> 
>> Thanks,
>> Nick
>> 
>> On Wed, Jul 24, 2013 at 11:27 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>> 
>>> Hi everyone,
>>> 
>>> As of yesterday, I've posted "final" patched on both HBASE-8201<
>> https://issues.apache.org/jira/browse/HBASE-8201>and
>>> HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The
>> former
>>> specifies on-disk format and the latter is the user-facing API. If you've
>>> already left me a review, thank you; please have another look at these
>>> patches. If you have an opinion here and haven't voiced it, we're
>>> approaching the "forever hold your peace" part of the ceremony.
>>> 
>>> Thanks,
>>> Nick
>>> 
>>> 
>>> On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com>
>> wrote:
>>> 
>>>> Thanks for having a look. If you don't mind terribly, I responded to
>> your
>>>> comments on JIRA [0].
>>>> 
>>>> Thanks,
>>>> Nick
>>>> 
>>>> [0]: https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
>>>> 
>>>> 
>>>> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <
>> theo.bertozzi@gmail.com
>>>>> wrote:
>>>> 
>>>>> I was looking at the HBASE-8693 patch, and looks good to me for the
>>>>> primitive types.
>>>>> but I can't see how do you plan to evolve stuff like the struct.
>>>>> By "evolve" I mean add/remove fields, or just query it with a subset of
>>>>> fields.
>>>>> the fields don't have an id, and on read you must specify all of them
>> in
>>>>> the same order as you've used for write.
>>>>> (but maybe is just an immutable/fixed list of fields, and I'm ok with
>>>>> just
>>>>> adding that info to the comment on top of the class)
>>>>> 
>>>>> 
>>>>> Matteo
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com>
>>>>> wrote:
>>>>> 
>>>>>> New patch posted. What do you think about the new isSkippable() and
>> the
>>>>>> associated limitation in Struct?
>>>>>> 
>>>>>> I also posted some "dogfeed" per Enis's suggestion.
>>>>>> 
>>>>>> -n
>>>>>> 
>>>>>> On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
>>>>>> 
>>>>>>> On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Did some chatting with Nick today.
>>>>>>>> 
>>>>>>>> I think it is really important to get this right, and for that we
>>>>> would
>>>>>>>> definitely need more eyes towards it. The current patch set is
>> in a
>>>>>> good
>>>>>>>> state to bolster the discussion.
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> I'll do another pass (Kicking others to give it a looksee too).
>>>>>>> St.Ack
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>

Re: Data types stage 1 is ready for reivew

Posted by Matt Corgan <mc...@hotpads.com>.
Looks great to me.  Without the strict dependencies on hadoop or hbase
it'll be easy to pull into its own standalone module or new project if
there's demand.


On Thu, Aug 1, 2013 at 6:30 PM, Nick Dimiduk <nd...@gmail.com> wrote:

> Finally-for-real-this-time patches posted. I'll take your +1's any time now
> ;)
>
> Thanks,
> Nick
>
> On Wed, Jul 24, 2013 at 11:27 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > As of yesterday, I've posted "final" patched on both HBASE-8201<
> https://issues.apache.org/jira/browse/HBASE-8201>and
> > HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The
> former
> > specifies on-disk format and the latter is the user-facing API. If you've
> > already left me a review, thank you; please have another look at these
> > patches. If you have an opinion here and haven't voiced it, we're
> > approaching the "forever hold your peace" part of the ceremony.
> >
> > Thanks,
> > Nick
> >
> >
> > On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com>
> wrote:
> >
> >> Thanks for having a look. If you don't mind terribly, I responded to
> your
> >> comments on JIRA [0].
> >>
> >> Thanks,
> >> Nick
> >>
> >> [0]: https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
> >>
> >>
> >> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <
> theo.bertozzi@gmail.com
> >> > wrote:
> >>
> >>> I was looking at the HBASE-8693 patch, and looks good to me for the
> >>> primitive types.
> >>> but I can't see how do you plan to evolve stuff like the struct.
> >>> By "evolve" I mean add/remove fields, or just query it with a subset of
> >>> fields.
> >>> the fields don't have an id, and on read you must specify all of them
> in
> >>> the same order as you've used for write.
> >>> (but maybe is just an immutable/fixed list of fields, and I'm ok with
> >>> just
> >>> adding that info to the comment on top of the class)
> >>>
> >>>
> >>> Matteo
> >>>
> >>>
> >>>
> >>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com>
> >>> wrote:
> >>>
> >>> > New patch posted. What do you think about the new isSkippable() and
> the
> >>> > associated limitation in Struct?
> >>> >
> >>> > I also posted some "dogfeed" per Enis's suggestion.
> >>> >
> >>> > -n
> >>> >
> >>> > On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
> >>> >
> >>> > > On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
> >>> wrote:
> >>> > >
> >>> > > > Did some chatting with Nick today.
> >>> > > >
> >>> > > > I think it is really important to get this right, and for that we
> >>> would
> >>> > > > definitely need more eyes towards it. The current patch set is
> in a
> >>> > good
> >>> > > > state to bolster the discussion.
> >>> > > >
> >>> > > >
> >>> > >
> >>> > > I'll do another pass (Kicking others to give it a looksee too).
> >>> > > St.Ack
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Re: Data types stage 1 is ready for reivew

Posted by Nick Dimiduk <nd...@gmail.com>.
Finally-for-real-this-time patches posted. I'll take your +1's any time now
;)

Thanks,
Nick

On Wed, Jul 24, 2013 at 11:27 AM, Nick Dimiduk <nd...@gmail.com> wrote:

> Hi everyone,
>
> As of yesterday, I've posted "final" patched on both HBASE-8201<https://issues.apache.org/jira/browse/HBASE-8201>and
> HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The former
> specifies on-disk format and the latter is the user-facing API. If you've
> already left me a review, thank you; please have another look at these
> patches. If you have an opinion here and haven't voiced it, we're
> approaching the "forever hold your peace" part of the ceremony.
>
> Thanks,
> Nick
>
>
> On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
>> Thanks for having a look. If you don't mind terribly, I responded to your
>> comments on JIRA [0].
>>
>> Thanks,
>> Nick
>>
>> [0]: https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
>>
>>
>> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <theo.bertozzi@gmail.com
>> > wrote:
>>
>>> I was looking at the HBASE-8693 patch, and looks good to me for the
>>> primitive types.
>>> but I can't see how do you plan to evolve stuff like the struct.
>>> By "evolve" I mean add/remove fields, or just query it with a subset of
>>> fields.
>>> the fields don't have an id, and on read you must specify all of them in
>>> the same order as you've used for write.
>>> (but maybe is just an immutable/fixed list of fields, and I'm ok with
>>> just
>>> adding that info to the comment on top of the class)
>>>
>>>
>>> Matteo
>>>
>>>
>>>
>>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com>
>>> wrote:
>>>
>>> > New patch posted. What do you think about the new isSkippable() and the
>>> > associated limitation in Struct?
>>> >
>>> > I also posted some "dogfeed" per Enis's suggestion.
>>> >
>>> > -n
>>> >
>>> > On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
>>> >
>>> > > On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
>>> wrote:
>>> > >
>>> > > > Did some chatting with Nick today.
>>> > > >
>>> > > > I think it is really important to get this right, and for that we
>>> would
>>> > > > definitely need more eyes towards it. The current patch set is in a
>>> > good
>>> > > > state to bolster the discussion.
>>> > > >
>>> > > >
>>> > >
>>> > > I'll do another pass (Kicking others to give it a looksee too).
>>> > > St.Ack
>>> > >
>>> >
>>>
>>
>>
>

Re: Data types stage 1 is ready for reivew

Posted by Nick Dimiduk <nd...@gmail.com>.
Hi everyone,

As of yesterday, I've posted "final" patched on both
HBASE-8201<https://issues.apache.org/jira/browse/HBASE-8201>and
HBASE-8693 <https://issues.apache.org/jira/browse/HBASE-8693>. The former
specifies on-disk format and the latter is the user-facing API. If you've
already left me a review, thank you; please have another look at these
patches. If you have an opinion here and haven't voiced it, we're
approaching the "forever hold your peace" part of the ceremony.

Thanks,
Nick

On Wed, Jul 17, 2013 at 9:33 AM, Nick Dimiduk <nd...@gmail.com> wrote:

> Thanks for having a look. If you don't mind terribly, I responded to your
> comments on JIRA [0].
>
> Thanks,
> Nick
>
> [0]: https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250
>
>
> On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <th...@gmail.com>wrote:
>
>> I was looking at the HBASE-8693 patch, and looks good to me for the
>> primitive types.
>> but I can't see how do you plan to evolve stuff like the struct.
>> By "evolve" I mean add/remove fields, or just query it with a subset of
>> fields.
>> the fields don't have an id, and on read you must specify all of them in
>> the same order as you've used for write.
>> (but maybe is just an immutable/fixed list of fields, and I'm ok with just
>> adding that info to the comment on top of the class)
>>
>>
>> Matteo
>>
>>
>>
>> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com>
>> wrote:
>>
>> > New patch posted. What do you think about the new isSkippable() and the
>> > associated limitation in Struct?
>> >
>> > I also posted some "dogfeed" per Enis's suggestion.
>> >
>> > -n
>> >
>> > On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
>> wrote:
>> > >
>> > > > Did some chatting with Nick today.
>> > > >
>> > > > I think it is really important to get this right, and for that we
>> would
>> > > > definitely need more eyes towards it. The current patch set is in a
>> > good
>> > > > state to bolster the discussion.
>> > > >
>> > > >
>> > >
>> > > I'll do another pass (Kicking others to give it a looksee too).
>> > > St.Ack
>> > >
>> >
>>
>
>

Re: Data types stage 1 is ready for reivew

Posted by Nick Dimiduk <nd...@gmail.com>.
Thanks for having a look. If you don't mind terribly, I responded to your
comments on JIRA [0].

Thanks,
Nick

[0]: https://issues.apache.org/jira/browse/HBASE-8693#comment-13711250

On Wed, Jul 17, 2013 at 1:42 AM, Matteo Bertozzi <th...@gmail.com>wrote:

> I was looking at the HBASE-8693 patch, and looks good to me for the
> primitive types.
> but I can't see how do you plan to evolve stuff like the struct.
> By "evolve" I mean add/remove fields, or just query it with a subset of
> fields.
> the fields don't have an id, and on read you must specify all of them in
> the same order as you've used for write.
> (but maybe is just an immutable/fixed list of fields, and I'm ok with just
> adding that info to the comment on top of the class)
>
>
> Matteo
>
>
>
> On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > New patch posted. What do you think about the new isSkippable() and the
> > associated limitation in Struct?
> >
> > I also posted some "dogfeed" per Enis's suggestion.
> >
> > -n
> >
> > On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
> >
> > > On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org>
> wrote:
> > >
> > > > Did some chatting with Nick today.
> > > >
> > > > I think it is really important to get this right, and for that we
> would
> > > > definitely need more eyes towards it. The current patch set is in a
> > good
> > > > state to bolster the discussion.
> > > >
> > > >
> > >
> > > I'll do another pass (Kicking others to give it a looksee too).
> > > St.Ack
> > >
> >
>

Re: Data types stage 1 is ready for reivew

Posted by Matteo Bertozzi <th...@gmail.com>.
I was looking at the HBASE-8693 patch, and looks good to me for the
primitive types.
but I can't see how do you plan to evolve stuff like the struct.
By "evolve" I mean add/remove fields, or just query it with a subset of
fields.
the fields don't have an id, and on read you must specify all of them in
the same order as you've used for write.
(but maybe is just an immutable/fixed list of fields, and I'm ok with just
adding that info to the comment on top of the class)


Matteo



On Wed, Jul 17, 2013 at 12:39 AM, Nick Dimiduk <nd...@gmail.com> wrote:

> New patch posted. What do you think about the new isSkippable() and the
> associated limitation in Struct?
>
> I also posted some "dogfeed" per Enis's suggestion.
>
> -n
>
> On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org> wrote:
> >
> > > Did some chatting with Nick today.
> > >
> > > I think it is really important to get this right, and for that we would
> > > definitely need more eyes towards it. The current patch set is in a
> good
> > > state to bolster the discussion.
> > >
> > >
> >
> > I'll do another pass (Kicking others to give it a looksee too).
> > St.Ack
> >
>

Re: Data types stage 1 is ready for reivew

Posted by Nick Dimiduk <nd...@gmail.com>.
New patch posted. What do you think about the new isSkippable() and the
associated limitation in Struct?

I also posted some "dogfeed" per Enis's suggestion.

-n

On Fri, Jul 12, 2013 at 1:38 PM, Stack <st...@duboce.net> wrote:

> On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org> wrote:
>
> > Did some chatting with Nick today.
> >
> > I think it is really important to get this right, and for that we would
> > definitely need more eyes towards it. The current patch set is in a good
> > state to bolster the discussion.
> >
> >
>
> I'll do another pass (Kicking others to give it a looksee too).
> St.Ack
>

Re: Data types stage 1 is ready for reivew

Posted by Stack <st...@duboce.net>.
On Fri, Jul 12, 2013 at 1:10 PM, Enis Söztutar <en...@apache.org> wrote:

> Did some chatting with Nick today.
>
> I think it is really important to get this right, and for that we would
> definitely need more eyes towards it. The current patch set is in a good
> state to bolster the discussion.
>
>

I'll do another pass (Kicking others to give it a looksee too).
St.Ack

Re: Data types stage 1 is ready for reivew

Posted by Enis Söztutar <en...@apache.org>.
Did some chatting with Nick today.

I think it is really important to get this right, and for that we would
definitely need more eyes towards it. The current patch set is in a good
state to bolster the discussion.

The other important thing is that we should be dogfeeding this in HBase
proper before marking it stable and public. For that, I suggested that we
can start with converting our META row keys and data model to use this
infrastructure as a first complex use case.

Depending on the timeline, I can see that these patches can make into the
0.96 line (before 0.96.0 or even after), but not declared stable in 0.96
releases.

Cheers,
Enis


On Thu, Jul 11, 2013 at 12:16 PM, Nick Dimiduk <nd...@gmail.com> wrote:

> Heya devs,
>
> I have two 90k patches I'd like to get some more eyes on. The first is the
> OrderedBytes patch, HBASE-8201 [0]. It is lightly analogous to the existing
> Bytes.toXXX methods, but implements order-preserving serialization
> strategies. Its encoding format can be inspected to determine the encoding
> used. The patch is nearing completion, so now's the time to add your
> concerns to RB [1]. The major outstanding issue with this patch is its use
> of ByteBuffers; there's concern about the amount of garbage that will be
> generated in tight loops. Our ByteRange was proposed as a mutable
> alternative. The other concern is performance of the encoding
> implementations. While performance is a consideration in this
> implementation, I've created a separate ticket [2] for evaluating and
> improving runtime performance. Please keep your attention on the on-disk
> formats and APIs this patch implements.
>
> The second is the Data Types API patch, HBASE-8693 [3]. It provides a
> user-extensible data type API and includes implementations of a number of
> simple types. Many of those types depend on the encoding provided by
> OrderedByes. It also provides a collection of "legacy" types, which all use
> the existing Bytes.toXXX methods. These are intended to ease the burden of
> transitioning an existing HBase application over to the new data types
> APIs. This patch is in an earlier stage, but it's conceptually quite simple
> so it should be easier to review. It's also available on RB [4].
>
> Thanks for your time and attention,
> Nick
>
> [0]: https://issues.apache.org/jira/browse/HBASE-8201
>  [1]: https://reviews.apache.org/r/11633/
> [2]: https://issues.apache.org/jira/browse/HBASE-8694
> [3]: https://issues.apache.org/jira/browse/HBASE-8693
> [4]: https://reviews.apache.org/r/12069/
>