You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Chia-Ping Tsai <ch...@apache.org> on 2017/09/28 09:25:46 UTC
[DISCUSS] Move Type out of KeyValue
hi folks,
User is allowed to create custom cell but the valid code of type - KeyValue#Type - is declared as IA.Private. As i see it, we should expose KeyValue#Type as Public Client. Three possible ways are shown below:
1) Change declaration of KeyValue#Type from IA.Private to IA.Public
2) Move KeyValue#Type into Cell.
3) Move KeyValue#Type to upper level
Any suggestions?
--
Chia-Ping
Re: [DISCUSS] Move Type out of KeyValue
Posted by Andrew Purtell <an...@gmail.com>.
Ok, thanks. I understand now.
+1
> On Sep 30, 2017, at 9:28 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> The "custom cell type" never exists in the story. (Sorry for misleading you)
>
> Here is the story. i add some custom cells (for saving memory) to Put via Put#add(Cell). The pseudocode of custom cell is shown below.
>
> {code}
> class MyObject() {
> Cell toCell() {
> return CellBuilderFactory.newBuilfer(SHALLOW_COPY)
> .setRow(sharedBuffer, myRowOffset, myRowLength).
> .setType(KeyValue.Type.Put.getCode()) // We call the IA.Private to get valid code of Put
> // set other fields
> .build();
> }
> }
>
> put.add(myObject.toCell);
> {code}
>
> And then, I noticed the Put#add is not optimized for our heavy table(a chunk of cells in single row), so I also extend the Put to add some #add methods for avoiding resizing collection.
>
> That was the story -- I try to reducer the cost of converting our object to Put/Cell. A another story i had mentioned is to build custom write path via Endpoint, but it is unrelated to this topic.
>
> All class we use are shown below:
> 1) Cell -> IA.Public
> 2) CellBuilder -> IA.Public
> 3) CellBuilderFactory -> IA.Public
> 4) Put -> IA.Public
> 5) Put#add(Cell) -> IA.Public
> 5) KeyValue#Type -> IA.Private
>
> That is why i want to make KeyValue#Type IA.Public.
>
> --
> Chia-Ping
>
>> On 2017-10-01 00:34, Andrew Purtell <an...@gmail.com> wrote:
>> Thanks for sharing these details. They are intriguing. If possible could you explain why the custom type is needed?
>>
>> Something has to be deployed on the server or the custom cell type isn’t guaranteed to be handled correctly. It may work now by accident. I’m a little surprised a custom cell type doesn’t cause an abort. Did you patch the code to handle it?
>>
>>
>>> On Sep 30, 2017, at 1:06 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
>>>
>>> Thanks for the nice suggestions. Andrew. Sorry for delay response. Busy today.
>>>
>>> The root reason we must build own Cell on client side is that the data are located on shared memory which is similar with MSLAB.
>>>
>>> You are right. We can use attribute to carry our data but the byte[] is not acceptable because we can’t assign the offset and length. In fact, the endpoint is a better way for our case because our object can be directly converted to PB object. Also it is easy to apply shared memory to manage our object. However, it will be easier and more readable to follow regular Put operation. All we have to do is to build own cell and extended Put. Nothing have to be deployed on server.
>>>
>>> I agree the custom cell is low level thing, and it should be used by advanced users. What I concern is the classes related to custom Cell have different IA declaration. I’am fine to make them IA.Private but building the custom cell may be a common case.
>>>
>>> —
>>> Chia-Ping
>>>
>>>> On 2017-09-30 06:05, Andrew Purtell <ap...@apache.org> wrote:
>>>> Construct a normal put or delete or batch mutation, add whatever extra
>>>> state you need in one or more operation attributes, and use a
>>>> regionobserver to extend normal processing to handle the extra state. I'm
>>>> curious what dispatching to extension code because of a custom cell type
>>>> buys you over dispatching to extension code because of the presence of an
>>>> attribute (or cell tag). For example, in security coprocessors we take
>>>> attribute data and attach it to the cell using cell tags. Later we check
>>>> for cell tag(s) to determine if we have to take special action when the
>>>> cell is accessed by a scanner, or during some operations (e.g. appends or
>>>> increments have to do extra handling for cell security tags).
>>>>
>>>>
>>>> On Fri, Sep 29, 2017 at 2:43 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
>>>>
>>>>>> Instead of a custom cell, could you use a regular cell with a custom
>>>>>> operation attribute (see OperationWithAttributes).
>>>>> Pardon me, I didn't get what you said.
>>>>>
>>>>>
>>>>>
>>>>>> On 2017-09-30 04:31, Andrew Purtell <ap...@apache.org> wrote:
>>>>>> Instead of a custom cell, could you use a regular cell with a custom
>>>>>> operation attribute (see OperationWithAttributes).
>>>>>>
>>>>>> On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>>> The custom cell help us to save memory consumption. We don't have own
>>>>>>> serialization/deserialization mechanism, hence to transform data from
>>>>>>> client to server needs many conversion phase (user data -> Put/Cell ->
>>>>> pb
>>>>>>> object). The cost of conversion is large in transferring bulk data. In
>>>>>>> fact, we also have custom mutation to manage the memory usage of inner
>>>>> cell
>>>>>>> collection.
>>>>>>>
>>>>>>>> On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
>>>>>>>> What are the use cases for a custom cell? It seems a dangerously low
>>>>>>> level
>>>>>>>> thing to attempt and perhaps we should unwind support for it. But
>>>>> perhaps
>>>>>>>> there is a compelling justification.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <
>>>>> chia7712@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for all comment.
>>>>>>>>>
>>>>>>>>> The problem i want to resolve is the valid code should be exposed
>>>>> as
>>>>>>>>> IA.Public. Otherwise, end user have to access the IA.Private class
>>>>> to
>>>>>>> build
>>>>>>>>> the custom cell.
>>>>>>>>>
>>>>>>>>> For example, I have a use case which plays a streaming role in our
>>>>>>>>> appliaction. It
>>>>>>>>> applies the CellBuilder(HBASE-18519) to build custom cells. These
>>>>> cells
>>>>>>>>> have many same fields so they are put in shared-memory for
>>>>> avoiding GC
>>>>>>>>> pause. Everything is wonderful. However, we have to access the
>>>>>>> IA.Private
>>>>>>>>> class - KeyValue#Type - to get the valid code of Put.
>>>>>>>>>
>>>>>>>>> I believe there are many use cases of custom cell, and
>>>>> consequently it
>>>>>>> is
>>>>>>>>> worth adding a way to get the valid type via IA.Public class.
>>>>>>> Otherwise, it
>>>>>>>>> may imply that the custom cell is based on a unstable way, because
>>>>> the
>>>>>>>>> related code can be changed at any time.
>>>>>>>>> --
>>>>>>>>> Chia-Ping
>>>>>>>>>
>>>>>>>>>> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
>>>>>>>>>> I agree with Stack. Was typing up a reply to Anoop but let me
>>>>> move it
>>>>>>>>> down
>>>>>>>>>> here.
>>>>>>>>>>
>>>>>>>>>> The type code exposes some low level details of how our current
>>>>>>> stores
>>>>>>>>> are
>>>>>>>>>> architected. But what if in the future you could swap out HStore
>>>>>>>>> implements
>>>>>>>>>> Store with PStore implements Store, where HStore is backed by
>>>>> HFiles
>>>>>>> and
>>>>>>>>>> PStore is backed by Parquet? Just as a hypothetical example. I
>>>>> know
>>>>>>> there
>>>>>>>>>> would be larger issues if this were actually attempted. Bear with
>>>>>>> me. You
>>>>>>>>>> can imagine some different new Store implementation that has some
>>>>>>>>>> advantages but is not a design derived from the log structured
>>>>> merge
>>>>>>> tree
>>>>>>>>>> if you like. Most values from a new Cell.Type based on
>>>>> KeyValue.Type
>>>>>>>>>> wouldn't apply to cells from such a thing because they are
>>>>>>> particular to
>>>>>>>>>> how LSMs work. I'm sure such a project if attempted would make a
>>>>>>> number
>>>>>>>>> of
>>>>>>>>>> changes requiring a major version increment and low level details
>>>>>>> could
>>>>>>>>> be
>>>>>>>>>> unwound from Cell then, but if we could avoid doing it in the
>>>>> first
>>>>>>>>> place,
>>>>>>>>>> I think it would better for maintainability.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
>>>>>>> chia7712@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> User is allowed to create custom cell but the valid code of
>>>>> type
>>>>>>> -
>>>>>>>>>>>> KeyValue#Type - is declared as IA.Private. As i see it, we
>>>>> should
>>>>>>>>> expose
>>>>>>>>>>>> KeyValue#Type as Public Client. Three possible ways are shown
>>>>>>> below:
>>>>>>>>>>>> 1) Change declaration of KeyValue#Type from IA.Private to
>>>>>>> IA.Public
>>>>>>>>>>>> 2) Move KeyValue#Type into Cell.
>>>>>>>>>>>> 3) Move KeyValue#Type to upper level
>>>>>>>>>>>>
>>>>>>>>>>>> Any suggestions?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> What is the problem that we are trying to solve Chia-Ping? You
>>>>>>> want to
>>>>>>>>> make
>>>>>>>>>>> Cells of a new Type?
>>>>>>>>>>>
>>>>>>>>>>> My first reaction is that KV#Type is particular to the KV
>>>>>>>>> implementation.
>>>>>>>>>>> Any new Cell implementation should not have to adopt the
>>>>> KeyValue
>>>>>>>>> typing
>>>>>>>>>>> mechanism.
>>>>>>>>>>>
>>>>>>>>>>> S
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best regards,
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>> Words like orphans lost among the crosstalk, meaning torn from
>>>>>>> truth's
>>>>>>>>>> decrepit hands
>>>>>>>>>> - A23, Crosstalk
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>> Words like orphans lost among the crosstalk, meaning torn from
>>>>> truth's
>>>>>>>> decrepit hands
>>>>>>>> - A23, Crosstalk
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Andrew
>>>>>>
>>>>>> Words like orphans lost among the crosstalk, meaning torn from truth's
>>>>>> decrepit hands
>>>>>> - A23, Crosstalk
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrew
>>>>
>>>> Words like orphans lost among the crosstalk, meaning torn from truth's
>>>> decrepit hands
>>>> - A23, Crosstalk
>>>>
>>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
The "custom cell type" never exists in the story. (Sorry for misleading you)
Here is the story. i add some custom cells (for saving memory) to Put via Put#add(Cell). The pseudocode of custom cell is shown below.
{code}
class MyObject() {
Cell toCell() {
return CellBuilderFactory.newBuilfer(SHALLOW_COPY)
.setRow(sharedBuffer, myRowOffset, myRowLength).
.setType(KeyValue.Type.Put.getCode()) // We call the IA.Private to get valid code of Put
// set other fields
.build();
}
}
put.add(myObject.toCell);
{code}
And then, I noticed the Put#add is not optimized for our heavy table(a chunk of cells in single row), so I also extend the Put to add some #add methods for avoiding resizing collection.
That was the story -- I try to reducer the cost of converting our object to Put/Cell. A another story i had mentioned is to build custom write path via Endpoint, but it is unrelated to this topic.
All class we use are shown below:
1) Cell -> IA.Public
2) CellBuilder -> IA.Public
3) CellBuilderFactory -> IA.Public
4) Put -> IA.Public
5) Put#add(Cell) -> IA.Public
5) KeyValue#Type -> IA.Private
That is why i want to make KeyValue#Type IA.Public.
--
Chia-Ping
On 2017-10-01 00:34, Andrew Purtell <an...@gmail.com> wrote:
> Thanks for sharing these details. They are intriguing. If possible could you explain why the custom type is needed?
>
> Something has to be deployed on the server or the custom cell type isnât guaranteed to be handled correctly. It may work now by accident. Iâm a little surprised a custom cell type doesnât cause an abort. Did you patch the code to handle it?
>
>
> > On Sep 30, 2017, at 1:06 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
> >
> > Thanks for the nice suggestions. Andrew. Sorry for delay response. Busy today.
> >
> > The root reason we must build own Cell on client side is that the data are located on shared memory which is similar with MSLAB.
> >
> > You are right. We can use attribute to carry our data but the byte[] is not acceptable because we canât assign the offset and length. In fact, the endpoint is a better way for our case because our object can be directly converted to PB object. Also it is easy to apply shared memory to manage our object. However, it will be easier and more readable to follow regular Put operation. All we have to do is to build own cell and extended Put. Nothing have to be deployed on server.
> >
> > I agree the custom cell is low level thing, and it should be used by advanced users. What I concern is the classes related to custom Cell have different IA declaration. Iâam fine to make them IA.Private but building the custom cell may be a common case.
> >
> > â
> > Chia-Ping
> >
> >> On 2017-09-30 06:05, Andrew Purtell <ap...@apache.org> wrote:
> >> âConstruct a normal put or delete or batch mutation, add whatever extra
> >> state you need in one or more operation attributes, and use a
> >> regionobserver to extend normal processing to handle the extra state. I'm
> >> curious what dispatching to extension code because of a custom cell type
> >> buys you over dispatching to extension code because of the presence of an
> >> attribute (or cell tag). For example, in security coprocessors we take
> >> attribute data and attach it to the cell using cell tags. Later we check
> >> for cell tag(s) to determine if we have to take special action when the
> >> cell is accessed by a scanner, or during some operations (e.g. appends or
> >> increments have to do extra handling for cell security tags).
> >>
> >>
> >> On Fri, Sep 29, 2017 at 2:43 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
> >>
> >>>> Instead of a custom cell, could you use a regular cell with a custom
> >>>> operation attribute (see OperationWithAttributes).
> >>> Pardon me, I didn't get what you said.
> >>>
> >>>
> >>>
> >>>> On 2017-09-30 04:31, Andrew Purtell <ap...@apache.org> wrote:
> >>>> Instead of a custom cell, could you use a regular cell with a custom
> >>>> operation attribute (see OperationWithAttributes).
> >>>>
> >>>> On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org>
> >>> wrote:
> >>>>
> >>>>> The custom cell help us to save memory consumption. We don't have own
> >>>>> serialization/deserialization mechanism, hence to transform data from
> >>>>> client to server needs many conversion phase (user data -> Put/Cell ->
> >>> pb
> >>>>> object). The cost of conversion is large in transferring bulk data. In
> >>>>> fact, we also have custom mutation to manage the memory usage of inner
> >>> cell
> >>>>> collection.
> >>>>>
> >>>>>> On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
> >>>>>> What are the use cases for a custom cell? It seems a dangerously low
> >>>>> level
> >>>>>> thing to attempt and perhaps we should unwind support for it. But
> >>> perhaps
> >>>>>> there is a compelling justification.
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <
> >>> chia7712@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for all comment.
> >>>>>>>
> >>>>>>> The problem i want to resolve is the valid code should be exposed
> >>> as
> >>>>>>> IA.Public. Otherwise, end user have to access the IA.Private class
> >>> to
> >>>>> build
> >>>>>>> the custom cell.
> >>>>>>>
> >>>>>>> For example, I have a use case which plays a streaming role in our
> >>>>>>> appliaction. It
> >>>>>>> applies the CellBuilder(HBASE-18519) to build custom cells. These
> >>> cells
> >>>>>>> have many same fields so they are put in shared-memory for
> >>> avoiding GC
> >>>>>>> pause. Everything is wonderful. However, we have to access the
> >>>>> IA.Private
> >>>>>>> class - KeyValue#Type - to get the valid code of Put.
> >>>>>>>
> >>>>>>> I believe there are many use cases of custom cell, and
> >>> consequently it
> >>>>> is
> >>>>>>> worth adding a way to get the valid type via IA.Public class.
> >>>>> Otherwise, it
> >>>>>>> may imply that the custom cell is based on a unstable way, because
> >>> the
> >>>>>>> related code can be changed at any time.
> >>>>>>> --
> >>>>>>> Chia-Ping
> >>>>>>>
> >>>>>>>> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> >>>>>>>> I agree with Stack. Was typing up a reply to Anoop but let me
> >>> move it
> >>>>>>> down
> >>>>>>>> here.
> >>>>>>>>
> >>>>>>>> The type code exposes some low level details of how our current
> >>>>> stores
> >>>>>>> are
> >>>>>>>> architected. But what if in the future you could swap out HStore
> >>>>>>> implements
> >>>>>>>> Store with PStore implements Store, where HStore is backed by
> >>> HFiles
> >>>>> and
> >>>>>>>> PStore is backed by Parquet? Just as a hypothetical example. I
> >>> know
> >>>>> there
> >>>>>>>> would be larger issues if this were actually attempted. Bear with
> >>>>> me. You
> >>>>>>>> can imagine some different new Store implementation that has some
> >>>>>>>> advantages but is not a design derived from the log structured
> >>> merge
> >>>>> tree
> >>>>>>>> if you like. Most values from a new Cell.Type based on
> >>> KeyValue.Type
> >>>>>>>> wouldn't apply to cells from such a thing because they are
> >>>>> particular to
> >>>>>>>> how LSMs work. I'm sure such a project if attempted would make a
> >>>>> number
> >>>>>>> of
> >>>>>>>> changes requiring a major version increment and low level details
> >>>>> could
> >>>>>>> be
> >>>>>>>> unwound from Cell then, but if we could avoid doing it in the
> >>> first
> >>>>>>> place,
> >>>>>>>> I think it would better for maintainability.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> >>>>>>>>>
> >>>>>>>>> On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> >>>>> chia7712@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> hi folks,
> >>>>>>>>>>
> >>>>>>>>>> User is allowed to create custom cell but the valid code of
> >>> type
> >>>>> -
> >>>>>>>>>> KeyValue#Type - is declared as IA.Private. As i see it, we
> >>> should
> >>>>>>> expose
> >>>>>>>>>> KeyValue#Type as Public Client. Three possible ways are shown
> >>>>> below:
> >>>>>>>>>> 1) Change declaration of KeyValue#Type from IA.Private to
> >>>>> IA.Public
> >>>>>>>>>> 2) Move KeyValue#Type into Cell.
> >>>>>>>>>> 3) Move KeyValue#Type to upper level
> >>>>>>>>>>
> >>>>>>>>>> Any suggestions?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> What is the problem that we are trying to solve Chia-Ping? You
> >>>>> want to
> >>>>>>> make
> >>>>>>>>> Cells of a new Type?
> >>>>>>>>>
> >>>>>>>>> My first reaction is that KV#Type is particular to the KV
> >>>>>>> implementation.
> >>>>>>>>> Any new Cell implementation should not have to adopt the
> >>> KeyValue
> >>>>>>> typing
> >>>>>>>>> mechanism.
> >>>>>>>>>
> >>>>>>>>> S
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best regards,
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> Words like orphans lost among the crosstalk, meaning torn from
> >>>>> truth's
> >>>>>>>> decrepit hands
> >>>>>>>> - A23, Crosstalk
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>> Andrew
> >>>>>>
> >>>>>> Words like orphans lost among the crosstalk, meaning torn from
> >>> truth's
> >>>>>> decrepit hands
> >>>>>> - A23, Crosstalk
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrew
> >>>>
> >>>> Words like orphans lost among the crosstalk, meaning torn from truth's
> >>>> decrepit hands
> >>>> - A23, Crosstalk
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Words like orphans lost among the crosstalk, meaning torn from truth's
> >> decrepit hands
> >> - A23, Crosstalk
> >>
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Andrew Purtell <an...@gmail.com>.
Thanks for sharing these details. They are intriguing. If possible could you explain why the custom type is needed?
Something has to be deployed on the server or the custom cell type isn’t guaranteed to be handled correctly. It may work now by accident. I’m a little surprised a custom cell type doesn’t cause an abort. Did you patch the code to handle it?
> On Sep 30, 2017, at 1:06 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> Thanks for the nice suggestions. Andrew. Sorry for delay response. Busy today.
>
> The root reason we must build own Cell on client side is that the data are located on shared memory which is similar with MSLAB.
>
> You are right. We can use attribute to carry our data but the byte[] is not acceptable because we can’t assign the offset and length. In fact, the endpoint is a better way for our case because our object can be directly converted to PB object. Also it is easy to apply shared memory to manage our object. However, it will be easier and more readable to follow regular Put operation. All we have to do is to build own cell and extended Put. Nothing have to be deployed on server.
>
> I agree the custom cell is low level thing, and it should be used by advanced users. What I concern is the classes related to custom Cell have different IA declaration. I’am fine to make them IA.Private but building the custom cell may be a common case.
>
> —
> Chia-Ping
>
>> On 2017-09-30 06:05, Andrew Purtell <ap...@apache.org> wrote:
>> Construct a normal put or delete or batch mutation, add whatever extra
>> state you need in one or more operation attributes, and use a
>> regionobserver to extend normal processing to handle the extra state. I'm
>> curious what dispatching to extension code because of a custom cell type
>> buys you over dispatching to extension code because of the presence of an
>> attribute (or cell tag). For example, in security coprocessors we take
>> attribute data and attach it to the cell using cell tags. Later we check
>> for cell tag(s) to determine if we have to take special action when the
>> cell is accessed by a scanner, or during some operations (e.g. appends or
>> increments have to do extra handling for cell security tags).
>>
>>
>> On Fri, Sep 29, 2017 at 2:43 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
>>
>>>> Instead of a custom cell, could you use a regular cell with a custom
>>>> operation attribute (see OperationWithAttributes).
>>> Pardon me, I didn't get what you said.
>>>
>>>
>>>
>>>> On 2017-09-30 04:31, Andrew Purtell <ap...@apache.org> wrote:
>>>> Instead of a custom cell, could you use a regular cell with a custom
>>>> operation attribute (see OperationWithAttributes).
>>>>
>>>> On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org>
>>> wrote:
>>>>
>>>>> The custom cell help us to save memory consumption. We don't have own
>>>>> serialization/deserialization mechanism, hence to transform data from
>>>>> client to server needs many conversion phase (user data -> Put/Cell ->
>>> pb
>>>>> object). The cost of conversion is large in transferring bulk data. In
>>>>> fact, we also have custom mutation to manage the memory usage of inner
>>> cell
>>>>> collection.
>>>>>
>>>>>> On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
>>>>>> What are the use cases for a custom cell? It seems a dangerously low
>>>>> level
>>>>>> thing to attempt and perhaps we should unwind support for it. But
>>> perhaps
>>>>>> there is a compelling justification.
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <
>>> chia7712@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for all comment.
>>>>>>>
>>>>>>> The problem i want to resolve is the valid code should be exposed
>>> as
>>>>>>> IA.Public. Otherwise, end user have to access the IA.Private class
>>> to
>>>>> build
>>>>>>> the custom cell.
>>>>>>>
>>>>>>> For example, I have a use case which plays a streaming role in our
>>>>>>> appliaction. It
>>>>>>> applies the CellBuilder(HBASE-18519) to build custom cells. These
>>> cells
>>>>>>> have many same fields so they are put in shared-memory for
>>> avoiding GC
>>>>>>> pause. Everything is wonderful. However, we have to access the
>>>>> IA.Private
>>>>>>> class - KeyValue#Type - to get the valid code of Put.
>>>>>>>
>>>>>>> I believe there are many use cases of custom cell, and
>>> consequently it
>>>>> is
>>>>>>> worth adding a way to get the valid type via IA.Public class.
>>>>> Otherwise, it
>>>>>>> may imply that the custom cell is based on a unstable way, because
>>> the
>>>>>>> related code can be changed at any time.
>>>>>>> --
>>>>>>> Chia-Ping
>>>>>>>
>>>>>>>> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
>>>>>>>> I agree with Stack. Was typing up a reply to Anoop but let me
>>> move it
>>>>>>> down
>>>>>>>> here.
>>>>>>>>
>>>>>>>> The type code exposes some low level details of how our current
>>>>> stores
>>>>>>> are
>>>>>>>> architected. But what if in the future you could swap out HStore
>>>>>>> implements
>>>>>>>> Store with PStore implements Store, where HStore is backed by
>>> HFiles
>>>>> and
>>>>>>>> PStore is backed by Parquet? Just as a hypothetical example. I
>>> know
>>>>> there
>>>>>>>> would be larger issues if this were actually attempted. Bear with
>>>>> me. You
>>>>>>>> can imagine some different new Store implementation that has some
>>>>>>>> advantages but is not a design derived from the log structured
>>> merge
>>>>> tree
>>>>>>>> if you like. Most values from a new Cell.Type based on
>>> KeyValue.Type
>>>>>>>> wouldn't apply to cells from such a thing because they are
>>>>> particular to
>>>>>>>> how LSMs work. I'm sure such a project if attempted would make a
>>>>> number
>>>>>>> of
>>>>>>>> changes requiring a major version increment and low level details
>>>>> could
>>>>>>> be
>>>>>>>> unwound from Cell then, but if we could avoid doing it in the
>>> first
>>>>>>> place,
>>>>>>>> I think it would better for maintainability.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>
>>>>>>>>> On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
>>>>> chia7712@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> hi folks,
>>>>>>>>>>
>>>>>>>>>> User is allowed to create custom cell but the valid code of
>>> type
>>>>> -
>>>>>>>>>> KeyValue#Type - is declared as IA.Private. As i see it, we
>>> should
>>>>>>> expose
>>>>>>>>>> KeyValue#Type as Public Client. Three possible ways are shown
>>>>> below:
>>>>>>>>>> 1) Change declaration of KeyValue#Type from IA.Private to
>>>>> IA.Public
>>>>>>>>>> 2) Move KeyValue#Type into Cell.
>>>>>>>>>> 3) Move KeyValue#Type to upper level
>>>>>>>>>>
>>>>>>>>>> Any suggestions?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> What is the problem that we are trying to solve Chia-Ping? You
>>>>> want to
>>>>>>> make
>>>>>>>>> Cells of a new Type?
>>>>>>>>>
>>>>>>>>> My first reaction is that KV#Type is particular to the KV
>>>>>>> implementation.
>>>>>>>>> Any new Cell implementation should not have to adopt the
>>> KeyValue
>>>>>>> typing
>>>>>>>>> mechanism.
>>>>>>>>>
>>>>>>>>> S
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Chia-Ping
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best regards,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>> Words like orphans lost among the crosstalk, meaning torn from
>>>>> truth's
>>>>>>>> decrepit hands
>>>>>>>> - A23, Crosstalk
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>> Andrew
>>>>>>
>>>>>> Words like orphans lost among the crosstalk, meaning torn from
>>> truth's
>>>>>> decrepit hands
>>>>>> - A23, Crosstalk
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrew
>>>>
>>>> Words like orphans lost among the crosstalk, meaning torn from truth's
>>>> decrepit hands
>>>> - A23, Crosstalk
>>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>> - A23, Crosstalk
>>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
Thanks for the nice suggestions. Andrew. Sorry for delay response. Busy today.
The root reason we must build own Cell on client side is that the data are located on shared memory which is similar with MSLAB.
You are right. We can use attribute to carry our data but the byte[] is not acceptable because we canât assign the offset and length. In fact, the endpoint is a better way for our case because our object can be directly converted to PB object. Also it is easy to apply shared memory to manage our object. However, it will be easier and more readable to follow regular Put operation. All we have to do is to build own cell and extended Put. Nothing have to be deployed on server.
I agree the custom cell is low level thing, and it should be used by advanced users. What I concern is the classes related to custom Cell have different IA declaration. Iâam fine to make them IA.Private but building the custom cell may be a common case.
â
Chia-Ping
On 2017-09-30 06:05, Andrew Purtell <ap...@apache.org> wrote:
> âConstruct a normal put or delete or batch mutation, add whatever extra
> state you need in one or more operation attributes, and use a
> regionobserver to extend normal processing to handle the extra state. I'm
> curious what dispatching to extension code because of a custom cell type
> buys you over dispatching to extension code because of the presence of an
> attribute (or cell tag). For example, in security coprocessors we take
> attribute data and attach it to the cell using cell tags. Later we check
> for cell tag(s) to determine if we have to take special action when the
> cell is accessed by a scanner, or during some operations (e.g. appends or
> increments have to do extra handling for cell security tags).
>
>
> On Fri, Sep 29, 2017 at 2:43 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > > Instead of a custom cell, could you use a regular cell with a custom
> > > operation attribute (see OperationWithAttributes).
> > Pardon me, I didn't get what you said.
> >
> >
> >
> > On 2017-09-30 04:31, Andrew Purtell <ap...@apache.org> wrote:
> > > Instead of a custom cell, could you use a regular cell with a custom
> > > operation attribute (see OperationWithAttributes).
> > >
> > > On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > The custom cell help us to save memory consumption. We don't have own
> > > > serialization/deserialization mechanism, hence to transform data from
> > > > client to server needs many conversion phase (user data -> Put/Cell ->
> > pb
> > > > object). The cost of conversion is large in transferring bulk data. In
> > > > fact, we also have custom mutation to manage the memory usage of inner
> > cell
> > > > collection.
> > > >
> > > > On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
> > > > > What are the use cases for a custom cell? It seems a dangerously low
> > > > level
> > > > > thing to attempt and perhaps we should unwind support for it. But
> > perhaps
> > > > > there is a compelling justification.
> > > > >
> > > > >
> > > > > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <
> > chia7712@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thanks for all comment.
> > > > > >
> > > > > > The problem i want to resolve is the valid code should be exposed
> > as
> > > > > > IA.Public. Otherwise, end user have to access the IA.Private class
> > to
> > > > build
> > > > > > the custom cell.
> > > > > >
> > > > > > For example, I have a use case which plays a streaming role in our
> > > > > > appliaction. It
> > > > > > applies the CellBuilder(HBASE-18519) to build custom cells. These
> > cells
> > > > > > have many same fields so they are put in shared-memory for
> > avoiding GC
> > > > > > pause. Everything is wonderful. However, we have to access the
> > > > IA.Private
> > > > > > class - KeyValue#Type - to get the valid code of Put.
> > > > > >
> > > > > > I believe there are many use cases of custom cell, and
> > consequently it
> > > > is
> > > > > > worth adding a way to get the valid type via IA.Public class.
> > > > Otherwise, it
> > > > > > may imply that the custom cell is based on a unstable way, because
> > the
> > > > > > related code can be changed at any time.
> > > > > > --
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > > > > > I agree with Stack. Was typing up a reply to Anoop but let me
> > move it
> > > > > > down
> > > > > > > here.
> > > > > > >
> > > > > > > The type code exposes some low level details of how our current
> > > > stores
> > > > > > are
> > > > > > > architected. But what if in the future you could swap out HStore
> > > > > > implements
> > > > > > > Store with PStore implements Store, where HStore is backed by
> > HFiles
> > > > and
> > > > > > > PStore is backed by Parquet? Just as a hypothetical example. I
> > know
> > > > there
> > > > > > > would be larger issues if this were actually attempted. Bear with
> > > > me. You
> > > > > > > can imagine some different new Store implementation that has some
> > > > > > > advantages but is not a design derived from the log structured
> > merge
> > > > tree
> > > > > > > if you like. Most values from a new Cell.Type based on
> > KeyValue.Type
> > > > > > > wouldn't apply to cells from such a thing because they are
> > > > particular to
> > > > > > > how LSMs work. I'm sure such a project if attempted would make a
> > > > number
> > > > > > of
> > > > > > > changes requiring a major version increment and low level details
> > > > could
> > > > > > be
> > > > > > > unwound from Cell then, but if we could avoid doing it in the
> > first
> > > > > > place,
> > > > > > > I think it would better for maintainability.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > > > > >
> > > > > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > > > chia7712@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > hi folks,
> > > > > > > > >
> > > > > > > > > User is allowed to create custom cell but the valid code of
> > type
> > > > -
> > > > > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> > should
> > > > > > expose
> > > > > > > > > KeyValue#Type as Public Client. Three possible ways are shown
> > > > below:
> > > > > > > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > > > IA.Public
> > > > > > > > > 2) Move KeyValue#Type into Cell.
> > > > > > > > > 3) Move KeyValue#Type to upper level
> > > > > > > > >
> > > > > > > > > Any suggestions?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > What is the problem that we are trying to solve Chia-Ping? You
> > > > want to
> > > > > > make
> > > > > > > > Cells of a new Type?
> > > > > > > >
> > > > > > > > My first reaction is that KV#Type is particular to the KV
> > > > > > implementation.
> > > > > > > > Any new Cell implementation should not have to adopt the
> > KeyValue
> > > > > > typing
> > > > > > > > mechanism.
> > > > > > > >
> > > > > > > > S
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > --
> > > > > > > > > Chia-Ping
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Andrew
> > > > > > >
> > > > > > > Words like orphans lost among the crosstalk, meaning torn from
> > > > truth's
> > > > > > > decrepit hands
> > > > > > > - A23, Crosstalk
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew
> > > > >
> > > > > Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > > > decrepit hands
> > > > > - A23, Crosstalk
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > > - A23, Crosstalk
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Andrew Purtell <ap...@apache.org>.
Construct a normal put or delete or batch mutation, add whatever extra
state you need in one or more operation attributes, and use a
regionobserver to extend normal processing to handle the extra state. I'm
curious what dispatching to extension code because of a custom cell type
buys you over dispatching to extension code because of the presence of an
attribute (or cell tag). For example, in security coprocessors we take
attribute data and attach it to the cell using cell tags. Later we check
for cell tag(s) to determine if we have to take special action when the
cell is accessed by a scanner, or during some operations (e.g. appends or
increments have to do extra handling for cell security tags).
On Fri, Sep 29, 2017 at 2:43 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
> > Instead of a custom cell, could you use a regular cell with a custom
> > operation attribute (see OperationWithAttributes).
> Pardon me, I didn't get what you said.
>
>
>
> On 2017-09-30 04:31, Andrew Purtell <ap...@apache.org> wrote:
> > Instead of a custom cell, could you use a regular cell with a custom
> > operation attribute (see OperationWithAttributes).
> >
> > On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > The custom cell help us to save memory consumption. We don't have own
> > > serialization/deserialization mechanism, hence to transform data from
> > > client to server needs many conversion phase (user data -> Put/Cell ->
> pb
> > > object). The cost of conversion is large in transferring bulk data. In
> > > fact, we also have custom mutation to manage the memory usage of inner
> cell
> > > collection.
> > >
> > > On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
> > > > What are the use cases for a custom cell? It seems a dangerously low
> > > level
> > > > thing to attempt and perhaps we should unwind support for it. But
> perhaps
> > > > there is a compelling justification.
> > > >
> > > >
> > > > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <
> chia7712@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks for all comment.
> > > > >
> > > > > The problem i want to resolve is the valid code should be exposed
> as
> > > > > IA.Public. Otherwise, end user have to access the IA.Private class
> to
> > > build
> > > > > the custom cell.
> > > > >
> > > > > For example, I have a use case which plays a streaming role in our
> > > > > appliaction. It
> > > > > applies the CellBuilder(HBASE-18519) to build custom cells. These
> cells
> > > > > have many same fields so they are put in shared-memory for
> avoiding GC
> > > > > pause. Everything is wonderful. However, we have to access the
> > > IA.Private
> > > > > class - KeyValue#Type - to get the valid code of Put.
> > > > >
> > > > > I believe there are many use cases of custom cell, and
> consequently it
> > > is
> > > > > worth adding a way to get the valid type via IA.Public class.
> > > Otherwise, it
> > > > > may imply that the custom cell is based on a unstable way, because
> the
> > > > > related code can be changed at any time.
> > > > > --
> > > > > Chia-Ping
> > > > >
> > > > > On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > > > > I agree with Stack. Was typing up a reply to Anoop but let me
> move it
> > > > > down
> > > > > > here.
> > > > > >
> > > > > > The type code exposes some low level details of how our current
> > > stores
> > > > > are
> > > > > > architected. But what if in the future you could swap out HStore
> > > > > implements
> > > > > > Store with PStore implements Store, where HStore is backed by
> HFiles
> > > and
> > > > > > PStore is backed by Parquet? Just as a hypothetical example. I
> know
> > > there
> > > > > > would be larger issues if this were actually attempted. Bear with
> > > me. You
> > > > > > can imagine some different new Store implementation that has some
> > > > > > advantages but is not a design derived from the log structured
> merge
> > > tree
> > > > > > if you like. Most values from a new Cell.Type based on
> KeyValue.Type
> > > > > > wouldn't apply to cells from such a thing because they are
> > > particular to
> > > > > > how LSMs work. I'm sure such a project if attempted would make a
> > > number
> > > > > of
> > > > > > changes requiring a major version increment and low level details
> > > could
> > > > > be
> > > > > > unwound from Cell then, but if we could avoid doing it in the
> first
> > > > > place,
> > > > > > I think it would better for maintainability.
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > > > >
> > > > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > hi folks,
> > > > > > > >
> > > > > > > > User is allowed to create custom cell but the valid code of
> type
> > > -
> > > > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> should
> > > > > expose
> > > > > > > > KeyValue#Type as Public Client. Three possible ways are shown
> > > below:
> > > > > > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > > IA.Public
> > > > > > > > 2) Move KeyValue#Type into Cell.
> > > > > > > > 3) Move KeyValue#Type to upper level
> > > > > > > >
> > > > > > > > Any suggestions?
> > > > > > > >
> > > > > > > >
> > > > > > > What is the problem that we are trying to solve Chia-Ping? You
> > > want to
> > > > > make
> > > > > > > Cells of a new Type?
> > > > > > >
> > > > > > > My first reaction is that KV#Type is particular to the KV
> > > > > implementation.
> > > > > > > Any new Cell implementation should not have to adopt the
> KeyValue
> > > > > typing
> > > > > > > mechanism.
> > > > > > >
> > > > > > > S
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > Chia-Ping
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Andrew
> > > > > >
> > > > > > Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > > > decrepit hands
> > > > > > - A23, Crosstalk
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > > - A23, Crosstalk
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> > - A23, Crosstalk
> >
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
> Instead of a custom cell, could you use a regular cell with a custom
> operation attribute (see OperationWithAttributes).
Pardon me, I didn't get what you said.
On 2017-09-30 04:31, Andrew Purtell <ap...@apache.org> wrote:
> Instead of a custom cell, could you use a regular cell with a custom
> operation attribute (see OperationWithAttributes).
>
> On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > The custom cell help us to save memory consumption. We don't have own
> > serialization/deserialization mechanism, hence to transform data from
> > client to server needs many conversion phase (user data -> Put/Cell -> pb
> > object). The cost of conversion is large in transferring bulk data. In
> > fact, we also have custom mutation to manage the memory usage of inner cell
> > collection.
> >
> > On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
> > > What are the use cases for a custom cell? It seems a dangerously low
> > level
> > > thing to attempt and perhaps we should unwind support for it. But perhaps
> > > there is a compelling justification.
> > >
> > >
> > > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > >
> > > > Thanks for all comment.
> > > >
> > > > The problem i want to resolve is the valid code should be exposed as
> > > > IA.Public. Otherwise, end user have to access the IA.Private class to
> > build
> > > > the custom cell.
> > > >
> > > > For example, I have a use case which plays a streaming role in our
> > > > appliaction. It
> > > > applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> > > > have many same fields so they are put in shared-memory for avoiding GC
> > > > pause. Everything is wonderful. However, we have to access the
> > IA.Private
> > > > class - KeyValue#Type - to get the valid code of Put.
> > > >
> > > > I believe there are many use cases of custom cell, and consequently it
> > is
> > > > worth adding a way to get the valid type via IA.Public class.
> > Otherwise, it
> > > > may imply that the custom cell is based on a unstable way, because the
> > > > related code can be changed at any time.
> > > > --
> > > > Chia-Ping
> > > >
> > > > On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > > > I agree with Stack. Was typing up a reply to Anoop but let me move it
> > > > down
> > > > > here.
> > > > >
> > > > > The type code exposes some low level details of how our current
> > stores
> > > > are
> > > > > architected. But what if in the future you could swap out HStore
> > > > implements
> > > > > Store with PStore implements Store, where HStore is backed by HFiles
> > and
> > > > > PStore is backed by Parquet? Just as a hypothetical example. I know
> > there
> > > > > would be larger issues if this were actually attempted. Bear with
> > me. You
> > > > > can imagine some different new Store implementation that has some
> > > > > advantages but is not a design derived from the log structured merge
> > tree
> > > > > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > > > wouldn't apply to cells from such a thing because they are
> > particular to
> > > > > how LSMs work. I'm sure such a project if attempted would make a
> > number
> > > > of
> > > > > changes requiring a major version increment and low level details
> > could
> > > > be
> > > > > unwound from Cell then, but if we could avoid doing it in the first
> > > > place,
> > > > > I think it would better for maintainability.
> > > > >
> > > > >
> > > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > chia7712@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > hi folks,
> > > > > > >
> > > > > > > User is allowed to create custom cell but the valid code of type
> > -
> > > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> > > > expose
> > > > > > > KeyValue#Type as Public Client. Three possible ways are shown
> > below:
> > > > > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > IA.Public
> > > > > > > 2) Move KeyValue#Type into Cell.
> > > > > > > 3) Move KeyValue#Type to upper level
> > > > > > >
> > > > > > > Any suggestions?
> > > > > > >
> > > > > > >
> > > > > > What is the problem that we are trying to solve Chia-Ping? You
> > want to
> > > > make
> > > > > > Cells of a new Type?
> > > > > >
> > > > > > My first reaction is that KV#Type is particular to the KV
> > > > implementation.
> > > > > > Any new Cell implementation should not have to adopt the KeyValue
> > > > typing
> > > > > > mechanism.
> > > > > >
> > > > > > S
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > --
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew
> > > > >
> > > > > Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > > > decrepit hands
> > > > > - A23, Crosstalk
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > > - A23, Crosstalk
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Andrew Purtell <ap...@apache.org>.
Instead of a custom cell, could you use a regular cell with a custom
operation attribute (see OperationWithAttributes).
On Fri, Sep 29, 2017 at 1:28 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
> The custom cell help us to save memory consumption. We don't have own
> serialization/deserialization mechanism, hence to transform data from
> client to server needs many conversion phase (user data -> Put/Cell -> pb
> object). The cost of conversion is large in transferring bulk data. In
> fact, we also have custom mutation to manage the memory usage of inner cell
> collection.
>
> On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
> > What are the use cases for a custom cell? It seems a dangerously low
> level
> > thing to attempt and perhaps we should unwind support for it. But perhaps
> > there is a compelling justification.
> >
> >
> > On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> >
> > > Thanks for all comment.
> > >
> > > The problem i want to resolve is the valid code should be exposed as
> > > IA.Public. Otherwise, end user have to access the IA.Private class to
> build
> > > the custom cell.
> > >
> > > For example, I have a use case which plays a streaming role in our
> > > appliaction. It
> > > applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> > > have many same fields so they are put in shared-memory for avoiding GC
> > > pause. Everything is wonderful. However, we have to access the
> IA.Private
> > > class - KeyValue#Type - to get the valid code of Put.
> > >
> > > I believe there are many use cases of custom cell, and consequently it
> is
> > > worth adding a way to get the valid type via IA.Public class.
> Otherwise, it
> > > may imply that the custom cell is based on a unstable way, because the
> > > related code can be changed at any time.
> > > --
> > > Chia-Ping
> > >
> > > On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > > I agree with Stack. Was typing up a reply to Anoop but let me move it
> > > down
> > > > here.
> > > >
> > > > The type code exposes some low level details of how our current
> stores
> > > are
> > > > architected. But what if in the future you could swap out HStore
> > > implements
> > > > Store with PStore implements Store, where HStore is backed by HFiles
> and
> > > > PStore is backed by Parquet? Just as a hypothetical example. I know
> there
> > > > would be larger issues if this were actually attempted. Bear with
> me. You
> > > > can imagine some different new Store implementation that has some
> > > > advantages but is not a design derived from the log structured merge
> tree
> > > > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > > wouldn't apply to cells from such a thing because they are
> particular to
> > > > how LSMs work. I'm sure such a project if attempted would make a
> number
> > > of
> > > > changes requiring a major version increment and low level details
> could
> > > be
> > > > unwound from Cell then, but if we could avoid doing it in the first
> > > place,
> > > > I think it would better for maintainability.
> > > >
> > > >
> > > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> chia7712@apache.org>
> > > > > wrote:
> > > > >
> > > > > > hi folks,
> > > > > >
> > > > > > User is allowed to create custom cell but the valid code of type
> -
> > > > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> > > expose
> > > > > > KeyValue#Type as Public Client. Three possible ways are shown
> below:
> > > > > > 1) Change declaration of KeyValue#Type from IA.Private to
> IA.Public
> > > > > > 2) Move KeyValue#Type into Cell.
> > > > > > 3) Move KeyValue#Type to upper level
> > > > > >
> > > > > > Any suggestions?
> > > > > >
> > > > > >
> > > > > What is the problem that we are trying to solve Chia-Ping? You
> want to
> > > make
> > > > > Cells of a new Type?
> > > > >
> > > > > My first reaction is that KV#Type is particular to the KV
> > > implementation.
> > > > > Any new Cell implementation should not have to adopt the KeyValue
> > > typing
> > > > > mechanism.
> > > > >
> > > > > S
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > --
> > > > > > Chia-Ping
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > > - A23, Crosstalk
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> > - A23, Crosstalk
> >
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
The custom cell help us to save memory consumption. We don't have own serialization/deserialization mechanism, hence to transform data from client to server needs many conversion phase (user data -> Put/Cell -> pb object). The cost of conversion is large in transferring bulk data. In fact, we also have custom mutation to manage the memory usage of inner cell collection.
On 2017-09-30 02:43, Andrew Purtell <ap...@apache.org> wrote:
> What are the use cases for a custom cell? It seems a dangerously low level
> thing to attempt and perhaps we should unwind support for it. But perhaps
> there is a compelling justification.
>
>
> On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
> > Thanks for all comment.
> >
> > The problem i want to resolve is the valid code should be exposed as
> > IA.Public. Otherwise, end user have to access the IA.Private class to build
> > the custom cell.
> >
> > For example, I have a use case which plays a streaming role in our
> > appliaction. It
> > applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> > have many same fields so they are put in shared-memory for avoiding GC
> > pause. Everything is wonderful. However, we have to access the IA.Private
> > class - KeyValue#Type - to get the valid code of Put.
> >
> > I believe there are many use cases of custom cell, and consequently it is
> > worth adding a way to get the valid type via IA.Public class. Otherwise, it
> > may imply that the custom cell is based on a unstable way, because the
> > related code can be changed at any time.
> > --
> > Chia-Ping
> >
> > On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > I agree with Stack. Was typing up a reply to Anoop but let me move it
> > down
> > > here.
> > >
> > > The type code exposes some low level details of how our current stores
> > are
> > > architected. But what if in the future you could swap out HStore
> > implements
> > > Store with PStore implements Store, where HStore is backed by HFiles and
> > > PStore is backed by Parquet? Just as a hypothetical example. I know there
> > > would be larger issues if this were actually attempted. Bear with me. You
> > > can imagine some different new Store implementation that has some
> > > advantages but is not a design derived from the log structured merge tree
> > > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > wouldn't apply to cells from such a thing because they are particular to
> > > how LSMs work. I'm sure such a project if attempted would make a number
> > of
> > > changes requiring a major version increment and low level details could
> > be
> > > unwound from Cell then, but if we could avoid doing it in the first
> > place,
> > > I think it would better for maintainability.
> > >
> > >
> > > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
> > > > wrote:
> > > >
> > > > > hi folks,
> > > > >
> > > > > User is allowed to create custom cell but the valid code of type -
> > > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> > expose
> > > > > KeyValue#Type as Public Client. Three possible ways are shown below:
> > > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > > > > 2) Move KeyValue#Type into Cell.
> > > > > 3) Move KeyValue#Type to upper level
> > > > >
> > > > > Any suggestions?
> > > > >
> > > > >
> > > > What is the problem that we are trying to solve Chia-Ping? You want to
> > make
> > > > Cells of a new Type?
> > > >
> > > > My first reaction is that KV#Type is particular to the KV
> > implementation.
> > > > Any new Cell implementation should not have to adopt the KeyValue
> > typing
> > > > mechanism.
> > > >
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > > > --
> > > > > Chia-Ping
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > > - A23, Crosstalk
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
[HBASE-14850]How compile simple-client with CMake?
Posted by Andrzej <bo...@wp.pl>.
How is best way to write CMakeLists.txt of simple client with all
dependencies? There ale many modules, files compilated by protoc, etc.
Re: [HBASE-14850]Will shared object client as task in near future?
Posted by Ted Yu <yu...@gmail.com>.
Design and implementation of HBASE-14850 project follows C++ (11/14)
standard.
Please design your client application accordingly.
Cheers
On Sun, Oct 1, 2017 at 6:25 AM, Andrzej <bo...@wp.pl> wrote:
> W dniu 30.09.2017 o 14:51, Ted Yu pisze:
>
>> bq. smart pointers, which hinders simple pointers values as arguments and
>> return values
>>
>> I don't quite get what you meant. Can you elaborate ?
>>
>
> My trial exported function:
>
> Table *soGetTable(Client *client, char *name)
> {
> //name ==> tn
> return client->Table(*tn).get(); //<---free pointer here
> }
>
>
> pointer will be freed because is smart and soGetTable returns bad pointer.
>
Re: [HBASE-14850]Will shared object client as task in near future?
Posted by Andrzej <bo...@wp.pl>.
W dniu 30.09.2017 o 14:51, Ted Yu pisze:
> bq. smart pointers, which hinders simple pointers values as arguments and
> return values
>
> I don't quite get what you meant. Can you elaborate ?
My trial exported function:
Table *soGetTable(Client *client, char *name)
{
//name ==> tn
return client->Table(*tn).get(); //<---free pointer here
}
pointer will be freed because is smart and soGetTable returns bad pointer.
Re: [HBASE-14850]Will shared object client as task in near future?
Posted by Ted Yu <yu...@gmail.com>.
Please watch HBASE-18901 <https://issues.apache.org/jira/browse/HBASE-18901>
.
libHBaseClient.a and libHBaseClient.so can be generated outside docker.
bq. smart pointers, which hinders simple pointers values as arguments and
return values
I don't quite get what you meant. Can you elaborate ?
Thanks
On Sat, Sep 30, 2017 at 12:58 AM, Andrzej <bo...@wp.pl> wrote:
> It is necessary shared library to fast communicate with external world.
> Shared library would encapsulate all HBase possibilities and could be used
> in big external project, compiled outside HBase docker with many other
> dependecies.
> There are two problems:
> - smart pointers, which hinders simple pointers values as arguments and
> return values
> - all code must be PIC, and is problem with libevent.a
>
[HBASE-14850]Will shared object client as task in near future?
Posted by Andrzej <bo...@wp.pl>.
It is necessary shared library to fast communicate with external world.
Shared library would encapsulate all HBase possibilities and could be
used in big external project, compiled outside HBase docker with many
other dependecies.
There are two problems:
- smart pointers, which hinders simple pointers values as arguments and
return values
- all code must be PIC, and is problem with libevent.a
Re: [DISCUSS] Move Type out of KeyValue
Posted by Andrew Purtell <ap...@apache.org>.
What are the use cases for a custom cell? It seems a dangerously low level
thing to attempt and perhaps we should unwind support for it. But perhaps
there is a compelling justification.
On Thu, Sep 28, 2017 at 10:20 PM, Chia-Ping Tsai <ch...@apache.org>
wrote:
> Thanks for all comment.
>
> The problem i want to resolve is the valid code should be exposed as
> IA.Public. Otherwise, end user have to access the IA.Private class to build
> the custom cell.
>
> For example, I have a use case which plays a streaming role in our
> appliaction. It
> applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> have many same fields so they are put in shared-memory for avoiding GC
> pause. Everything is wonderful. However, we have to access the IA.Private
> class - KeyValue#Type - to get the valid code of Put.
>
> I believe there are many use cases of custom cell, and consequently it is
> worth adding a way to get the valid type via IA.Public class. Otherwise, it
> may imply that the custom cell is based on a unstable way, because the
> related code can be changed at any time.
> --
> Chia-Ping
>
> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > I agree with Stack. Was typing up a reply to Anoop but let me move it
> down
> > here.
> >
> > The type code exposes some low level details of how our current stores
> are
> > architected. But what if in the future you could swap out HStore
> implements
> > Store with PStore implements Store, where HStore is backed by HFiles and
> > PStore is backed by Parquet? Just as a hypothetical example. I know there
> > would be larger issues if this were actually attempted. Bear with me. You
> > can imagine some different new Store implementation that has some
> > advantages but is not a design derived from the log structured merge tree
> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > wouldn't apply to cells from such a thing because they are particular to
> > how LSMs work. I'm sure such a project if attempted would make a number
> of
> > changes requiring a major version increment and low level details could
> be
> > unwound from Cell then, but if we could avoid doing it in the first
> place,
> > I think it would better for maintainability.
> >
> >
> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > >
> > > > hi folks,
> > > >
> > > > User is allowed to create custom cell but the valid code of type -
> > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> expose
> > > > KeyValue#Type as Public Client. Three possible ways are shown below:
> > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > > > 2) Move KeyValue#Type into Cell.
> > > > 3) Move KeyValue#Type to upper level
> > > >
> > > > Any suggestions?
> > > >
> > > >
> > > What is the problem that we are trying to solve Chia-Ping? You want to
> make
> > > Cells of a new Type?
> > >
> > > My first reaction is that KV#Type is particular to the KV
> implementation.
> > > Any new Cell implementation should not have to adopt the KeyValue
> typing
> > > mechanism.
> > >
> > > S
> > >
> > >
> > >
> > >
> > > > --
> > > > Chia-Ping
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> > - A23, Crosstalk
> >
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
I don't want to impact the Cell...Maybe the new class "CellType" should be added into CellBuilderType.
The Cell#getTypeByte() won't be changed.
On 2017-10-02 21:50, Ted Yu <yu...@gmail.com> wrote:
> If CellType is added, should the following method of Cell be affected ?
>
> /**
>
> * @return The byte representation of the KeyValue.TYPE of this cell: one
> of Put, Delete, etc
>
> */
>
> byte getTypeByte();
>
> Ideally CellType should be returned instead of a byte.
>
> On Mon, Oct 2, 2017 at 1:54 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > How about introducing an new enum "CellType" which is subset of
> > KeyValue#Type? It will be exposed as IA.Public to end user for helping
> > build the custom cell (via CellBuilder). The types which "CellType" should
> > have are shown below.
> > 1) Put
> > 2) Delete
> > 3) DeleteFamilyVersion
> > 4) DeleteColumn
> > 5) DeleteFamily
> > Hence, the CellBuilder#setType(byte) will be replaced by
> > CellBuilder#setType(CellType). Our internal use still reference to
> > KeyValue#Type.
> >
> >
> > On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> > > Ya as Chia-Ping said, the problem he is trying to solve is very basic
> > > one. As long as we allow custom Cell creation (Via CellBuilder API)
> > > and allow Mutations to be added with Cells and pass that from client
> > > side APIs, we have to make the Type public accessible.
> > > Or else the Cell building APIs should not be taking in a type byte.
> > > We have to some way allow user to make put/delete cells stc.
> > >
> > > Is type that bound for only KV? We have getType in Cell also right?
> > > The type in full form what we have in KV now, may be making us confuse
> > > here? As Ram said it contains some internal types also which the user
> > > has never to know abt. Pls correct if saying in wrong way.
> > >
> > > Good that Chia-Ping brought this out here. We have to either way
> > > solve it and make the public API fully public.
> > >
> > > -Anoop-
> > >
> > > On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> > > <ra...@gmail.com> wrote:
> > > > Even if we are trying to move out I think only few of the types are
> > really
> > > > user readable. So we should be very careful here. So since we have
> > > > CellBuilder way it is better we check what type of cells a user can
> > build.
> > > > I think for now the Cellbuilder is not client exposed?
> > > > But again moving to Cell means it becomes public which is not right
> > IMO and
> > > > I thinks others here also agree to it.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > > > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks for all comment.
> > > >>
> > > >> The problem i want to resolve is the valid code should be exposed as
> > > >> IA.Public. Otherwise, end user have to access the IA.Private class to
> > build
> > > >> the custom cell.
> > > >>
> > > >> For example, I have a use case which plays a streaming role in our
> > > >> appliaction. It
> > > >> applies the CellBuilder(HBASE-18519) to build custom cells. These
> > cells
> > > >> have many same fields so they are put in shared-memory for avoiding GC
> > > >> pause. Everything is wonderful. However, we have to access the
> > IA.Private
> > > >> class - KeyValue#Type - to get the valid code of Put.
> > > >>
> > > >> I believe there are many use cases of custom cell, and consequently
> > it is
> > > >> worth adding a way to get the valid type via IA.Public class.
> > Otherwise, it
> > > >> may imply that the custom cell is based on a unstable way, because the
> > > >> related code can be changed at any time.
> > > >> --
> > > >> Chia-Ping
> > > >>
> > > >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > >> > I agree with Stack. Was typing up a reply to Anoop but let me move
> > it
> > > >> down
> > > >> > here.
> > > >> >
> > > >> > The type code exposes some low level details of how our current
> > stores
> > > >> are
> > > >> > architected. But what if in the future you could swap out HStore
> > > >> implements
> > > >> > Store with PStore implements Store, where HStore is backed by
> > HFiles and
> > > >> > PStore is backed by Parquet? Just as a hypothetical example. I know
> > there
> > > >> > would be larger issues if this were actually attempted. Bear with
> > me. You
> > > >> > can imagine some different new Store implementation that has some
> > > >> > advantages but is not a design derived from the log structured
> > merge tree
> > > >> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > >> > wouldn't apply to cells from such a thing because they are
> > particular to
> > > >> > how LSMs work. I'm sure such a project if attempted would make a
> > number
> > > >> of
> > > >> > changes requiring a major version increment and low level details
> > could
> > > >> be
> > > >> > unwound from Cell then, but if we could avoid doing it in the first
> > > >> place,
> > > >> > I think it would better for maintainability.
> > > >> >
> > > >> >
> > > >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > >> >
> > > >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > chia7712@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > hi folks,
> > > >> > > >
> > > >> > > > User is allowed to create custom cell but the valid code of
> > type -
> > > >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> > should
> > > >> expose
> > > >> > > > KeyValue#Type as Public Client. Three possible ways are shown
> > below:
> > > >> > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > IA.Public
> > > >> > > > 2) Move KeyValue#Type into Cell.
> > > >> > > > 3) Move KeyValue#Type to upper level
> > > >> > > >
> > > >> > > > Any suggestions?
> > > >> > > >
> > > >> > > >
> > > >> > > What is the problem that we are trying to solve Chia-Ping? You
> > want to
> > > >> make
> > > >> > > Cells of a new Type?
> > > >> > >
> > > >> > > My first reaction is that KV#Type is particular to the KV
> > > >> implementation.
> > > >> > > Any new Cell implementation should not have to adopt the KeyValue
> > > >> typing
> > > >> > > mechanism.
> > > >> > >
> > > >> > > S
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > > --
> > > >> > > > Chia-Ping
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Best regards,
> > > >> > Andrew
> > > >> >
> > > >> > Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > >> > decrepit hands
> > > >> > - A23, Crosstalk
> > > >> >
> > > >>
> > >
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Ted Yu <yu...@gmail.com>.
If CellType is added, should the following method of Cell be affected ?
/**
* @return The byte representation of the KeyValue.TYPE of this cell: one
of Put, Delete, etc
*/
byte getTypeByte();
Ideally CellType should be returned instead of a byte.
On Mon, Oct 2, 2017 at 1:54 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
> How about introducing an new enum "CellType" which is subset of
> KeyValue#Type? It will be exposed as IA.Public to end user for helping
> build the custom cell (via CellBuilder). The types which "CellType" should
> have are shown below.
> 1) Put
> 2) Delete
> 3) DeleteFamilyVersion
> 4) DeleteColumn
> 5) DeleteFamily
> Hence, the CellBuilder#setType(byte) will be replaced by
> CellBuilder#setType(CellType). Our internal use still reference to
> KeyValue#Type.
>
>
> On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> > Ya as Chia-Ping said, the problem he is trying to solve is very basic
> > one. As long as we allow custom Cell creation (Via CellBuilder API)
> > and allow Mutations to be added with Cells and pass that from client
> > side APIs, we have to make the Type public accessible.
> > Or else the Cell building APIs should not be taking in a type byte.
> > We have to some way allow user to make put/delete cells stc.
> >
> > Is type that bound for only KV? We have getType in Cell also right?
> > The type in full form what we have in KV now, may be making us confuse
> > here? As Ram said it contains some internal types also which the user
> > has never to know abt. Pls correct if saying in wrong way.
> >
> > Good that Chia-Ping brought this out here. We have to either way
> > solve it and make the public API fully public.
> >
> > -Anoop-
> >
> > On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> > <ra...@gmail.com> wrote:
> > > Even if we are trying to move out I think only few of the types are
> really
> > > user readable. So we should be very careful here. So since we have
> > > CellBuilder way it is better we check what type of cells a user can
> build.
> > > I think for now the Cellbuilder is not client exposed?
> > > But again moving to Cell means it becomes public which is not right
> IMO and
> > > I thinks others here also agree to it.
> > >
> > > Regards
> > > Ram
> > >
> > > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > >
> > >> Thanks for all comment.
> > >>
> > >> The problem i want to resolve is the valid code should be exposed as
> > >> IA.Public. Otherwise, end user have to access the IA.Private class to
> build
> > >> the custom cell.
> > >>
> > >> For example, I have a use case which plays a streaming role in our
> > >> appliaction. It
> > >> applies the CellBuilder(HBASE-18519) to build custom cells. These
> cells
> > >> have many same fields so they are put in shared-memory for avoiding GC
> > >> pause. Everything is wonderful. However, we have to access the
> IA.Private
> > >> class - KeyValue#Type - to get the valid code of Put.
> > >>
> > >> I believe there are many use cases of custom cell, and consequently
> it is
> > >> worth adding a way to get the valid type via IA.Public class.
> Otherwise, it
> > >> may imply that the custom cell is based on a unstable way, because the
> > >> related code can be changed at any time.
> > >> --
> > >> Chia-Ping
> > >>
> > >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > >> > I agree with Stack. Was typing up a reply to Anoop but let me move
> it
> > >> down
> > >> > here.
> > >> >
> > >> > The type code exposes some low level details of how our current
> stores
> > >> are
> > >> > architected. But what if in the future you could swap out HStore
> > >> implements
> > >> > Store with PStore implements Store, where HStore is backed by
> HFiles and
> > >> > PStore is backed by Parquet? Just as a hypothetical example. I know
> there
> > >> > would be larger issues if this were actually attempted. Bear with
> me. You
> > >> > can imagine some different new Store implementation that has some
> > >> > advantages but is not a design derived from the log structured
> merge tree
> > >> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > >> > wouldn't apply to cells from such a thing because they are
> particular to
> > >> > how LSMs work. I'm sure such a project if attempted would make a
> number
> > >> of
> > >> > changes requiring a major version increment and low level details
> could
> > >> be
> > >> > unwound from Cell then, but if we could avoid doing it in the first
> > >> place,
> > >> > I think it would better for maintainability.
> > >> >
> > >> >
> > >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > >> >
> > >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> chia7712@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > hi folks,
> > >> > > >
> > >> > > > User is allowed to create custom cell but the valid code of
> type -
> > >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> should
> > >> expose
> > >> > > > KeyValue#Type as Public Client. Three possible ways are shown
> below:
> > >> > > > 1) Change declaration of KeyValue#Type from IA.Private to
> IA.Public
> > >> > > > 2) Move KeyValue#Type into Cell.
> > >> > > > 3) Move KeyValue#Type to upper level
> > >> > > >
> > >> > > > Any suggestions?
> > >> > > >
> > >> > > >
> > >> > > What is the problem that we are trying to solve Chia-Ping? You
> want to
> > >> make
> > >> > > Cells of a new Type?
> > >> > >
> > >> > > My first reaction is that KV#Type is particular to the KV
> > >> implementation.
> > >> > > Any new Cell implementation should not have to adopt the KeyValue
> > >> typing
> > >> > > mechanism.
> > >> > >
> > >> > > S
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > > --
> > >> > > > Chia-Ping
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Best regards,
> > >> > Andrew
> > >> >
> > >> > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > >> > decrepit hands
> > >> > - A23, Crosstalk
> > >> >
> > >>
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Stack <st...@duboce.net>.
On Mon, Oct 2, 2017 at 11:46 PM, Chia-Ping Tsai <ch...@apache.org> wrote:
> ...
>
> > We'd still be stuck at read time when all we had was Cell#getTypeByte
> > returning a byte.
> >
> > We could add a CellType with the KV#Type static utility codeToType but
> > would be sweet if we could do without having to explain the byte to
> users.
> It seems to me that Cell is a general interface for kv format and others
> (not happen now), but we design the Cell's APIs according to the KV format.
> The cell's type should not be exposed to end user - I mean the
> Cell#getTypeByte should be removed - because it is hard to say other
> formats have the type field.
Or they may chose to implement typing differently; i.e. not via a dedicated
byte.
So, yes, Cell#getTypeByte is problematic in our Cell interface. We should
at least note its problems in the Interface so some future implementer who
has not seen this conversation doesn't get hung up on it.
S
The KV should have own interface which used in HRegion/HStore/HStoreFIle.
> The ExtendedCell is a good candidate because i don't think we need to mix
> together the kv format and other format in the same table/region/file. For
> normal user, all cells they get via Scan/Get are real Cell (not Delete mark
> or other type), so the type API is redundant. For advanced user, they can
> get the IA.LimitedPrivate Cell, such as KVInterface, via raw scan or other
> advanced operations.
>
> Cell(IA.Public) <---- KVInterface(IA.LimitedPrivate) <--- KeyValue,
> Bufferedxxx (IA.Private)
> <---- XXXInterface(IA.LimitedPrivate) <--- xxx
> cell (IA.Private)
>
>
>
> On 2017-10-03 13:29, Stack <st...@duboce.net> wrote:
> > On Mon, Oct 2, 2017 at 1:54 AM, Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > How about introducing an new enum "CellType" which is subset of
> > > KeyValue#Type? It will be exposed as IA.Public to end user for helping
> > > build the custom cell (via CellBuilder). The types which "CellType"
> should
> > > have are shown below.
> > > 1) Put
> > > 2) Delete
> > > 3) DeleteFamilyVersion
> > > 4) DeleteColumn
> > > 5) DeleteFamily
> > > Hence, the CellBuilder#setType(byte) will be replaced by
> > > CellBuilder#setType(CellType). Our internal use still reference to
> > > KeyValue#Type.
> > >
> > >
> > There is a 'hole' in our Cell Interface; we have a getTypeByte but expose
> > no means of signaling what the byte stands for.
> >
> > You could add to CellBuilder methods to do setPutType, setDeleteType,
> etc.,
> > which would work for build time allowing you could keep Type byte
> private.
> > This would seem to be enough for your case Chia-Ping?
> >
> > We'd still be stuck at read time when all we had was Cell#getTypeByte
> > returning a byte.
> >
> > We could add a CellType with the KV#Type static utility codeToType but
> > would be sweet if we could do without having to explain the byte to
> users.
> >
> > S
> >
> >
> >
> >
> >
> >
> >
> > >
> > > On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> > > > Ya as Chia-Ping said, the problem he is trying to solve is very basic
> > > > one. As long as we allow custom Cell creation (Via CellBuilder API)
> > > > and allow Mutations to be added with Cells and pass that from client
> > > > side APIs, we have to make the Type public accessible.
> > > > Or else the Cell building APIs should not be taking in a type byte.
> > > > We have to some way allow user to make put/delete cells stc.
> > > >
> > > > Is type that bound for only KV? We have getType in Cell also right?
> > > > The type in full form what we have in KV now, may be making us
> confuse
> > > > here? As Ram said it contains some internal types also which the
> user
> > > > has never to know abt. Pls correct if saying in wrong way.
> > > >
> > > > Good that Chia-Ping brought this out here. We have to either way
> > > > solve it and make the public API fully public.
> > > >
> > > > -Anoop-
> > > >
> > > > On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> > > > <ra...@gmail.com> wrote:
> > > > > Even if we are trying to move out I think only few of the types are
> > > really
> > > > > user readable. So we should be very careful here. So since we have
> > > > > CellBuilder way it is better we check what type of cells a user can
> > > build.
> > > > > I think for now the Cellbuilder is not client exposed?
> > > > > But again moving to Cell means it becomes public which is not right
> > > IMO and
> > > > > I thinks others here also agree to it.
> > > > >
> > > > > Regards
> > > > > Ram
> > > > >
> > > > > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <
> chia7712@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Thanks for all comment.
> > > > >>
> > > > >> The problem i want to resolve is the valid code should be exposed
> as
> > > > >> IA.Public. Otherwise, end user have to access the IA.Private
> class to
> > > build
> > > > >> the custom cell.
> > > > >>
> > > > >> For example, I have a use case which plays a streaming role in our
> > > > >> appliaction. It
> > > > >> applies the CellBuilder(HBASE-18519) to build custom cells. These
> > > cells
> > > > >> have many same fields so they are put in shared-memory for
> avoiding GC
> > > > >> pause. Everything is wonderful. However, we have to access the
> > > IA.Private
> > > > >> class - KeyValue#Type - to get the valid code of Put.
> > > > >>
> > > > >> I believe there are many use cases of custom cell, and
> consequently
> > > it is
> > > > >> worth adding a way to get the valid type via IA.Public class.
> > > Otherwise, it
> > > > >> may imply that the custom cell is based on a unstable way,
> because the
> > > > >> related code can be changed at any time.
> > > > >> --
> > > > >> Chia-Ping
> > > > >>
> > > > >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > > >> > I agree with Stack. Was typing up a reply to Anoop but let me
> move
> > > it
> > > > >> down
> > > > >> > here.
> > > > >> >
> > > > >> > The type code exposes some low level details of how our current
> > > stores
> > > > >> are
> > > > >> > architected. But what if in the future you could swap out HStore
> > > > >> implements
> > > > >> > Store with PStore implements Store, where HStore is backed by
> > > HFiles and
> > > > >> > PStore is backed by Parquet? Just as a hypothetical example. I
> know
> > > there
> > > > >> > would be larger issues if this were actually attempted. Bear
> with
> > > me. You
> > > > >> > can imagine some different new Store implementation that has
> some
> > > > >> > advantages but is not a design derived from the log structured
> > > merge tree
> > > > >> > if you like. Most values from a new Cell.Type based on
> KeyValue.Type
> > > > >> > wouldn't apply to cells from such a thing because they are
> > > particular to
> > > > >> > how LSMs work. I'm sure such a project if attempted would make a
> > > number
> > > > >> of
> > > > >> > changes requiring a major version increment and low level
> details
> > > could
> > > > >> be
> > > > >> > unwound from Cell then, but if we could avoid doing it in the
> first
> > > > >> place,
> > > > >> > I think it would better for maintainability.
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net>
> wrote:
> > > > >> >
> > > > >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > hi folks,
> > > > >> > > >
> > > > >> > > > User is allowed to create custom cell but the valid code of
> > > type -
> > > > >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> > > should
> > > > >> expose
> > > > >> > > > KeyValue#Type as Public Client. Three possible ways are
> shown
> > > below:
> > > > >> > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > > IA.Public
> > > > >> > > > 2) Move KeyValue#Type into Cell.
> > > > >> > > > 3) Move KeyValue#Type to upper level
> > > > >> > > >
> > > > >> > > > Any suggestions?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > What is the problem that we are trying to solve Chia-Ping? You
> > > want to
> > > > >> make
> > > > >> > > Cells of a new Type?
> > > > >> > >
> > > > >> > > My first reaction is that KV#Type is particular to the KV
> > > > >> implementation.
> > > > >> > > Any new Cell implementation should not have to adopt the
> KeyValue
> > > > >> typing
> > > > >> > > mechanism.
> > > > >> > >
> > > > >> > > S
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > > --
> > > > >> > > > Chia-Ping
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Best regards,
> > > > >> > Andrew
> > > > >> >
> > > > >> > Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > >> > decrepit hands
> > > > >> > - A23, Crosstalk
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
(reply again...)
> You could add to CellBuilder methods to do setPutType, setDeleteType, etc.,
> which would work for build time allowing you could keep Type byte private.
> This would seem to be enough for your case Chia-Ping?
Yep, i will file a issue to address it.
> We could add a CellType with the KV#Type static utility codeToType but
> would be sweet if we could do without having to explain the byte to users.
It seems to me that Cell is a general interface for kv format and others (not happen now), but we design the Cell's APIs according to the KV format. The cell's type should not be exposed to end user - I mean the Cell#getTypeByte should be removed - because it is hard to say other formats have the type field. The KV should have own interface which used in HRegion/HStore/HStoreFIle. The ExtendedCell is a good candidate because i don't think we need to mix together the kv format and other format in the same table/region/file. For normal user, all cells they get via Scan/Get are real Cell (not Delete mark or other type), so the type API is redundant. For advanced user, they can get the IA.LimitedPrivate Cell, such as KVInterface, via raw scan or other advanced operations.
Cell(IA.Public) <---- KVInterface(IA.LimitedPrivate) <--- KeyValue, Bufferedxxx (IA.Private)
<---- XXXInterface(IA.LimitedPrivate) <--- xxx cell (IA.Private)
On 2017-10-03 14:46, "Chia-Ping Tsai"<ch...@apache.org> wrote:
> > You could add to CellBuilder methods to do setPutType, setDeleteType, etc.,
> > which would work for build time allowing you could keep Type byte private.
> > This would seem to be enough for your case Chia-Ping?
> Yep, i will file a issue to address it.
>
> > We'd still be stuck at read time when all we had was Cell#getTypeByte
> > returning a byte.
> >
> > We could add a CellType with the KV#Type static utility codeToType but
> > would be sweet if we could do without having to explain the byte to users.
> It seems to me that Cell is a general interface for kv format and others (not happen now), but we design the Cell's APIs according to the KV format. The cell's type should not be exposed to end user - I mean the Cell#getTypeByte should be removed - because it is hard to say other formats have the type field. The KV should have own interface which used in HRegion/HStore/HStoreFIle. The ExtendedCell is a good candidate because i don't think we need to mix together the kv format and other format in the same table/region/file. For normal user, all cells they get via Scan/Get are real Cell (not Delete mark or other type), so the type API is redundant. For advanced user, they can get the IA.LimitedPrivate Cell, such as KVInterface, via raw scan or other advanced operations.
>
> Cell(IA.Public) <---- KVInterface(IA.LimitedPrivate) <--- KeyValue, Bufferedxxx (IA.Private)
> <---- XXXInterface(IA.LimitedPrivate) <--- xxx cell (IA.Private)
>
>
> On 2017-10-03 13:29, Stack <st...@duboce.net> wrote:
> > On Mon, Oct 2, 2017 at 1:54 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
> >
> > > How about introducing an new enum "CellType" which is subset of
> > > KeyValue#Type? It will be exposed as IA.Public to end user for helping
> > > build the custom cell (via CellBuilder). The types which "CellType" should
> > > have are shown below.
> > > 1) Put
> > > 2) Delete
> > > 3) DeleteFamilyVersion
> > > 4) DeleteColumn
> > > 5) DeleteFamily
> > > Hence, the CellBuilder#setType(byte) will be replaced by
> > > CellBuilder#setType(CellType). Our internal use still reference to
> > > KeyValue#Type.
> > >
> > >
> > There is a 'hole' in our Cell Interface; we have a getTypeByte but expose
> > no means of signaling what the byte stands for.
> >
> > You could add to CellBuilder methods to do setPutType, setDeleteType, etc.,
> > which would work for build time allowing you could keep Type byte private.
> > This would seem to be enough for your case Chia-Ping?
> >
> > We'd still be stuck at read time when all we had was Cell#getTypeByte
> > returning a byte.
> >
> > We could add a CellType with the KV#Type static utility codeToType but
> > would be sweet if we could do without having to explain the byte to users.
> >
> > S
> >
> >
> >
> >
> >
> >
> >
> > >
> > > On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> > > > Ya as Chia-Ping said, the problem he is trying to solve is very basic
> > > > one. As long as we allow custom Cell creation (Via CellBuilder API)
> > > > and allow Mutations to be added with Cells and pass that from client
> > > > side APIs, we have to make the Type public accessible.
> > > > Or else the Cell building APIs should not be taking in a type byte.
> > > > We have to some way allow user to make put/delete cells stc.
> > > >
> > > > Is type that bound for only KV? We have getType in Cell also right?
> > > > The type in full form what we have in KV now, may be making us confuse
> > > > here? As Ram said it contains some internal types also which the user
> > > > has never to know abt. Pls correct if saying in wrong way.
> > > >
> > > > Good that Chia-Ping brought this out here. We have to either way
> > > > solve it and make the public API fully public.
> > > >
> > > > -Anoop-
> > > >
> > > > On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> > > > <ra...@gmail.com> wrote:
> > > > > Even if we are trying to move out I think only few of the types are
> > > really
> > > > > user readable. So we should be very careful here. So since we have
> > > > > CellBuilder way it is better we check what type of cells a user can
> > > build.
> > > > > I think for now the Cellbuilder is not client exposed?
> > > > > But again moving to Cell means it becomes public which is not right
> > > IMO and
> > > > > I thinks others here also agree to it.
> > > > >
> > > > > Regards
> > > > > Ram
> > > > >
> > > > > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> Thanks for all comment.
> > > > >>
> > > > >> The problem i want to resolve is the valid code should be exposed as
> > > > >> IA.Public. Otherwise, end user have to access the IA.Private class to
> > > build
> > > > >> the custom cell.
> > > > >>
> > > > >> For example, I have a use case which plays a streaming role in our
> > > > >> appliaction. It
> > > > >> applies the CellBuilder(HBASE-18519) to build custom cells. These
> > > cells
> > > > >> have many same fields so they are put in shared-memory for avoiding GC
> > > > >> pause. Everything is wonderful. However, we have to access the
> > > IA.Private
> > > > >> class - KeyValue#Type - to get the valid code of Put.
> > > > >>
> > > > >> I believe there are many use cases of custom cell, and consequently
> > > it is
> > > > >> worth adding a way to get the valid type via IA.Public class.
> > > Otherwise, it
> > > > >> may imply that the custom cell is based on a unstable way, because the
> > > > >> related code can be changed at any time.
> > > > >> --
> > > > >> Chia-Ping
> > > > >>
> > > > >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > > >> > I agree with Stack. Was typing up a reply to Anoop but let me move
> > > it
> > > > >> down
> > > > >> > here.
> > > > >> >
> > > > >> > The type code exposes some low level details of how our current
> > > stores
> > > > >> are
> > > > >> > architected. But what if in the future you could swap out HStore
> > > > >> implements
> > > > >> > Store with PStore implements Store, where HStore is backed by
> > > HFiles and
> > > > >> > PStore is backed by Parquet? Just as a hypothetical example. I know
> > > there
> > > > >> > would be larger issues if this were actually attempted. Bear with
> > > me. You
> > > > >> > can imagine some different new Store implementation that has some
> > > > >> > advantages but is not a design derived from the log structured
> > > merge tree
> > > > >> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > > >> > wouldn't apply to cells from such a thing because they are
> > > particular to
> > > > >> > how LSMs work. I'm sure such a project if attempted would make a
> > > number
> > > > >> of
> > > > >> > changes requiring a major version increment and low level details
> > > could
> > > > >> be
> > > > >> > unwound from Cell then, but if we could avoid doing it in the first
> > > > >> place,
> > > > >> > I think it would better for maintainability.
> > > > >> >
> > > > >> >
> > > > >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > > >> >
> > > > >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > hi folks,
> > > > >> > > >
> > > > >> > > > User is allowed to create custom cell but the valid code of
> > > type -
> > > > >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> > > should
> > > > >> expose
> > > > >> > > > KeyValue#Type as Public Client. Three possible ways are shown
> > > below:
> > > > >> > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > > IA.Public
> > > > >> > > > 2) Move KeyValue#Type into Cell.
> > > > >> > > > 3) Move KeyValue#Type to upper level
> > > > >> > > >
> > > > >> > > > Any suggestions?
> > > > >> > > >
> > > > >> > > >
> > > > >> > > What is the problem that we are trying to solve Chia-Ping? You
> > > want to
> > > > >> make
> > > > >> > > Cells of a new Type?
> > > > >> > >
> > > > >> > > My first reaction is that KV#Type is particular to the KV
> > > > >> implementation.
> > > > >> > > Any new Cell implementation should not have to adopt the KeyValue
> > > > >> typing
> > > > >> > > mechanism.
> > > > >> > >
> > > > >> > > S
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > > --
> > > > >> > > > Chia-Ping
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Best regards,
> > > > >> > Andrew
> > > > >> >
> > > > >> > Words like orphans lost among the crosstalk, meaning torn from
> > > truth's
> > > > >> > decrepit hands
> > > > >> > - A23, Crosstalk
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
> You could add to CellBuilder methods to do setPutType, setDeleteType, etc.,
> which would work for build time allowing you could keep Type byte private.
> This would seem to be enough for your case Chia-Ping?
Yep, i will file a issue to address it.
> We'd still be stuck at read time when all we had was Cell#getTypeByte
> returning a byte.
>
> We could add a CellType with the KV#Type static utility codeToType but
> would be sweet if we could do without having to explain the byte to users.
It seems to me that Cell is a general interface for kv format and others (not happen now), but we design the Cell's APIs according to the KV format. The cell's type should not be exposed to end user - I mean the Cell#getTypeByte should be removed - because it is hard to say other formats have the type field. The KV should have own interface which used in HRegion/HStore/HStoreFIle. The ExtendedCell is a good candidate because i don't think we need to mix together the kv format and other format in the same table/region/file. For normal user, all cells they get via Scan/Get are real Cell (not Delete mark or other type), so the type API is redundant. For advanced user, they can get the IA.LimitedPrivate Cell, such as KVInterface, via raw scan or other advanced operations.
Cell(IA.Public) <---- KVInterface(IA.LimitedPrivate) <--- KeyValue, Bufferedxxx (IA.Private)
<---- XXXInterface(IA.LimitedPrivate) <--- xxx cell (IA.Private)
On 2017-10-03 13:29, Stack <st...@duboce.net> wrote:
> On Mon, Oct 2, 2017 at 1:54 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > How about introducing an new enum "CellType" which is subset of
> > KeyValue#Type? It will be exposed as IA.Public to end user for helping
> > build the custom cell (via CellBuilder). The types which "CellType" should
> > have are shown below.
> > 1) Put
> > 2) Delete
> > 3) DeleteFamilyVersion
> > 4) DeleteColumn
> > 5) DeleteFamily
> > Hence, the CellBuilder#setType(byte) will be replaced by
> > CellBuilder#setType(CellType). Our internal use still reference to
> > KeyValue#Type.
> >
> >
> There is a 'hole' in our Cell Interface; we have a getTypeByte but expose
> no means of signaling what the byte stands for.
>
> You could add to CellBuilder methods to do setPutType, setDeleteType, etc.,
> which would work for build time allowing you could keep Type byte private.
> This would seem to be enough for your case Chia-Ping?
>
> We'd still be stuck at read time when all we had was Cell#getTypeByte
> returning a byte.
>
> We could add a CellType with the KV#Type static utility codeToType but
> would be sweet if we could do without having to explain the byte to users.
>
> S
>
>
>
>
>
>
>
> >
> > On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> > > Ya as Chia-Ping said, the problem he is trying to solve is very basic
> > > one. As long as we allow custom Cell creation (Via CellBuilder API)
> > > and allow Mutations to be added with Cells and pass that from client
> > > side APIs, we have to make the Type public accessible.
> > > Or else the Cell building APIs should not be taking in a type byte.
> > > We have to some way allow user to make put/delete cells stc.
> > >
> > > Is type that bound for only KV? We have getType in Cell also right?
> > > The type in full form what we have in KV now, may be making us confuse
> > > here? As Ram said it contains some internal types also which the user
> > > has never to know abt. Pls correct if saying in wrong way.
> > >
> > > Good that Chia-Ping brought this out here. We have to either way
> > > solve it and make the public API fully public.
> > >
> > > -Anoop-
> > >
> > > On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> > > <ra...@gmail.com> wrote:
> > > > Even if we are trying to move out I think only few of the types are
> > really
> > > > user readable. So we should be very careful here. So since we have
> > > > CellBuilder way it is better we check what type of cells a user can
> > build.
> > > > I think for now the Cellbuilder is not client exposed?
> > > > But again moving to Cell means it becomes public which is not right
> > IMO and
> > > > I thinks others here also agree to it.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > > > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> > > > wrote:
> > > >
> > > >> Thanks for all comment.
> > > >>
> > > >> The problem i want to resolve is the valid code should be exposed as
> > > >> IA.Public. Otherwise, end user have to access the IA.Private class to
> > build
> > > >> the custom cell.
> > > >>
> > > >> For example, I have a use case which plays a streaming role in our
> > > >> appliaction. It
> > > >> applies the CellBuilder(HBASE-18519) to build custom cells. These
> > cells
> > > >> have many same fields so they are put in shared-memory for avoiding GC
> > > >> pause. Everything is wonderful. However, we have to access the
> > IA.Private
> > > >> class - KeyValue#Type - to get the valid code of Put.
> > > >>
> > > >> I believe there are many use cases of custom cell, and consequently
> > it is
> > > >> worth adding a way to get the valid type via IA.Public class.
> > Otherwise, it
> > > >> may imply that the custom cell is based on a unstable way, because the
> > > >> related code can be changed at any time.
> > > >> --
> > > >> Chia-Ping
> > > >>
> > > >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > > >> > I agree with Stack. Was typing up a reply to Anoop but let me move
> > it
> > > >> down
> > > >> > here.
> > > >> >
> > > >> > The type code exposes some low level details of how our current
> > stores
> > > >> are
> > > >> > architected. But what if in the future you could swap out HStore
> > > >> implements
> > > >> > Store with PStore implements Store, where HStore is backed by
> > HFiles and
> > > >> > PStore is backed by Parquet? Just as a hypothetical example. I know
> > there
> > > >> > would be larger issues if this were actually attempted. Bear with
> > me. You
> > > >> > can imagine some different new Store implementation that has some
> > > >> > advantages but is not a design derived from the log structured
> > merge tree
> > > >> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > > >> > wouldn't apply to cells from such a thing because they are
> > particular to
> > > >> > how LSMs work. I'm sure such a project if attempted would make a
> > number
> > > >> of
> > > >> > changes requiring a major version increment and low level details
> > could
> > > >> be
> > > >> > unwound from Cell then, but if we could avoid doing it in the first
> > > >> place,
> > > >> > I think it would better for maintainability.
> > > >> >
> > > >> >
> > > >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > > >> >
> > > >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> > chia7712@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > hi folks,
> > > >> > > >
> > > >> > > > User is allowed to create custom cell but the valid code of
> > type -
> > > >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> > should
> > > >> expose
> > > >> > > > KeyValue#Type as Public Client. Three possible ways are shown
> > below:
> > > >> > > > 1) Change declaration of KeyValue#Type from IA.Private to
> > IA.Public
> > > >> > > > 2) Move KeyValue#Type into Cell.
> > > >> > > > 3) Move KeyValue#Type to upper level
> > > >> > > >
> > > >> > > > Any suggestions?
> > > >> > > >
> > > >> > > >
> > > >> > > What is the problem that we are trying to solve Chia-Ping? You
> > want to
> > > >> make
> > > >> > > Cells of a new Type?
> > > >> > >
> > > >> > > My first reaction is that KV#Type is particular to the KV
> > > >> implementation.
> > > >> > > Any new Cell implementation should not have to adopt the KeyValue
> > > >> typing
> > > >> > > mechanism.
> > > >> > >
> > > >> > > S
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > > --
> > > >> > > > Chia-Ping
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Best regards,
> > > >> > Andrew
> > > >> >
> > > >> > Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > >> > decrepit hands
> > > >> > - A23, Crosstalk
> > > >> >
> > > >>
> > >
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Stack <st...@duboce.net>.
On Mon, Oct 2, 2017 at 1:54 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
> How about introducing an new enum "CellType" which is subset of
> KeyValue#Type? It will be exposed as IA.Public to end user for helping
> build the custom cell (via CellBuilder). The types which "CellType" should
> have are shown below.
> 1) Put
> 2) Delete
> 3) DeleteFamilyVersion
> 4) DeleteColumn
> 5) DeleteFamily
> Hence, the CellBuilder#setType(byte) will be replaced by
> CellBuilder#setType(CellType). Our internal use still reference to
> KeyValue#Type.
>
>
There is a 'hole' in our Cell Interface; we have a getTypeByte but expose
no means of signaling what the byte stands for.
You could add to CellBuilder methods to do setPutType, setDeleteType, etc.,
which would work for build time allowing you could keep Type byte private.
This would seem to be enough for your case Chia-Ping?
We'd still be stuck at read time when all we had was Cell#getTypeByte
returning a byte.
We could add a CellType with the KV#Type static utility codeToType but
would be sweet if we could do without having to explain the byte to users.
S
>
> On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> > Ya as Chia-Ping said, the problem he is trying to solve is very basic
> > one. As long as we allow custom Cell creation (Via CellBuilder API)
> > and allow Mutations to be added with Cells and pass that from client
> > side APIs, we have to make the Type public accessible.
> > Or else the Cell building APIs should not be taking in a type byte.
> > We have to some way allow user to make put/delete cells stc.
> >
> > Is type that bound for only KV? We have getType in Cell also right?
> > The type in full form what we have in KV now, may be making us confuse
> > here? As Ram said it contains some internal types also which the user
> > has never to know abt. Pls correct if saying in wrong way.
> >
> > Good that Chia-Ping brought this out here. We have to either way
> > solve it and make the public API fully public.
> >
> > -Anoop-
> >
> > On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> > <ra...@gmail.com> wrote:
> > > Even if we are trying to move out I think only few of the types are
> really
> > > user readable. So we should be very careful here. So since we have
> > > CellBuilder way it is better we check what type of cells a user can
> build.
> > > I think for now the Cellbuilder is not client exposed?
> > > But again moving to Cell means it becomes public which is not right
> IMO and
> > > I thinks others here also agree to it.
> > >
> > > Regards
> > > Ram
> > >
> > > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > >
> > >> Thanks for all comment.
> > >>
> > >> The problem i want to resolve is the valid code should be exposed as
> > >> IA.Public. Otherwise, end user have to access the IA.Private class to
> build
> > >> the custom cell.
> > >>
> > >> For example, I have a use case which plays a streaming role in our
> > >> appliaction. It
> > >> applies the CellBuilder(HBASE-18519) to build custom cells. These
> cells
> > >> have many same fields so they are put in shared-memory for avoiding GC
> > >> pause. Everything is wonderful. However, we have to access the
> IA.Private
> > >> class - KeyValue#Type - to get the valid code of Put.
> > >>
> > >> I believe there are many use cases of custom cell, and consequently
> it is
> > >> worth adding a way to get the valid type via IA.Public class.
> Otherwise, it
> > >> may imply that the custom cell is based on a unstable way, because the
> > >> related code can be changed at any time.
> > >> --
> > >> Chia-Ping
> > >>
> > >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > >> > I agree with Stack. Was typing up a reply to Anoop but let me move
> it
> > >> down
> > >> > here.
> > >> >
> > >> > The type code exposes some low level details of how our current
> stores
> > >> are
> > >> > architected. But what if in the future you could swap out HStore
> > >> implements
> > >> > Store with PStore implements Store, where HStore is backed by
> HFiles and
> > >> > PStore is backed by Parquet? Just as a hypothetical example. I know
> there
> > >> > would be larger issues if this were actually attempted. Bear with
> me. You
> > >> > can imagine some different new Store implementation that has some
> > >> > advantages but is not a design derived from the log structured
> merge tree
> > >> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > >> > wouldn't apply to cells from such a thing because they are
> particular to
> > >> > how LSMs work. I'm sure such a project if attempted would make a
> number
> > >> of
> > >> > changes requiring a major version increment and low level details
> could
> > >> be
> > >> > unwound from Cell then, but if we could avoid doing it in the first
> > >> place,
> > >> > I think it would better for maintainability.
> > >> >
> > >> >
> > >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> > >> >
> > >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <
> chia7712@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > hi folks,
> > >> > > >
> > >> > > > User is allowed to create custom cell but the valid code of
> type -
> > >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we
> should
> > >> expose
> > >> > > > KeyValue#Type as Public Client. Three possible ways are shown
> below:
> > >> > > > 1) Change declaration of KeyValue#Type from IA.Private to
> IA.Public
> > >> > > > 2) Move KeyValue#Type into Cell.
> > >> > > > 3) Move KeyValue#Type to upper level
> > >> > > >
> > >> > > > Any suggestions?
> > >> > > >
> > >> > > >
> > >> > > What is the problem that we are trying to solve Chia-Ping? You
> want to
> > >> make
> > >> > > Cells of a new Type?
> > >> > >
> > >> > > My first reaction is that KV#Type is particular to the KV
> > >> implementation.
> > >> > > Any new Cell implementation should not have to adopt the KeyValue
> > >> typing
> > >> > > mechanism.
> > >> > >
> > >> > > S
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > > --
> > >> > > > Chia-Ping
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Best regards,
> > >> > Andrew
> > >> >
> > >> > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > >> > decrepit hands
> > >> > - A23, Crosstalk
> > >> >
> > >>
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
How about introducing an new enum "CellType" which is subset of KeyValue#Type? It will be exposed as IA.Public to end user for helping build the custom cell (via CellBuilder). The types which "CellType" should have are shown below.
1) Put
2) Delete
3) DeleteFamilyVersion
4) DeleteColumn
5) DeleteFamily
Hence, the CellBuilder#setType(byte) will be replaced by CellBuilder#setType(CellType). Our internal use still reference to KeyValue#Type.
On 2017-09-29 18:39, Anoop John <an...@gmail.com> wrote:
> Ya as Chia-Ping said, the problem he is trying to solve is very basic
> one. As long as we allow custom Cell creation (Via CellBuilder API)
> and allow Mutations to be added with Cells and pass that from client
> side APIs, we have to make the Type public accessible.
> Or else the Cell building APIs should not be taking in a type byte.
> We have to some way allow user to make put/delete cells stc.
>
> Is type that bound for only KV? We have getType in Cell also right?
> The type in full form what we have in KV now, may be making us confuse
> here? As Ram said it contains some internal types also which the user
> has never to know abt. Pls correct if saying in wrong way.
>
> Good that Chia-Ping brought this out here. We have to either way
> solve it and make the public API fully public.
>
> -Anoop-
>
> On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
> <ra...@gmail.com> wrote:
> > Even if we are trying to move out I think only few of the types are really
> > user readable. So we should be very careful here. So since we have
> > CellBuilder way it is better we check what type of cells a user can build.
> > I think for now the Cellbuilder is not client exposed?
> > But again moving to Cell means it becomes public which is not right IMO and
> > I thinks others here also agree to it.
> >
> > Regards
> > Ram
> >
> > On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> >
> >> Thanks for all comment.
> >>
> >> The problem i want to resolve is the valid code should be exposed as
> >> IA.Public. Otherwise, end user have to access the IA.Private class to build
> >> the custom cell.
> >>
> >> For example, I have a use case which plays a streaming role in our
> >> appliaction. It
> >> applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> >> have many same fields so they are put in shared-memory for avoiding GC
> >> pause. Everything is wonderful. However, we have to access the IA.Private
> >> class - KeyValue#Type - to get the valid code of Put.
> >>
> >> I believe there are many use cases of custom cell, and consequently it is
> >> worth adding a way to get the valid type via IA.Public class. Otherwise, it
> >> may imply that the custom cell is based on a unstable way, because the
> >> related code can be changed at any time.
> >> --
> >> Chia-Ping
> >>
> >> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> >> > I agree with Stack. Was typing up a reply to Anoop but let me move it
> >> down
> >> > here.
> >> >
> >> > The type code exposes some low level details of how our current stores
> >> are
> >> > architected. But what if in the future you could swap out HStore
> >> implements
> >> > Store with PStore implements Store, where HStore is backed by HFiles and
> >> > PStore is backed by Parquet? Just as a hypothetical example. I know there
> >> > would be larger issues if this were actually attempted. Bear with me. You
> >> > can imagine some different new Store implementation that has some
> >> > advantages but is not a design derived from the log structured merge tree
> >> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> >> > wouldn't apply to cells from such a thing because they are particular to
> >> > how LSMs work. I'm sure such a project if attempted would make a number
> >> of
> >> > changes requiring a major version increment and low level details could
> >> be
> >> > unwound from Cell then, but if we could avoid doing it in the first
> >> place,
> >> > I think it would better for maintainability.
> >> >
> >> >
> >> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
> >> > > wrote:
> >> > >
> >> > > > hi folks,
> >> > > >
> >> > > > User is allowed to create custom cell but the valid code of type -
> >> > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> >> expose
> >> > > > KeyValue#Type as Public Client. Three possible ways are shown below:
> >> > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> >> > > > 2) Move KeyValue#Type into Cell.
> >> > > > 3) Move KeyValue#Type to upper level
> >> > > >
> >> > > > Any suggestions?
> >> > > >
> >> > > >
> >> > > What is the problem that we are trying to solve Chia-Ping? You want to
> >> make
> >> > > Cells of a new Type?
> >> > >
> >> > > My first reaction is that KV#Type is particular to the KV
> >> implementation.
> >> > > Any new Cell implementation should not have to adopt the KeyValue
> >> typing
> >> > > mechanism.
> >> > >
> >> > > S
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > > --
> >> > > > Chia-Ping
> >> > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> > Andrew
> >> >
> >> > Words like orphans lost among the crosstalk, meaning torn from truth's
> >> > decrepit hands
> >> > - A23, Crosstalk
> >> >
> >>
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Anoop John <an...@gmail.com>.
Ya as Chia-Ping said, the problem he is trying to solve is very basic
one. As long as we allow custom Cell creation (Via CellBuilder API)
and allow Mutations to be added with Cells and pass that from client
side APIs, we have to make the Type public accessible.
Or else the Cell building APIs should not be taking in a type byte.
We have to some way allow user to make put/delete cells stc.
Is type that bound for only KV? We have getType in Cell also right?
The type in full form what we have in KV now, may be making us confuse
here? As Ram said it contains some internal types also which the user
has never to know abt. Pls correct if saying in wrong way.
Good that Chia-Ping brought this out here. We have to either way
solve it and make the public API fully public.
-Anoop-
On Fri, Sep 29, 2017 at 2:27 PM, ramkrishna vasudevan
<ra...@gmail.com> wrote:
> Even if we are trying to move out I think only few of the types are really
> user readable. So we should be very careful here. So since we have
> CellBuilder way it is better we check what type of cells a user can build.
> I think for now the Cellbuilder is not client exposed?
> But again moving to Cell means it becomes public which is not right IMO and
> I thinks others here also agree to it.
>
> Regards
> Ram
>
> On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
>> Thanks for all comment.
>>
>> The problem i want to resolve is the valid code should be exposed as
>> IA.Public. Otherwise, end user have to access the IA.Private class to build
>> the custom cell.
>>
>> For example, I have a use case which plays a streaming role in our
>> appliaction. It
>> applies the CellBuilder(HBASE-18519) to build custom cells. These cells
>> have many same fields so they are put in shared-memory for avoiding GC
>> pause. Everything is wonderful. However, we have to access the IA.Private
>> class - KeyValue#Type - to get the valid code of Put.
>>
>> I believe there are many use cases of custom cell, and consequently it is
>> worth adding a way to get the valid type via IA.Public class. Otherwise, it
>> may imply that the custom cell is based on a unstable way, because the
>> related code can be changed at any time.
>> --
>> Chia-Ping
>>
>> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
>> > I agree with Stack. Was typing up a reply to Anoop but let me move it
>> down
>> > here.
>> >
>> > The type code exposes some low level details of how our current stores
>> are
>> > architected. But what if in the future you could swap out HStore
>> implements
>> > Store with PStore implements Store, where HStore is backed by HFiles and
>> > PStore is backed by Parquet? Just as a hypothetical example. I know there
>> > would be larger issues if this were actually attempted. Bear with me. You
>> > can imagine some different new Store implementation that has some
>> > advantages but is not a design derived from the log structured merge tree
>> > if you like. Most values from a new Cell.Type based on KeyValue.Type
>> > wouldn't apply to cells from such a thing because they are particular to
>> > how LSMs work. I'm sure such a project if attempted would make a number
>> of
>> > changes requiring a major version increment and low level details could
>> be
>> > unwound from Cell then, but if we could avoid doing it in the first
>> place,
>> > I think it would better for maintainability.
>> >
>> >
>> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
>> >
>> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
>> > > wrote:
>> > >
>> > > > hi folks,
>> > > >
>> > > > User is allowed to create custom cell but the valid code of type -
>> > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
>> expose
>> > > > KeyValue#Type as Public Client. Three possible ways are shown below:
>> > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
>> > > > 2) Move KeyValue#Type into Cell.
>> > > > 3) Move KeyValue#Type to upper level
>> > > >
>> > > > Any suggestions?
>> > > >
>> > > >
>> > > What is the problem that we are trying to solve Chia-Ping? You want to
>> make
>> > > Cells of a new Type?
>> > >
>> > > My first reaction is that KV#Type is particular to the KV
>> implementation.
>> > > Any new Cell implementation should not have to adopt the KeyValue
>> typing
>> > > mechanism.
>> > >
>> > > S
>> > >
>> > >
>> > >
>> > >
>> > > > --
>> > > > Chia-Ping
>> > > >
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Andrew
>> >
>> > Words like orphans lost among the crosstalk, meaning torn from truth's
>> > decrepit hands
>> > - A23, Crosstalk
>> >
>>
Re: [DISCUSS] Move Type out of KeyValue
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Even if we are trying to move out I think only few of the types are really
user readable. So we should be very careful here. So since we have
CellBuilder way it is better we check what type of cells a user can build.
I think for now the Cellbuilder is not client exposed?
But again moving to Cell means it becomes public which is not right IMO and
I thinks others here also agree to it.
Regards
Ram
On Fri, Sep 29, 2017 at 10:50 AM, Chia-Ping Tsai <ch...@apache.org>
wrote:
> Thanks for all comment.
>
> The problem i want to resolve is the valid code should be exposed as
> IA.Public. Otherwise, end user have to access the IA.Private class to build
> the custom cell.
>
> For example, I have a use case which plays a streaming role in our
> appliaction. It
> applies the CellBuilder(HBASE-18519) to build custom cells. These cells
> have many same fields so they are put in shared-memory for avoiding GC
> pause. Everything is wonderful. However, we have to access the IA.Private
> class - KeyValue#Type - to get the valid code of Put.
>
> I believe there are many use cases of custom cell, and consequently it is
> worth adding a way to get the valid type via IA.Public class. Otherwise, it
> may imply that the custom cell is based on a unstable way, because the
> related code can be changed at any time.
> --
> Chia-Ping
>
> On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> > I agree with Stack. Was typing up a reply to Anoop but let me move it
> down
> > here.
> >
> > The type code exposes some low level details of how our current stores
> are
> > architected. But what if in the future you could swap out HStore
> implements
> > Store with PStore implements Store, where HStore is backed by HFiles and
> > PStore is backed by Parquet? Just as a hypothetical example. I know there
> > would be larger issues if this were actually attempted. Bear with me. You
> > can imagine some different new Store implementation that has some
> > advantages but is not a design derived from the log structured merge tree
> > if you like. Most values from a new Cell.Type based on KeyValue.Type
> > wouldn't apply to cells from such a thing because they are particular to
> > how LSMs work. I'm sure such a project if attempted would make a number
> of
> > changes requiring a major version increment and low level details could
> be
> > unwound from Cell then, but if we could avoid doing it in the first
> place,
> > I think it would better for maintainability.
> >
> >
> > On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > >
> > > > hi folks,
> > > >
> > > > User is allowed to create custom cell but the valid code of type -
> > > > KeyValue#Type - is declared as IA.Private. As i see it, we should
> expose
> > > > KeyValue#Type as Public Client. Three possible ways are shown below:
> > > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > > > 2) Move KeyValue#Type into Cell.
> > > > 3) Move KeyValue#Type to upper level
> > > >
> > > > Any suggestions?
> > > >
> > > >
> > > What is the problem that we are trying to solve Chia-Ping? You want to
> make
> > > Cells of a new Type?
> > >
> > > My first reaction is that KV#Type is particular to the KV
> implementation.
> > > Any new Cell implementation should not have to adopt the KeyValue
> typing
> > > mechanism.
> > >
> > > S
> > >
> > >
> > >
> > >
> > > > --
> > > > Chia-Ping
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> > - A23, Crosstalk
> >
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Chia-Ping Tsai <ch...@apache.org>.
Thanks for all comment.
The problem i want to resolve is the valid code should be exposed as IA.Public. Otherwise, end user have to access the IA.Private class to build the custom cell.
For example, I have a use case which plays a streaming role in our appliaction. It
applies the CellBuilder(HBASE-18519) to build custom cells. These cells have many same fields so they are put in shared-memory for avoiding GC pause. Everything is wonderful. However, we have to access the IA.Private class - KeyValue#Type - to get the valid code of Put.
I believe there are many use cases of custom cell, and consequently it is worth adding a way to get the valid type via IA.Public class. Otherwise, it may imply that the custom cell is based on a unstable way, because the related code can be changed at any time.
--
Chia-Ping
On 2017-09-29 00:49, Andrew Purtell <ap...@apache.org> wrote:
> I agree with Stack. Was typing up a reply to Anoop but let me move it down
> here.
>
> The type code exposes some low level details of how our current stores are
> architected. But what if in the future you could swap out HStore implements
> Store with PStore implements Store, where HStore is backed by HFiles and
> PStore is backed by Parquet? Just as a hypothetical example. I know there
> would be larger issues if this were actually attempted. Bear with me. You
> can imagine some different new Store implementation that has some
> advantages but is not a design derived from the log structured merge tree
> if you like. Most values from a new Cell.Type based on KeyValue.Type
> wouldn't apply to cells from such a thing because they are particular to
> how LSMs work. I'm sure such a project if attempted would make a number of
> changes requiring a major version increment and low level details could be
> unwound from Cell then, but if we could avoid doing it in the first place,
> I think it would better for maintainability.
>
>
> On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
>
> > On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> >
> > > hi folks,
> > >
> > > User is allowed to create custom cell but the valid code of type -
> > > KeyValue#Type - is declared as IA.Private. As i see it, we should expose
> > > KeyValue#Type as Public Client. Three possible ways are shown below:
> > > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > > 2) Move KeyValue#Type into Cell.
> > > 3) Move KeyValue#Type to upper level
> > >
> > > Any suggestions?
> > >
> > >
> > What is the problem that we are trying to solve Chia-Ping? You want to make
> > Cells of a new Type?
> >
> > My first reaction is that KV#Type is particular to the KV implementation.
> > Any new Cell implementation should not have to adopt the KeyValue typing
> > mechanism.
> >
> > S
> >
> >
> >
> >
> > > --
> > > Chia-Ping
> > >
> > >
> >
>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
> - A23, Crosstalk
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Andrew Purtell <ap...@apache.org>.
I agree with Stack. Was typing up a reply to Anoop but let me move it down
here.
The type code exposes some low level details of how our current stores are
architected. But what if in the future you could swap out HStore implements
Store with PStore implements Store, where HStore is backed by HFiles and
PStore is backed by Parquet? Just as a hypothetical example. I know there
would be larger issues if this were actually attempted. Bear with me. You
can imagine some different new Store implementation that has some
advantages but is not a design derived from the log structured merge tree
if you like. Most values from a new Cell.Type based on KeyValue.Type
wouldn't apply to cells from such a thing because they are particular to
how LSMs work. I'm sure such a project if attempted would make a number of
changes requiring a major version increment and low level details could be
unwound from Cell then, but if we could avoid doing it in the first place,
I think it would better for maintainability.
On Thu, Sep 28, 2017 at 9:39 AM, Stack <st...@duboce.net> wrote:
> On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
> > hi folks,
> >
> > User is allowed to create custom cell but the valid code of type -
> > KeyValue#Type - is declared as IA.Private. As i see it, we should expose
> > KeyValue#Type as Public Client. Three possible ways are shown below:
> > 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> > 2) Move KeyValue#Type into Cell.
> > 3) Move KeyValue#Type to upper level
> >
> > Any suggestions?
> >
> >
> What is the problem that we are trying to solve Chia-Ping? You want to make
> Cells of a new Type?
>
> My first reaction is that KV#Type is particular to the KV implementation.
> Any new Cell implementation should not have to adopt the KeyValue typing
> mechanism.
>
> S
>
>
>
>
> > --
> > Chia-Ping
> >
> >
>
--
Best regards,
Andrew
Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
- A23, Crosstalk
Re: [DISCUSS] Move Type out of KeyValue
Posted by Stack <st...@duboce.net>.
On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
> hi folks,
>
> User is allowed to create custom cell but the valid code of type -
> KeyValue#Type - is declared as IA.Private. As i see it, we should expose
> KeyValue#Type as Public Client. Three possible ways are shown below:
> 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> 2) Move KeyValue#Type into Cell.
> 3) Move KeyValue#Type to upper level
>
> Any suggestions?
>
>
What is the problem that we are trying to solve Chia-Ping? You want to make
Cells of a new Type?
My first reaction is that KV#Type is particular to the KV implementation.
Any new Cell implementation should not have to adopt the KeyValue typing
mechanism.
S
> --
> Chia-Ping
>
>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Anoop John <an...@gmail.com>.
Moving out of KV means it is BC break right? Even though KV is
Private, using that only client side can create cell objects (At least
Type specify) . Should we have a deprecation cycle?
Deprecate KV.Type and add Cell.Type (?)
-Anoop-
On Thu, Sep 28, 2017 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
> How about choosing option #2 ?
>
> Cell is annotated Public.
>
> On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
>
>> hi folks,
>>
>> User is allowed to create custom cell but the valid code of type -
>> KeyValue#Type - is declared as IA.Private. As i see it, we should expose
>> KeyValue#Type as Public Client. Three possible ways are shown below:
>> 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
>> 2) Move KeyValue#Type into Cell.
>> 3) Move KeyValue#Type to upper level
>>
>> Any suggestions?
>>
>> --
>> Chia-Ping
>>
>>
Re: [DISCUSS] Move Type out of KeyValue
Posted by Ted Yu <yu...@gmail.com>.
How about choosing option #2 ?
Cell is annotated Public.
On Thu, Sep 28, 2017 at 2:25 AM, Chia-Ping Tsai <ch...@apache.org> wrote:
> hi folks,
>
> User is allowed to create custom cell but the valid code of type -
> KeyValue#Type - is declared as IA.Private. As i see it, we should expose
> KeyValue#Type as Public Client. Three possible ways are shown below:
> 1) Change declaration of KeyValue#Type from IA.Private to IA.Public
> 2) Move KeyValue#Type into Cell.
> 3) Move KeyValue#Type to upper level
>
> Any suggestions?
>
> --
> Chia-Ping
>
>