API rework discussion summary: 03-03-2014

[Sandeep Nayak <os...@gmail.com>]

Hi guys,

There have not been any code changes since the last update on
03-01-2014 however there was some discussion on the design of the
Helix API today which is summarized below

Participants: Kishore, Kanak and myself.

Kishore and Kanak please correct me if I misrepresented any of the discussion.

(1)  HelixProperty will be renamed to HelixRecord which will
internally have HelixKey and HelixValue.

(2) HelixKey is what is currently PropertyKey in Helix. A key is not
versioned. A key is a path to the value stored. There was much
discussion on trying to surface the segmented nature of the key in the
API by expressing it as a URN but was tabled to be revisited later.

(3) HelixValue is a value the key points to and is currently ZNRecord.
A value is versioned and each key points to the latest value. The
version helps build a CAS (CompareAndSet) semantic for update calls.

(4) Helix Record will be placed in the model package along with
HelixKey and HelixValue

(5) Model package will carry classes with accessors only, all
mutations should go through commands for the same. There is concern
that we may end up proliferating commands so we need to be cognizant
of this when we build the commands.

(6) Command builders will be put in place to allow creating the commands

(7) There was much discussion on the nomenclature of Instance v/s
(Participant, Observer or Admin) in Helix. The conclusion was to avoid
using Instance in the API as it is confusing and was legacy. Instead
we will use the specific terms of Participant, Observer, Admin in
command objects. For the sake of completion, a Node is seen as a
physical instance which is why the abstraction of a Node was avoided.

(8) Another example of command is ResourceCommand, it will have to
carry the resource configuration and will need some way to express the
Rebalancer config. So a command like
ResourceCommand.using(ResourceConfig).with(RebalancerConfig) maybe.
More details to be discussed as we move closer to the command design.

(9) The HelixAdmin is thus shaping to have an API like

newValue = HelixAdmin.do(Command) or
newValue = HelixAdmin.execute(Command).

The call internally reads existing data from the persistent store,
identifies the deltas to be applied based on the command issued and
applies the deltas to the store through the data accessor. The return
value is the new model value.

(10) This leads to ZNRecordDelta not being necessary in the API

Based on the above next things I plan to tackle in order are

(a) the HelixProperty realignment as outlined in (1) above
(b) reworking some objects from config into model (see email exchange
for API rework summary 03-01-2014 thread)
(c) move more models into API, cleanup as necessary
(d) start on command objects

I will send a summary as and when I tackle these and check-in code.


Thanks,

Sandeep

Re: API rework discussion summary: 03-03-2014

[Zhen Zhang <ne...@gmail.com>]

Looks good to me.

Thanks,
Jason


On Mon, Mar 3, 2014 at 10:15 AM, kishore g <g....@gmail.com> wrote:

> Sounds good to me.
>
> Couple of things.
>
> HelixRecord, HelixKey, HelixValue looks good. Not sure if we need Helix
> prefix every where.
>
> will user access Record, key, value directly. This is probably needed only
> for the spi implementation. similarly dataaccessor will only be of interest
> to spi.
>
> Users will only use model classes and helixadmin for CRUD. Should we
> differentiate what is exposed to user v/s spi implementation.
>
> ResourceConfig is too complex, do we need to break it down?
>
>
> thanks,
> Kishore G
>
>
>
>
> On Mon, Mar 3, 2014 at 1:21 AM, Sandeep Nayak <os...@gmail.com> wrote:
>
> > Hi guys,
> >
> > There have not been any code changes since the last update on
> > 03-01-2014 however there was some discussion on the design of the
> > Helix API today which is summarized below
> >
> > Participants: Kishore, Kanak and myself.
> >
> > Kishore and Kanak please correct me if I misrepresented any of the
> > discussion.
> >
> > (1)  HelixProperty will be renamed to HelixRecord which will
> > internally have HelixKey and HelixValue.
> >
> > (2) HelixKey is what is currently PropertyKey in Helix. A key is not
> > versioned. A key is a path to the value stored. There was much
> > discussion on trying to surface the segmented nature of the key in the
> > API by expressing it as a URN but was tabled to be revisited later.
> >
> > (3) HelixValue is a value the key points to and is currently ZNRecord.
> > A value is versioned and each key points to the latest value. The
> > version helps build a CAS (CompareAndSet) semantic for update calls.
> >
> > (4) Helix Record will be placed in the model package along with
> > HelixKey and HelixValue
> >
> > (5) Model package will carry classes with accessors only, all
> > mutations should go through commands for the same. There is concern
> > that we may end up proliferating commands so we need to be cognizant
> > of this when we build the commands.
> >
> > (6) Command builders will be put in place to allow creating the commands
> >
> > (7) There was much discussion on the nomenclature of Instance v/s
> > (Participant, Observer or Admin) in Helix. The conclusion was to avoid
> > using Instance in the API as it is confusing and was legacy. Instead
> > we will use the specific terms of Participant, Observer, Admin in
> > command objects. For the sake of completion, a Node is seen as a
> > physical instance which is why the abstraction of a Node was avoided.
> >
> > (8) Another example of command is ResourceCommand, it will have to
> > carry the resource configuration and will need some way to express the
> > Rebalancer config. So a command like
> > ResourceCommand.using(ResourceConfig).with(RebalancerConfig) maybe.
> > More details to be discussed as we move closer to the command design.
> >
> > (9) The HelixAdmin is thus shaping to have an API like
> >
> > newValue = HelixAdmin.do(Command) or
> > newValue = HelixAdmin.execute(Command).
> >
> > The call internally reads existing data from the persistent store,
> > identifies the deltas to be applied based on the command issued and
> > applies the deltas to the store through the data accessor. The return
> > value is the new model value.
> >
> > (10) This leads to ZNRecordDelta not being necessary in the API
> >
> > Based on the above next things I plan to tackle in order are
> >
> > (a) the HelixProperty realignment as outlined in (1) above
> > (b) reworking some objects from config into model (see email exchange
> > for API rework summary 03-01-2014 thread)
> > (c) move more models into API, cleanup as necessary
> > (d) start on command objects
> >
> > I will send a summary as and when I tackle these and check-in code.
> >
> >
> > Thanks,
> >
> > Sandeep
> >
>

Re: API rework discussion summary: 03-03-2014

[Sandeep Nayak <os...@gmail.com>]

See responses inline.

On Mon, Mar 3, 2014 at 10:15 AM, kishore g <g....@gmail.com> wrote:
> Sounds good to me.
>
> Couple of things.
>
> HelixRecord, HelixKey, HelixValue looks good. Not sure if we need Helix
> prefix every where.

[Sandeep] I don't mind either way, we can drop the prefix if the name
sounds too verbose.
I will retain it for now, we can revisit before we finalize API, will
add a note in the relevant
classes.

>
> will user access Record, key, value directly.

[Sandeep] This is where your input will drive what we expose, if we
don't need to give them out then yes
only SPI should have access to getters.

>This is probably needed only
> for the spi implementation. similarly dataaccessor will only be of interest
> to spi.

[Sandeep] Yes, dataaccessor will be SPI only, was only referencing it
to layout the call-graph.

>
> Users will only use model classes and helixadmin for CRUD. Should we
> differentiate what is exposed to user v/s spi implementation.

[Sandeep] Lets finish with API or atleast get to some level of
completeness before we start
tackling SPI, it might be too much too soon if we try to do both.
Maybe create an internal annotation
which can be placed on every class which we feel should be SPI and
mark it with the annotation so
we dont lose track of it. We can then cleanup the annotation before we are done.

>
> ResourceConfig is too complex, do we need to break it down?

[Sandeep] Lets discuss when we get to builders which reference the
resource-config, you probably
are right that we may need to break it down. Lets discuss when we get
to that point?

>
>
> thanks,
> Kishore G
>
>
>
>
> On Mon, Mar 3, 2014 at 1:21 AM, Sandeep Nayak <os...@gmail.com> wrote:
>
>> Hi guys,
>>
>> There have not been any code changes since the last update on
>> 03-01-2014 however there was some discussion on the design of the
>> Helix API today which is summarized below
>>
>> Participants: Kishore, Kanak and myself.
>>
>> Kishore and Kanak please correct me if I misrepresented any of the
>> discussion.
>>
>> (1)  HelixProperty will be renamed to HelixRecord which will
>> internally have HelixKey and HelixValue.
>>
>> (2) HelixKey is what is currently PropertyKey in Helix. A key is not
>> versioned. A key is a path to the value stored. There was much
>> discussion on trying to surface the segmented nature of the key in the
>> API by expressing it as a URN but was tabled to be revisited later.
>>
>> (3) HelixValue is a value the key points to and is currently ZNRecord.
>> A value is versioned and each key points to the latest value. The
>> version helps build a CAS (CompareAndSet) semantic for update calls.
>>
>> (4) Helix Record will be placed in the model package along with
>> HelixKey and HelixValue
>>
>> (5) Model package will carry classes with accessors only, all
>> mutations should go through commands for the same. There is concern
>> that we may end up proliferating commands so we need to be cognizant
>> of this when we build the commands.
>>
>> (6) Command builders will be put in place to allow creating the commands
>>
>> (7) There was much discussion on the nomenclature of Instance v/s
>> (Participant, Observer or Admin) in Helix. The conclusion was to avoid
>> using Instance in the API as it is confusing and was legacy. Instead
>> we will use the specific terms of Participant, Observer, Admin in
>> command objects. For the sake of completion, a Node is seen as a
>> physical instance which is why the abstraction of a Node was avoided.
>>
>> (8) Another example of command is ResourceCommand, it will have to
>> carry the resource configuration and will need some way to express the
>> Rebalancer config. So a command like
>> ResourceCommand.using(ResourceConfig).with(RebalancerConfig) maybe.
>> More details to be discussed as we move closer to the command design.
>>
>> (9) The HelixAdmin is thus shaping to have an API like
>>
>> newValue = HelixAdmin.do(Command) or
>> newValue = HelixAdmin.execute(Command).
>>
>> The call internally reads existing data from the persistent store,
>> identifies the deltas to be applied based on the command issued and
>> applies the deltas to the store through the data accessor. The return
>> value is the new model value.
>>
>> (10) This leads to ZNRecordDelta not being necessary in the API
>>
>> Based on the above next things I plan to tackle in order are
>>
>> (a) the HelixProperty realignment as outlined in (1) above
>> (b) reworking some objects from config into model (see email exchange
>> for API rework summary 03-01-2014 thread)
>> (c) move more models into API, cleanup as necessary
>> (d) start on command objects
>>
>> I will send a summary as and when I tackle these and check-in code.
>>
>>
>> Thanks,
>>
>> Sandeep
>>

Re: API rework discussion summary: 03-03-2014

[kishore g <g....@gmail.com>]

Sounds good to me.

Couple of things.

HelixRecord, HelixKey, HelixValue looks good. Not sure if we need Helix
prefix every where.

will user access Record, key, value directly. This is probably needed only
for the spi implementation. similarly dataaccessor will only be of interest
to spi.

Users will only use model classes and helixadmin for CRUD. Should we
differentiate what is exposed to user v/s spi implementation.

ResourceConfig is too complex, do we need to break it down?


thanks,
Kishore G




On Mon, Mar 3, 2014 at 1:21 AM, Sandeep Nayak <os...@gmail.com> wrote:

> Hi guys,
>
> There have not been any code changes since the last update on
> 03-01-2014 however there was some discussion on the design of the
> Helix API today which is summarized below
>
> Participants: Kishore, Kanak and myself.
>
> Kishore and Kanak please correct me if I misrepresented any of the
> discussion.
>
> (1)  HelixProperty will be renamed to HelixRecord which will
> internally have HelixKey and HelixValue.
>
> (2) HelixKey is what is currently PropertyKey in Helix. A key is not
> versioned. A key is a path to the value stored. There was much
> discussion on trying to surface the segmented nature of the key in the
> API by expressing it as a URN but was tabled to be revisited later.
>
> (3) HelixValue is a value the key points to and is currently ZNRecord.
> A value is versioned and each key points to the latest value. The
> version helps build a CAS (CompareAndSet) semantic for update calls.
>
> (4) Helix Record will be placed in the model package along with
> HelixKey and HelixValue
>
> (5) Model package will carry classes with accessors only, all
> mutations should go through commands for the same. There is concern
> that we may end up proliferating commands so we need to be cognizant
> of this when we build the commands.
>
> (6) Command builders will be put in place to allow creating the commands
>
> (7) There was much discussion on the nomenclature of Instance v/s
> (Participant, Observer or Admin) in Helix. The conclusion was to avoid
> using Instance in the API as it is confusing and was legacy. Instead
> we will use the specific terms of Participant, Observer, Admin in
> command objects. For the sake of completion, a Node is seen as a
> physical instance which is why the abstraction of a Node was avoided.
>
> (8) Another example of command is ResourceCommand, it will have to
> carry the resource configuration and will need some way to express the
> Rebalancer config. So a command like
> ResourceCommand.using(ResourceConfig).with(RebalancerConfig) maybe.
> More details to be discussed as we move closer to the command design.
>
> (9) The HelixAdmin is thus shaping to have an API like
>
> newValue = HelixAdmin.do(Command) or
> newValue = HelixAdmin.execute(Command).
>
> The call internally reads existing data from the persistent store,
> identifies the deltas to be applied based on the command issued and
> applies the deltas to the store through the data accessor. The return
> value is the new model value.
>
> (10) This leads to ZNRecordDelta not being necessary in the API
>
> Based on the above next things I plan to tackle in order are
>
> (a) the HelixProperty realignment as outlined in (1) above
> (b) reworking some objects from config into model (see email exchange
> for API rework summary 03-01-2014 thread)
> (c) move more models into API, cleanup as necessary
> (d) start on command objects
>
> I will send a summary as and when I tackle these and check-in code.
>
>
> Thanks,
>
> Sandeep
>