You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Andrey Gura <ag...@gridgain.com> on 2016/01/14 17:10:50 UTC
Re: Json support in Ignite
Hi,
> > Also we need following functionality for json objects
> >
> > 1. Possibility to map json cache key to affinity key. This can be
> achieved
> > with custom affinity mapper:
> >
> > cache.setAffinityMapper(new AffinityKeyMapper() {
> > @Override public Object affinityKey(Object key) {
> > return ((JsonObject)key).getJsonNumber("orgId").longValue();
> > }
> > });
> >
>
> We can also have a special field within JSON Object, called
> ignite_affinity_field=bla
+1
It looks like we are over architecting here.
>
> Keys should not have many fields, and most likely will have one or two.
> Iterating through fields and generating hash-code or equals the standard
> way seems reasonable (as opposed to creating a whole architecture around
> how to get hash and equals fields).
>
> If it makes sense, we should cache the hash code value inside the key
> object after it has been generated once.
>
I don't think that we should have some constraints here. If user wants to
use specific set of fields for hashCode/equals he should have such
possibility.
There is one more problem in hashCode/equals: how to handle cases when
specified field isn't mandatory?
On Wed, Jul 15, 2015 at 11:32 AM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> On Wed, Jul 15, 2015 at 12:00 AM, Semyon Boikov <sb...@gridgain.com>
> wrote:
>
> > Hi all,
> >
> > As part of integration with Node.Js (IGNITE-961) we age going to add in
> > Ignite support for JSON (possibility for IgniteCache to put, get, index
> and
> > query JSON data).
> >
> > We can implement standard 'javax.json' API (JSR 353), with this API work
> > with cache will look like in this example:
> >
> >
> Are we going to implement JSON specification ourselves? Sounds a bit fishy.
> Can we just take some available implementation (assuming that the license
> fits)?
>
>
>
> > Ignite should somehow provide access to 'javax.json' API, unfortunately
> > 'javax.json' is not part of JDK so we have several options how it can be
> > added in Ignite:
> >
> > 1. Add dependency for 'javax.json-api' in core module, in this case we
> can
> > have method to get access for json API on 'org.apache.ignite.Ignite':
> >
> > * interface Ignite {*
> > * ...*
> >
> > * javax.json.JsonProvider json();*
> > * }*
> >
> > * JsonProvider json = ignite.json();*
> >
> > Not sure it is ok, since now we don't have 3rd party dependencies in
> ignite
> > 'core' module except 'javax.cache' API.
> >
> > 2. Added another optional module 'json', this module will have dependency
> > for 'javax.json' API, and in this module we can have factory providing
> > access to 'javax.json' API:
> >
>
> I think an optional module makes sense.
>
>
>
> >
> > 3. Create plugin for json, in this case plugin will provide access to
> > 'javax.json' API:
> >
> > * IgniteJsonPlugin jsonPlugin = ignite.plugin(IgniteJsonPlugin.NAME);*
> >
> > * javax.json.JsonProvider provider = jsonPlugin.json();*
> >
> >
> > Not sure this aproach with plugin is ok, since now we don't have 'core'
> > ignite functionality implemented as plugins.
> >
>
> Agree, this probably won't work.
>
>
>
> > Also we need following functionality for json objects
> >
> > 1. Possibility to map json cache key to affinity key. This can be
> achieved
> > with custom affinity mapper:
> >
> > cache.setAffinityMapper(new AffinityKeyMapper() {
> > @Override public Object affinityKey(Object key) {
> > return ((JsonObject)key).getJsonNumber("orgId").longValue();
> > }
> > });
> >
>
> We can also have a special field within JSON Object, called
> ignite_affinity_field=bla
>
>
> >
> > 2. Efficient implementation for equals/hashCode for json cache keys.
> >
> > One option is just have equals/hashCode like in HashMap.
> >
> > Another option can be add special JsonConfiguration, it should be set in
> > CacheConfiguration and can contain following settings:
> > - list of fields to use in 'equals' and 'hashCode' calculation
> > - alternatively for equals/hashCode we can provide possibility to
> implement
> > custom comparator/hash code calculator
> > - JsonConfiguration can have setting for affinity key
> >
> > So JsonConfiguration can look like this:
> >
> > *class JsonConfiguration {*
> > * /***
> > * * Field names for hash code calculation.*
> > * */*
> > * public Collection<String> getKeyHashCodeFields();*
> >
> > * /***
> > * * Field names to use .*
> > * */*
> > * public Collection<String> getKeyEqualsFields();*
> >
> > * /***
> > * * Field name to get key which will be used for node affinity
> > calculation.*
> > * */*
> > * public String getAffinityKeyField();*
> > *}*
> >
>
> It looks like we are over architecting here.
>
> Keys should not have many fields, and most likely will have one or two.
> Iterating through fields and generating hash-code or equals the standard
> way seems reasonable (as opposed to creating a whole architecture around
> how to get hash and equals fields).
>
> If it makes sense, we should cache the hash code value inside the key
> object after it has been generated once.
>
>
> >
> >
> > I need your feedback on these questions:
> > - is it ok to implement 'javax.json' API to support json for Ignite
> caches?
> >
>
> Can we get some existing implementation instead?
>
>
> > - if we implement 'javax.json' what is best way to add it in Ignite (see
> my
> > suggestion above: just add dependency for 'javax.json' API in 'core'
> > module, add special module or plugin)
> >
>
> Optional module?
>
>
> > - do we need to adde JsonConfiguration I described above?
> >
>
> Most likely not
>
>
> >
> > Thanks!
> >
>
--
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com
Re: Json support in Ignite
Posted by Konstantin Boudnik <co...@apache.org>.
One other thing to keep in mind, that JSON data is often not-flattened ie
coming with hierarchical structures. In which case, querying it from Ignite
would be a non-trivial if at all possible.
Cos
On Thu, Jan 14, 2016 at 09:09AM, Dmitriy Setrakyan wrote:
> I would consider a case for generating hash-code by iterating through all
> the fields by default. We need to make sure that the iteration order is the
> same on both ends.
>
> User can always override it, no?
>
> D.
>
> On Thu, Jan 14, 2016 at 8:10 AM, Andrey Gura <ag...@gridgain.com> wrote:
>
> > Hi,
> >
> >
> > > > Also we need following functionality for json objects
> > > >
> > > > 1. Possibility to map json cache key to affinity key. This can be
> > > achieved
> > > > with custom affinity mapper:
> > > >
> > > > cache.setAffinityMapper(new AffinityKeyMapper() {
> > > > @Override public Object affinityKey(Object key) {
> > > > return
> > ((JsonObject)key).getJsonNumber("orgId").longValue();
> > > > }
> > > > });
> > > >
> > >
> > > We can also have a special field within JSON Object, called
> > > ignite_affinity_field=bla
> >
> >
> > +1
> >
> > It looks like we are over architecting here.
> > >
> > > Keys should not have many fields, and most likely will have one or two.
> > > Iterating through fields and generating hash-code or equals the standard
> > > way seems reasonable (as opposed to creating a whole architecture around
> > > how to get hash and equals fields).
> > >
> > > If it makes sense, we should cache the hash code value inside the key
> > > object after it has been generated once.
> > >
> >
> > I don't think that we should have some constraints here. If user wants to
> > use specific set of fields for hashCode/equals he should have such
> > possibility.
> >
> > There is one more problem in hashCode/equals: how to handle cases when
> > specified field isn't mandatory?
> >
> >
> > On Wed, Jul 15, 2015 at 11:32 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> > >
> > wrote:
> >
> > > On Wed, Jul 15, 2015 at 12:00 AM, Semyon Boikov <sb...@gridgain.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > As part of integration with Node.Js (IGNITE-961) we age going to add in
> > > > Ignite support for JSON (possibility for IgniteCache to put, get, index
> > > and
> > > > query JSON data).
> > > >
> > > > We can implement standard 'javax.json' API (JSR 353), with this API
> > work
> > > > with cache will look like in this example:
> > > >
> > > >
> > > Are we going to implement JSON specification ourselves? Sounds a bit
> > fishy.
> > > Can we just take some available implementation (assuming that the license
> > > fits)?
> > >
> > >
> > >
> > > > Ignite should somehow provide access to 'javax.json' API, unfortunately
> > > > 'javax.json' is not part of JDK so we have several options how it can
> > be
> > > > added in Ignite:
> > > >
> > > > 1. Add dependency for 'javax.json-api' in core module, in this case we
> > > can
> > > > have method to get access for json API on 'org.apache.ignite.Ignite':
> > > >
> > > > * interface Ignite {*
> > > > * ...*
> > > >
> > > > * javax.json.JsonProvider json();*
> > > > * }*
> > > >
> > > > * JsonProvider json = ignite.json();*
> > > >
> > > > Not sure it is ok, since now we don't have 3rd party dependencies in
> > > ignite
> > > > 'core' module except 'javax.cache' API.
> > > >
> > > > 2. Added another optional module 'json', this module will have
> > dependency
> > > > for 'javax.json' API, and in this module we can have factory providing
> > > > access to 'javax.json' API:
> > > >
> > >
> > > I think an optional module makes sense.
> > >
> > >
> > >
> > > >
> > > > 3. Create plugin for json, in this case plugin will provide access to
> > > > 'javax.json' API:
> > > >
> > > > * IgniteJsonPlugin jsonPlugin = ignite.plugin(IgniteJsonPlugin.NAME);*
> > > >
> > > > * javax.json.JsonProvider provider = jsonPlugin.json();*
> > > >
> > > >
> > > > Not sure this aproach with plugin is ok, since now we don't have 'core'
> > > > ignite functionality implemented as plugins.
> > > >
> > >
> > > Agree, this probably won't work.
> > >
> > >
> > >
> > > > Also we need following functionality for json objects
> > > >
> > > > 1. Possibility to map json cache key to affinity key. This can be
> > > achieved
> > > > with custom affinity mapper:
> > > >
> > > > cache.setAffinityMapper(new AffinityKeyMapper() {
> > > > @Override public Object affinityKey(Object key) {
> > > > return
> > ((JsonObject)key).getJsonNumber("orgId").longValue();
> > > > }
> > > > });
> > > >
> > >
> > > We can also have a special field within JSON Object, called
> > > ignite_affinity_field=bla
> > >
> > >
> > > >
> > > > 2. Efficient implementation for equals/hashCode for json cache keys.
> > > >
> > > > One option is just have equals/hashCode like in HashMap.
> > > >
> > > > Another option can be add special JsonConfiguration, it should be set
> > in
> > > > CacheConfiguration and can contain following settings:
> > > > - list of fields to use in 'equals' and 'hashCode' calculation
> > > > - alternatively for equals/hashCode we can provide possibility to
> > > implement
> > > > custom comparator/hash code calculator
> > > > - JsonConfiguration can have setting for affinity key
> > > >
> > > > So JsonConfiguration can look like this:
> > > >
> > > > *class JsonConfiguration {*
> > > > * /***
> > > > * * Field names for hash code calculation.*
> > > > * */*
> > > > * public Collection<String> getKeyHashCodeFields();*
> > > >
> > > > * /***
> > > > * * Field names to use .*
> > > > * */*
> > > > * public Collection<String> getKeyEqualsFields();*
> > > >
> > > > * /***
> > > > * * Field name to get key which will be used for node affinity
> > > > calculation.*
> > > > * */*
> > > > * public String getAffinityKeyField();*
> > > > *}*
> > > >
> > >
> > > It looks like we are over architecting here.
> > >
> > > Keys should not have many fields, and most likely will have one or two.
> > > Iterating through fields and generating hash-code or equals the standard
> > > way seems reasonable (as opposed to creating a whole architecture around
> > > how to get hash and equals fields).
> > >
> > > If it makes sense, we should cache the hash code value inside the key
> > > object after it has been generated once.
> > >
> > >
> > > >
> > > >
> > > > I need your feedback on these questions:
> > > > - is it ok to implement 'javax.json' API to support json for Ignite
> > > caches?
> > > >
> > >
> > > Can we get some existing implementation instead?
> > >
> > >
> > > > - if we implement 'javax.json' what is best way to add it in Ignite
> > (see
> > > my
> > > > suggestion above: just add dependency for 'javax.json' API in 'core'
> > > > module, add special module or plugin)
> > > >
> > >
> > > Optional module?
> > >
> > >
> > > > - do we need to adde JsonConfiguration I described above?
> > > >
> > >
> > > Most likely not
> > >
> > >
> > > >
> > > > Thanks!
> > > >
> > >
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
> >
Re: Json support in Ignite
Posted by Dmitriy Setrakyan <ds...@apache.org>.
I would consider a case for generating hash-code by iterating through all
the fields by default. We need to make sure that the iteration order is the
same on both ends.
User can always override it, no?
D.
On Thu, Jan 14, 2016 at 8:10 AM, Andrey Gura <ag...@gridgain.com> wrote:
> Hi,
>
>
> > > Also we need following functionality for json objects
> > >
> > > 1. Possibility to map json cache key to affinity key. This can be
> > achieved
> > > with custom affinity mapper:
> > >
> > > cache.setAffinityMapper(new AffinityKeyMapper() {
> > > @Override public Object affinityKey(Object key) {
> > > return
> ((JsonObject)key).getJsonNumber("orgId").longValue();
> > > }
> > > });
> > >
> >
> > We can also have a special field within JSON Object, called
> > ignite_affinity_field=bla
>
>
> +1
>
> It looks like we are over architecting here.
> >
> > Keys should not have many fields, and most likely will have one or two.
> > Iterating through fields and generating hash-code or equals the standard
> > way seems reasonable (as opposed to creating a whole architecture around
> > how to get hash and equals fields).
> >
> > If it makes sense, we should cache the hash code value inside the key
> > object after it has been generated once.
> >
>
> I don't think that we should have some constraints here. If user wants to
> use specific set of fields for hashCode/equals he should have such
> possibility.
>
> There is one more problem in hashCode/equals: how to handle cases when
> specified field isn't mandatory?
>
>
> On Wed, Jul 15, 2015 at 11:32 AM, Dmitriy Setrakyan <dsetrakyan@apache.org
> >
> wrote:
>
> > On Wed, Jul 15, 2015 at 12:00 AM, Semyon Boikov <sb...@gridgain.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > As part of integration with Node.Js (IGNITE-961) we age going to add in
> > > Ignite support for JSON (possibility for IgniteCache to put, get, index
> > and
> > > query JSON data).
> > >
> > > We can implement standard 'javax.json' API (JSR 353), with this API
> work
> > > with cache will look like in this example:
> > >
> > >
> > Are we going to implement JSON specification ourselves? Sounds a bit
> fishy.
> > Can we just take some available implementation (assuming that the license
> > fits)?
> >
> >
> >
> > > Ignite should somehow provide access to 'javax.json' API, unfortunately
> > > 'javax.json' is not part of JDK so we have several options how it can
> be
> > > added in Ignite:
> > >
> > > 1. Add dependency for 'javax.json-api' in core module, in this case we
> > can
> > > have method to get access for json API on 'org.apache.ignite.Ignite':
> > >
> > > * interface Ignite {*
> > > * ...*
> > >
> > > * javax.json.JsonProvider json();*
> > > * }*
> > >
> > > * JsonProvider json = ignite.json();*
> > >
> > > Not sure it is ok, since now we don't have 3rd party dependencies in
> > ignite
> > > 'core' module except 'javax.cache' API.
> > >
> > > 2. Added another optional module 'json', this module will have
> dependency
> > > for 'javax.json' API, and in this module we can have factory providing
> > > access to 'javax.json' API:
> > >
> >
> > I think an optional module makes sense.
> >
> >
> >
> > >
> > > 3. Create plugin for json, in this case plugin will provide access to
> > > 'javax.json' API:
> > >
> > > * IgniteJsonPlugin jsonPlugin = ignite.plugin(IgniteJsonPlugin.NAME);*
> > >
> > > * javax.json.JsonProvider provider = jsonPlugin.json();*
> > >
> > >
> > > Not sure this aproach with plugin is ok, since now we don't have 'core'
> > > ignite functionality implemented as plugins.
> > >
> >
> > Agree, this probably won't work.
> >
> >
> >
> > > Also we need following functionality for json objects
> > >
> > > 1. Possibility to map json cache key to affinity key. This can be
> > achieved
> > > with custom affinity mapper:
> > >
> > > cache.setAffinityMapper(new AffinityKeyMapper() {
> > > @Override public Object affinityKey(Object key) {
> > > return
> ((JsonObject)key).getJsonNumber("orgId").longValue();
> > > }
> > > });
> > >
> >
> > We can also have a special field within JSON Object, called
> > ignite_affinity_field=bla
> >
> >
> > >
> > > 2. Efficient implementation for equals/hashCode for json cache keys.
> > >
> > > One option is just have equals/hashCode like in HashMap.
> > >
> > > Another option can be add special JsonConfiguration, it should be set
> in
> > > CacheConfiguration and can contain following settings:
> > > - list of fields to use in 'equals' and 'hashCode' calculation
> > > - alternatively for equals/hashCode we can provide possibility to
> > implement
> > > custom comparator/hash code calculator
> > > - JsonConfiguration can have setting for affinity key
> > >
> > > So JsonConfiguration can look like this:
> > >
> > > *class JsonConfiguration {*
> > > * /***
> > > * * Field names for hash code calculation.*
> > > * */*
> > > * public Collection<String> getKeyHashCodeFields();*
> > >
> > > * /***
> > > * * Field names to use .*
> > > * */*
> > > * public Collection<String> getKeyEqualsFields();*
> > >
> > > * /***
> > > * * Field name to get key which will be used for node affinity
> > > calculation.*
> > > * */*
> > > * public String getAffinityKeyField();*
> > > *}*
> > >
> >
> > It looks like we are over architecting here.
> >
> > Keys should not have many fields, and most likely will have one or two.
> > Iterating through fields and generating hash-code or equals the standard
> > way seems reasonable (as opposed to creating a whole architecture around
> > how to get hash and equals fields).
> >
> > If it makes sense, we should cache the hash code value inside the key
> > object after it has been generated once.
> >
> >
> > >
> > >
> > > I need your feedback on these questions:
> > > - is it ok to implement 'javax.json' API to support json for Ignite
> > caches?
> > >
> >
> > Can we get some existing implementation instead?
> >
> >
> > > - if we implement 'javax.json' what is best way to add it in Ignite
> (see
> > my
> > > suggestion above: just add dependency for 'javax.json' API in 'core'
> > > module, add special module or plugin)
> > >
> >
> > Optional module?
> >
> >
> > > - do we need to adde JsonConfiguration I described above?
> > >
> >
> > Most likely not
> >
> >
> > >
> > > Thanks!
> > >
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>