You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Dmitry Frolov <df...@gmail.com> on 2020/05/20 11:55:42 UTC
[DISCUSSION] Key and Value fields with same name and SQL DML
Igniters,
Currently Ignite Key/Value API allows to have both Key and Value fields with
the same name. Use of SQL DML to update such an entry will update only the
key field, leaving the value field unchanged. Seems that it's a bit
inconvenient for the mixed K/V and SQL API apps.
This is a proposal to add enhancements to Cache and SQL API to address this
inconvenience.
Proposal to API change:
*Cache API*: add new method /setKeyValueFields(keyValueFields: Set<String>):
QueryEntity/ to class /QueryEntity/
- The method marks Cache API Key and Value fields that SQL API must
initialize (on INSERT/MERGE) and update (on UPDATE/MERGE) together.
- It is still possible to use Cache API to initialize the fields marked with
/setKeyValueFields/ to different values. SQL SELECT statement returns value
of such a field from the Key entity.
- The method accepts a set of field names and returns the declaring class
instance for chaining.
- The method throws /ArgumentException/ if the Key and Value types are
available and the field types are different within the Key and Value
entities.
*SQL API*: add /KEY_VALUE_FIELDS/ parameter to /CREATE TABLE/ statement's
additional parameters list.
- The parameter's value is a space-separated list of field names with the
semantics equivalent to that of the /setKeyValueFields/ method described
above.
- The parameter can be specified only if both the /KEY_TYPE/ and
/VALUE_TYPE/ parameters are specified.
Let's take a look on the following example:
Consider a business domain entity Person { id: int, passportNo: String,
name: String }
Suppose an application development team decided to map the Person entity to
Ignite data model as:
PersonKey { id: int, passportNo: String }
Person { passportNo: String, name: String }
Use Case - inserting into Key and Value fields with same name initializes
both fields in QueryEntity-Defined Cache
- GIVEN a Person cache from the example data model configured like this in
Ignite:
new CacheConfiguration<PersonKey, Person>("CACHE")
.setQueryEntities(Collections.singleton(
new QueryEntity(PersonKey.class, Person.class)
.addQueryField("id", int.class.getName(), null)
.addQueryField("passportNo", String.class.getName(),
null)
.addQueryField("name", String.class.getName(), null)
.setKeyFields(Collections.singleton("id"))
.setKeyValueFields(Collections.singleton("passportNo"))
));
- AND an entry is added to the cache with this SQL statement:
INSERT INTO CACHE.Person (ID, PASSPORTNO, NAME) VALUES (1, '11111',
'Name1')
- WHEN the user gets the entity using Cache API:
final PersonKey K = new PersonKey(1, "11111");
Person v = cache.get(K);
- THEN the *passportNo* field is initialized to the same value within the
key and value entities:
assertEquals(K.passportNo, v.passportNo);
Other use cases with inserting and querying could be found at [1].
[1] https://issues.apache.org/jira/browse/IGNITE-12807
WDYT?
Will it be a useful feature for the Apache Ignite?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Sasov <sa...@gmail.com>.
Hello, Denis. This is the analogy of DB records. We've got keys in the table.
But querying the record with the key we also have this key in the returned
record. The question of having all data in the value is a domain driven one.
If i have a model (ex. Person) containing PassportNo, I don't want to change
the model just to store it in cache. A lot of outer code uses these classes,
so i have to either create an independent model layer to store it in cache,
or copy data from keys to values before returning object to callers.
Thanks.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Sasov <sa...@gmail.com>.
Denis, hello. Thanks for your explanation. I can see some difficulties in
matching your new idea with the current functionality of key-value approach.
From this point of view, Alexey Kukushkin's approach looks less confusing to
me and easier to implement. But this could be due to the lack of
understanding of the product. In general, i'm also quite comfortable with
your way of performing the feature.
Thanks in advance, Alexey.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Nikolay Izhikov <ni...@apache.org>.
> Nikolay, btw, what's your thinking on the approach proposed by me?
Makes sense.
> 26 мая 2020 г., в 14:07, Alexey Sasov <sa...@gmail.com> написал(а):
>
> Hello. Looks good. What whould be returned from the query like this "select
> _key from cache" if we don't know the type of the key? ExpandoObject, Tuple,
> BinaryObject? How would you change ICache interface which has this member?
>
> /// <summary>Queries cache.</summary>
> /// Query.
> /// <returns>Cursor.</returns>
> IQueryCursor<ICacheEntry<TK, TV>> Query(QueryBase qry);
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Denis Magda <dm...@apache.org>.
Hi Alexey,
Let me answer the second question first
How would you change ICache interface which has this member?
We will keep the key-value interface at least for backward compatibility.
Plus, it might be preferred by those who use caching APIs only. So, this
method will stay as-is.
What whould be returned from the query like this "select
> _key from cache" if we don't know the type of the key? ExpandoObject,
> Tuple,
> BinaryObject?
I would go for BinaryObject. Also, I would review if we really need to make
"_key" visible for application developers.
-
Denis
On Tue, May 26, 2020 at 4:22 AM Alexey Sasov <sa...@gmail.com> wrote:
> Hello. Looks good. What whould be returned from the query like this "select
> _key from cache" if we don't know the type of the key? ExpandoObject,
> Tuple,
> BinaryObject? How would you change ICache interface which has this member?
>
> /// <summary>Queries cache.</summary>
> /// Query.
> /// <returns>Cursor.</returns>
> IQueryCursor<ICacheEntry<TK, TV>> Query(QueryBase qry);
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Sasov <sa...@gmail.com>.
Hello. Looks good. What whould be returned from the query like this "select
_key from cache" if we don't know the type of the key? ExpandoObject, Tuple,
BinaryObject? How would you change ICache interface which has this member?
/// <summary>Queries cache.</summary>
/// Query.
/// <returns>Cursor.</returns>
IQueryCursor<ICacheEntry<TK, TV>> Query(QueryBase qry);
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Kukushkin <ku...@gmail.com>.
Denis, I like your proposal and will be glad to participate in developing a
solution in that way.
In additional to Nikolay's opinion about your ideas I think it is important
to get Dmitry Frolov's (who started the discussion) and Alexey Sasov's (who
also voted for the having this enhancement) opinions.
Dmitry F, Alexey S, suppose we solve this like Denis suggested - will you
then be happy to use the new API instead of fixing the existing K/V API? Or
you still think the K/V API has to be fixed?
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Denis Magda <dm...@apache.org>.
Alexey,
If you support my idea, why don't we go ahead and support it for upcoming
Ignite releases? We'll preserve existing key-value APIs for now and will
rather expand them with a new set of methods that no longer require to pass
a key object. We'll be glad to work together with you on an IEP.
Nikolay, btw, what's your thinking on the approach proposed by me?
-
Denis
On Fri, May 22, 2020 at 1:12 AM Alexey Kukushkin <ku...@gmail.com>
wrote:
> Nikolay,
>
> I think you and I agree here: there is no problem at all if the user uses
> K/Vs in domain model OR developed a domain-to-K/V mapper OR does not use
> SQL
> DML.
>
> I think we just have different experience as to how "popular" is the user's
> unwillingness to use one of the three existing approaches and desire to
> have
> that addressed in Ignite.
>
> Let's see what other community members think.
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Kukushkin <ku...@gmail.com>.
Nikolay,
I think you and I agree here: there is no problem at all if the user uses
K/Vs in domain model OR developed a domain-to-K/V mapper OR does not use SQL
DML.
I think we just have different experience as to how "popular" is the user's
unwillingness to use one of the three existing approaches and desire to have
that addressed in Ignite.
Let's see what other community members think.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Nikolay Izhikov <ni...@apache.org>.
Hello, Alexey.
Thanks for your feedback and clarification.
> you have to explicitly specify fields having the same name inside the Key and Value.
By all of this we just make Ignite more complex in favor of single, wrong use-case.
Moreover, you hurt another use-cases with this feature.
I don’t think it’s a right for the Ignite.
Let’s hear another opinion.
> - Many are the existing (legacy) apps adding Ignite as one of the pieces.
But in your example POJOs that are developed especially for Ignite.
> I already mentioned popular problems should be solved in re-usable parts
I don’t think it's a popular issue.
From my understanding it’s a union of the following design choices:
1. Don’t want to pass both KEY and VALUE objects to the application logic layer - therefore we want to duplicate some fields both in KEY and in VALUE.
2. Want to use SQL Updates to manage KV objects.
Both of these decisions is questionable:
1. Can we work with two objects on the business logic layer?
2. Can we combine KEY and VALUE in the single object before passing it on the application logic layer?
3. Updating entities from SQL has significant downsides - no transactions support, no atomic changes, etc.
> 22 мая 2020 г., в 10:06, Alexey Kukushkin <ku...@gmail.com> написал(а):
>
>> «Why SQL sets both fields? I just want to set the key object field!»
>
> The solution from IGNITE-12807 has backward compatible API: you have to
> explicitly specify fields having same name inside the Key and Value.
> Otherwise the semantics is as is.
>
> Denis's solution (that I like) is a new API not changing the K/V API at all.
>
>> We want to make Ignite(or any database) too smart.
>
> I agree that introducing a new feature or even a new small configuration
> setting often makes a product more complex: more expensive to learn and
> maintain. One should think hard whether benefits of some new function
> overweight added complexity to the product.
>
> To me in this case complexity and costs of the new configuration setting
> (KeyValue field list in QueryEntity and CREATE TABLE params) justifies the
> value for the users.
>
> In my experience "just rename the fields" is a huge inconvenience for real
> applications:
>
> - Many are the existing (legacy) apps adding Ignite as one of the pieces.
> Many are large enterprise apps with many thousands lines of code deployed in
> mission critical PROD environments. Such apps already have a domain model
> used by other enterprise components. "Renaming a field" in the domain model
> entities is just not a practical solution for them. They will choose
> developing a data access layer to map domain model to K/Vs.
>
> - As for having the user to always develop a data access layer - I already
> mentioned popular problems should be solved in re-usable parts, which is
> Ignite in this case. Besides, additional layers have performance and memory
> impact and the suggested solution has none.
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Kukushkin <ku...@gmail.com>.
> «Why SQL sets both fields? I just want to set the key object field!»
The solution from IGNITE-12807 has backward compatible API: you have to
explicitly specify fields having same name inside the Key and Value.
Otherwise the semantics is as is.
Denis's solution (that I like) is a new API not changing the K/V API at all.
> We want to make Ignite(or any database) too smart.
I agree that introducing a new feature or even a new small configuration
setting often makes a product more complex: more expensive to learn and
maintain. One should think hard whether benefits of some new function
overweight added complexity to the product.
To me in this case complexity and costs of the new configuration setting
(KeyValue field list in QueryEntity and CREATE TABLE params) justifies the
value for the users.
In my experience "just rename the fields" is a huge inconvenience for real
applications:
- Many are the existing (legacy) apps adding Ignite as one of the pieces.
Many are large enterprise apps with many thousands lines of code deployed in
mission critical PROD environments. Such apps already have a domain model
used by other enterprise components. "Renaming a field" in the domain model
entities is just not a practical solution for them. They will choose
developing a data access layer to map domain model to K/Vs.
- As for having the user to always develop a data access layer - I already
mentioned popular problems should be solved in re-usable parts, which is
Ignite in this case. Besides, additional layers have performance and memory
impact and the suggested solution has none.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Nikolay Izhikov <ni...@apache.org>.
> Personally, I treat this as an anti-pattern that should have been influenced by some of the old Ignite examples.
+1
I don’t see any reason to support fields with the same name both in key and in value from SQL.
Moreover, I think if we implement this feature then for some part of the users it will be confusing.
«Why SQL sets both fields? I just want to set the key object field!»
It seems to me like one of the examples where
1. The issue can be easily fixed on the application side(just rename the fields)
2. We want to make Ignite(or any database) too smart.
Please, consider renaming of the key or value field.
> 22 мая 2020 г., в 00:14, Alexey Kukushkin <ku...@gmail.com> написал(а):
>
> Denis,I like your idea. It would be like native "OKVM" (Object-KeyValue
> Mapper) allowing Ignite to natively operate on domain entities. It also
> means no key and value data duplication and thus addressing the original SQL
> DML problem when only key field is modified.We could add a
> configuration-based alternative to the AOP-style API you suggested. Of all
> languages that Ignite supports I think only Java and C# have AOP. We need to
> think about all languages when we add a new feature. I think your solution
> does not completely overlap with the solution suggested in IGNITE-12807
> <https://issues.apache.org/jira/browse/IGNITE-12807> : - You suggested a
> new API that does not have a "Key" entity by definition. A huge advantage of
> the existing Key/Value is it is a widely adopted JCache standard supported
> by all Ignite competitors I am aware of. That means low cost of switching to
> Ignite for users of other product and more confidence for existing Ignite
> users. IGNITE-12807 suggests solving the problem with the Key/Value API. -
> Implementing your solution is probably a significantly bigger effort than
> what IGNITE-12807 suggests. IGNITE-12807 already has a patch with a
> proof-of-concept implemented.Still, I think the urge for addressing it in
> K/V API would be less if we have your kind of API in Java and C#.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Kukushkin <ku...@gmail.com>.
Denis,I like your idea. It would be like native "OKVM" (Object-KeyValue
Mapper) allowing Ignite to natively operate on domain entities. It also
means no key and value data duplication and thus addressing the original SQL
DML problem when only key field is modified.We could add a
configuration-based alternative to the AOP-style API you suggested. Of all
languages that Ignite supports I think only Java and C# have AOP. We need to
think about all languages when we add a new feature. I think your solution
does not completely overlap with the solution suggested in IGNITE-12807
<https://issues.apache.org/jira/browse/IGNITE-12807> : - You suggested a
new API that does not have a "Key" entity by definition. A huge advantage of
the existing Key/Value is it is a widely adopted JCache standard supported
by all Ignite competitors I am aware of. That means low cost of switching to
Ignite for users of other product and more confidence for existing Ignite
users. IGNITE-12807 suggests solving the problem with the Key/Value API. -
Implementing your solution is probably a significantly bigger effort than
what IGNITE-12807 suggests. IGNITE-12807 already has a patch with a
proof-of-concept implemented.Still, I think the urge for addressing it in
K/V API would be less if we have your kind of API in Java and C#.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Denis Magda <dm...@apache.org>.
>
> I do acknowledge that this awkward usability issue is quite annoying and
> complicates applications' architecture. Problem with Spring Data
Sorry, sent the message with the first paragraph left unfinished. "Problem
with Spring Data" - just wanted to highlight that the current separation on
key and value POJOs leads to other clumsy usability issues:
https://issues.apache.org/jira/browse/IGNITE-13028
-
Denis
On Thu, May 21, 2020 at 12:03 PM Denis Magda <dm...@apache.org> wrote:
> Folks, thanks for the details. I do acknowledge that this awkward
> usability issue is quite annoying and complicates applications'
> architecture. Problem with Spring Data
>
> At the moment, I use the workaround mentioned by Alexey Kukushkin. Guess
> many of us already use it in productions:
>
> - PersonKey and PersonValue POJOs are used by Ignite repository
> abstractions that put or read data to/from Ignite. Those POJOs don't have
> repetitive fields.
> - Person POJO class is a high-level model that has all the fields of
> the table (key + value). PersonKey and PersonValue are converted to a
> Person instance by a repository abstraction and that instance is used by
> other high-level abstractions of my applications (services, controller,
> etc.).
>
> Personally, the proposed solution for the problem still sounds complicated
> to me. It still requires me, as an application developer, to do some extra
> settings in the CacheConfiguration and CREATE TABLE statements.
>
> Let me know what you think about an alternative approach, I've been
> contemplating for a while. Those of us who use JPA, Spring Data or
> Hibernate frameworks are aware of the Entity abstraction. Which is
> basically a POJO class of your business domain model that goes with all the
> fields defined in an associated relational/NoSQL table. You use special
> annotations to define indexes, primary keys and other relations in that
> Entity class. My thinking was to upgrade Ignite cache APIs to the state
> when we no longer require to create key POJOs. Instead, you create a single
> POJO/Entity, annotate its primary key and pass that object to the cache API
> or get it back for read operations.
>
> As an example, let's assume we have this POJO:
>
> @IgnitePrimaryKey(fields = {"id", "passportNo"}, affinityKey="id")
>> class Person {
>> int id;
>> String passportNo;
>> String name;
>> }
>
>
> (the primary key fields must always be annotated so that Ignite could
> extract those fields internally).
>
> Next, if you want to write an object instance into the cache you do this:
>
> Person personInstance = new Person(5, "23324234", "Thomas");
>> ignite.cache("Person").put(personInstance);
>
>
> To read it back, you just do this
>
> Person personInstance = ignite.cache("Person").get(5, "23324234"); //the
>> method can accept an arbitrary number of fields that define a primary key).
>
>
>
> The idea is rough and not finalized but I would like us to explore the
> APIs evolution in this direction.
>
> -
> Denis
>
>
> On Thu, May 21, 2020 at 7:19 AM Alexey Kukushkin <
> kukushkinalexey@gmail.com> wrote:
>
>> I vote for implementing this enhancement: having key information separated
>> from value means either cluttering domain model with
>> KeyValue/CacheEntry-like structures or developing an Ignite data access
>> layer to insert key info into domain entity after getting it from Ignite.
>>
>> Although both the options solve the issue, I think the right approach is
>> to
>> address it in Ignite rather than make all the Ignite users repeatedly deal
>> with the issue. It is the Ignite which is a re-usable platform and not
>> vise-versa.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>
>
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Denis Magda <dm...@apache.org>.
Folks, thanks for the details. I do acknowledge that this awkward usability
issue is quite annoying and complicates applications' architecture. Problem
with Spring Data
At the moment, I use the workaround mentioned by Alexey Kukushkin. Guess
many of us already use it in productions:
- PersonKey and PersonValue POJOs are used by Ignite repository
abstractions that put or read data to/from Ignite. Those POJOs don't have
repetitive fields.
- Person POJO class is a high-level model that has all the fields of the
table (key + value). PersonKey and PersonValue are converted to a Person
instance by a repository abstraction and that instance is used by other
high-level abstractions of my applications (services, controller, etc.).
Personally, the proposed solution for the problem still sounds complicated
to me. It still requires me, as an application developer, to do some extra
settings in the CacheConfiguration and CREATE TABLE statements.
Let me know what you think about an alternative approach, I've been
contemplating for a while. Those of us who use JPA, Spring Data or
Hibernate frameworks are aware of the Entity abstraction. Which is
basically a POJO class of your business domain model that goes with all the
fields defined in an associated relational/NoSQL table. You use special
annotations to define indexes, primary keys and other relations in that
Entity class. My thinking was to upgrade Ignite cache APIs to the state
when we no longer require to create key POJOs. Instead, you create a single
POJO/Entity, annotate its primary key and pass that object to the cache API
or get it back for read operations.
As an example, let's assume we have this POJO:
@IgnitePrimaryKey(fields = {"id", "passportNo"}, affinityKey="id")
> class Person {
> int id;
> String passportNo;
> String name;
> }
(the primary key fields must always be annotated so that Ignite could
extract those fields internally).
Next, if you want to write an object instance into the cache you do this:
Person personInstance = new Person(5, "23324234", "Thomas");
> ignite.cache("Person").put(personInstance);
To read it back, you just do this
Person personInstance = ignite.cache("Person").get(5, "23324234"); //the
> method can accept an arbitrary number of fields that define a primary key).
The idea is rough and not finalized but I would like us to explore the APIs
evolution in this direction.
-
Denis
On Thu, May 21, 2020 at 7:19 AM Alexey Kukushkin <ku...@gmail.com>
wrote:
> I vote for implementing this enhancement: having key information separated
> from value means either cluttering domain model with
> KeyValue/CacheEntry-like structures or developing an Ignite data access
> layer to insert key info into domain entity after getting it from Ignite.
>
> Although both the options solve the issue, I think the right approach is to
> address it in Ignite rather than make all the Ignite users repeatedly deal
> with the issue. It is the Ignite which is a re-usable platform and not
> vise-versa.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Kukushkin <ku...@gmail.com>.
I vote for implementing this enhancement: having key information separated
from value means either cluttering domain model with
KeyValue/CacheEntry-like structures or developing an Ignite data access
layer to insert key info into domain entity after getting it from Ignite.
Although both the options solve the issue, I think the right approach is to
address it in Ignite rather than make all the Ignite users repeatedly deal
with the issue. It is the Ignite which is a re-usable platform and not
vise-versa.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Denis Magda <dm...@apache.org>.
Hi Dmitry,
PersonKey { id: int, passportNo: String }
> Person { passportNo: String, name: String }
What's the primary reason for keeping a copy of "passportNo" field in the
value object? Personally, I treat this as an anti-pattern that should have
been influenced by some of the old Ignite examples.
-
Denis
On Wed, May 20, 2020 at 7:19 AM Alexey Sasov <sa...@gmail.com> wrote:
> Hello. Sure this feature is of great importance and we'd like to get it
> asap.
> It is very confusing to support the logic of copying values between key and
> value classes.
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
Re: [DISCUSSION] Key and Value fields with same name and SQL DML
Posted by Alexey Sasov <sa...@gmail.com>.
Hello. Sure this feature is of great importance and we'd like to get it asap.
It is very confusing to support the logic of copying values between key and
value classes.
Thanks.
--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/