You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Cong Guo <co...@huawei.com> on 2018/06/15 13:29:13 UTC
SQL cannot find data of new class definition
Hi all,
I am trying to use BinaryObject to support data of different class definitions in one cache. This is for my system upgrade. I first start a cache with data, and then launch a new node to join the cluster and put new data into the existing cache. The new data has a different class definition. I use the same class name, but add a member to the class. I can add the objects of this new class to the cache. The cache size changes and I can get both the new and old data using keys. However, when I use SQLFieldsQuery like "select id from myclassname" where id is a member exists in both the versions of classes, I can get only the old data. There is no error or exception. SQL just cannot find the data of the new class definition.
How can I use SQL queries to find both the new and old data? The new data is in the cache, but it seems not being in the table using my class name. Where is the new data? Is there a new table? If yes, what is the table name? I do not expect to see the new column using the old query. I just hope to see the old fields of new data using the old queries.
BTW, I use QueryEntity to set up fields and indexes in my codes. Does anyone has an example about how to add fields to existing cache dynamically? Thank you!
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
Hi,
Thank you for the reply. In my original test, I do not create a table using SQL. I just create a cache. I think a table using the value class name is created implicitely. I add the new field/column using ALTER TABLE before I put new data into the cache, but I still cannot find the data of the new class in the table with the class name.
It is easy to reproduce my original test. I use the Person class from ignite example.
In the old code:
CacheConfiguration<Long, Person> personCacheCfg = new CacheConfiguration<>(PERSON_CACHE_NAME);
personCacheCfg.setCacheMode(CacheMode.REPLICATED);
personCacheCfg.setQueryEntities(Arrays.asList(createPersonQueryEntity()));
try(IgniteCache<Long, Person> personCache = ignite.getOrCreateCache(personCacheCfg)){
// add some data here
Person p1 = new Person(…);
personCache.put(1L, p1);
// keep the node running and run the SQL query
}
private static QueryEntity createPersonQueryEntity() {
QueryEntity personEntity = new QueryEntity();
personEntity.setValueType(Person.class.getName());
personEntity.setKeyType(Long.class.getName());
LinkedHashMap<String, String> fields = new LinkedHashMap<>();
fields.put("id", Long.class.getName());
fields.put("orgId", Long.class.getName());
fields.put("firstName", String.class.getName());
fields.put("lastName", String.class.getName());
fields.put("resume", String.class.getName());
fields.put("salary", Double.class.getName());
personEntity.setFields(fields);
personEntity.setIndexes(Arrays.asList(
new QueryIndex("id"),
new QueryIndex("orgId")
));
return personEntity;
}
The SQL query is:
IgniteCache<BinaryObject, BinaryObject> binaryCache = personCache.withKeepBinary();
SqlFieldsQuery qry = new SqlFieldsQuery("select salary from Person");
QueryCursor<List<?>> answers = binaryCache.query(qry);
List<List<?>> salaryList = answers.getAll();
for(List<?> row : salaryList) {
Double salary = (Double)row.get(0);
System.out.println(salary);
}
In the new code:
I add a member to the Person class which is “private in addOn”.
try(IgniteCache<Long, Person> personCache = ignite.cache(PERSON_CACHE_NAME)){
// add the new data and then check the cache size
Person p2 = new Person(…);
personCache.put(2L, p2);
System.out.println("Size of the cache is: " + personCache.size(CachePeekMode.ALL));
}
I can only get the data of the old class P1 using the SQL query, but there is no error.
I use BinaryObject in the first place because the document says BinaryObject “enables you to add and remove fields from objects of the same type”
https://apacheignite.readme.io/docs/binary-marshaller
I can get the data of different class definitions using get(key), but I also need the SQL fields query.
IgniteCache<Long, BinaryObject> binaryCache = personCache.<Long, BinaryObject>withKeepBinary();
BinaryObject bObj = binaryCache.get(1L);
System.out.println(bObj.type().field("firstName").value(bObj) + " " + bObj.type().field("salary").value(bObj));
System.out.println("" + bObj.type().field("addON").value(bObj));
BinaryObject bObj2 = binaryCache.get(2L);
System.out.println(bObj2.type().field("firstName").value(bObj2) + " " + bObj2.type().field("salary").value(bObj2));
System.out.println("" + bObj2.type().field("addON").value(bObj2));
Thanks,
Cong
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: 2018年6月15日 9:37
To: user@ignite.apache.org
Subject: Re: SQL cannot find data of new class definition
Hello!
You can add fields to existing SQL-backed cache using ALTER TABLE ... ADD COLUMN command:
https://apacheignite-sql.readme.io/docs/alter-table
The recommendation for your use case, where the layout of dat1a is expected to change, is to just use SQL (DDL) defined tables and forget about BinaryObject's.
With regards to your original case, i.e., a different class definition: I could spend time debugging it if you had more info, but this approach is not recommended anyway.
Regards,
--
Ilya Kasnacheev
2018-06-15 16:29 GMT+03:00 Cong Guo <co...@huawei.com>>:
Hi all,
I am trying to use BinaryObject to support data of different class definitions in one cache. This is for my system upgrade. I first start a cache with data, and then launch a new node to join the cluster and put new data into the existing cache. The new data has a different class definition. I use the same class name, but add a member to the class. I can add the objects of this new class to the cache. The cache size changes and I can get both the new and old data using keys. However, when I use SQLFieldsQuery like “select id from myclassname” where id is a member exists in both the versions of classes, I can get only the old data. There is no error or exception. SQL just cannot find the data of the new class definition.
How can I use SQL queries to find both the new and old data? The new data is in the cache, but it seems not being in the table using my class name. Where is the new data? Is there a new table? If yes, what is the table name? I do not expect to see the new column using the old query. I just hope to see the old fields of new data using the old queries.
BTW, I use QueryEntity to set up fields and indexes in my codes. Does anyone has an example about how to add fields to existing cache dynamically? Thank you!
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
I run the SQL query only after the cache size has changed. The new data should be already in the cache when I run the query.
From: Cong Guo
Sent: 2018年6月15日 10:01
To: user@ignite.apache.org
Subject: RE: SQL cannot find data of new class definition
Hi,
Thank you for the reply. In my original test, I do not create a table using SQL. I just create a cache. I think a table using the value class name is created implicitely. I add the new field/column using ALTER TABLE before I put new data into the cache, but I still cannot find the data of the new class in the table with the class name.
It is easy to reproduce my original test. I use the Person class from ignite example.
In the old code:
CacheConfiguration<Long, Person> personCacheCfg = new CacheConfiguration<>(PERSON_CACHE_NAME);
personCacheCfg.setCacheMode(CacheMode.REPLICATED);
personCacheCfg.setQueryEntities(Arrays.asList(createPersonQueryEntity()));
try(IgniteCache<Long, Person> personCache = ignite.getOrCreateCache(personCacheCfg)){
// add some data here
Person p1 = new Person(…);
personCache.put(1L, p1);
// keep the node running and run the SQL query
}
private static QueryEntity createPersonQueryEntity() {
QueryEntity personEntity = new QueryEntity();
personEntity.setValueType(Person.class.getName());
personEntity.setKeyType(Long.class.getName());
LinkedHashMap<String, String> fields = new LinkedHashMap<>();
fields.put("id", Long.class.getName());
fields.put("orgId", Long.class.getName());
fields.put("firstName", String.class.getName());
fields.put("lastName", String.class.getName());
fields.put("resume", String.class.getName());
fields.put("salary", Double.class.getName());
personEntity.setFields(fields);
personEntity.setIndexes(Arrays.asList(
new QueryIndex("id"),
new QueryIndex("orgId")
));
return personEntity;
}
The SQL query is:
IgniteCache<BinaryObject, BinaryObject> binaryCache = personCache.withKeepBinary();
SqlFieldsQuery qry = new SqlFieldsQuery("select salary from Person");
QueryCursor<List<?>> answers = binaryCache.query(qry);
List<List<?>> salaryList = answers.getAll();
for(List<?> row : salaryList) {
Double salary = (Double)row.get(0);
System.out.println(salary);
}
In the new code:
I add a member to the Person class which is “private in addOn”.
try(IgniteCache<Long, Person> personCache = ignite.cache(PERSON_CACHE_NAME)){
// add the new data and then check the cache size
Person p2 = new Person(…);
personCache.put(2L, p2);
System.out.println("Size of the cache is: " + personCache.size(CachePeekMode.ALL));
}
I can only get the data of the old class P1 using the SQL query, but there is no error.
I use BinaryObject in the first place because the document says BinaryObject “enables you to add and remove fields from objects of the same type”
https://apacheignite.readme.io/docs/binary-marshaller
I can get the data of different class definitions using get(key), but I also need the SQL fields query.
IgniteCache<Long, BinaryObject> binaryCache = personCache.<Long, BinaryObject>withKeepBinary();
BinaryObject bObj = binaryCache.get(1L);
System.out.println(bObj.type().field("firstName").value(bObj) + " " + bObj.type().field("salary").value(bObj));
System.out.println("" + bObj.type().field("addON").value(bObj));
BinaryObject bObj2 = binaryCache.get(2L);
System.out.println(bObj2.type().field("firstName").value(bObj2) + " " + bObj2.type().field("salary").value(bObj2));
System.out.println("" + bObj2.type().field("addON").value(bObj2));
Thanks,
Cong
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: 2018年6月15日 9:37
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: SQL cannot find data of new class definition
Hello!
You can add fields to existing SQL-backed cache using ALTER TABLE ... ADD COLUMN command:
https://apacheignite-sql.readme.io/docs/alter-table
The recommendation for your use case, where the layout of dat1a is expected to change, is to just use SQL (DDL) defined tables and forget about BinaryObject's.
With regards to your original case, i.e., a different class definition: I could spend time debugging it if you had more info, but this approach is not recommended anyway.
Regards,
--
Ilya Kasnacheev
2018-06-15 16:29 GMT+03:00 Cong Guo <co...@huawei.com>>:
Hi all,
I am trying to use BinaryObject to support data of different class definitions in one cache. This is for my system upgrade. I first start a cache with data, and then launch a new node to join the cluster and put new data into the existing cache. The new data has a different class definition. I use the same class name, but add a member to the class. I can add the objects of this new class to the cache. The cache size changes and I can get both the new and old data using keys. However, when I use SQLFieldsQuery like “select id from myclassname” where id is a member exists in both the versions of classes, I can get only the old data. There is no error or exception. SQL just cannot find the data of the new class definition.
How can I use SQL queries to find both the new and old data? The new data is in the cache, but it seems not being in the table using my class name. Where is the new data? Is there a new table? If yes, what is the table name? I do not expect to see the new column using the old query. I just hope to see the old fields of new data using the old queries.
BTW, I use QueryEntity to set up fields and indexes in my codes. Does anyone has an example about how to add fields to existing cache dynamically? Thank you!
RE: SQL cannot find data of new class definition
Posted by "slava.koptilin" <sl...@gmail.com>.
Hello,
> So when we need to add a new field, say in "addr", we can use current
> ALTER TABLE to add a normal column,
> but now the problem is how to map the new field to the column.
Yes, that is a problem. ALTER TABLE command does not support adding new
columns to nested objects.
So, if you need to update SQL schema dynamically, then I would suggest
avoiding nested objects.
> I think the problem is to support dynamic update of fields in QueryEntity.
I am afraid that there is no plan to support that. At least, I didn't find a
ticket in JIRA.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
Hi,
I don't think this feature requires any change in the SQL API.
When we create a cache, even if the value object contains a nested object, the fields in the nested object can be mapped to columns in the table. Now we can do this using QueryEntity, for example,
QueryEntity personEntity = new QueryEntity();
personEntity.setValueType(Person.class.getName());
personEntity.setKeyType(Long.class.getName());
LinkedHashMap<String, String> fields = new LinkedHashMap<>();
fields.put("addr.streetNum", Integer.class.getName());
fields.put("addr.streetName", String.class.getName());
personEntity.setFields(fields);
There will be two columns named streetNum and streetName in the table automatically.
So when we need to add a new field, say in "addr", we can use current ALTER TABLE to add a normal column, but now the problem is how to map the new field to the column. Now we cannot modify the QueryEntity dynamically, right? I think the problem is to support dynamic update of fields in QueryEntity.
-----Original Message-----
From: slava.koptilin [mailto:slava.koptilin@gmail.com]
Sent: 2018年6月21日 11:27
To: user@ignite.apache.org
Subject: RE: SQL cannot find data of new class definition
Hello Cong,
> when we add a field to the first-level value object and add a column
> to the table dynamically, they can be connected automatically.
Yes, that is correct.
> So now the problem is when we add a field to the nested object and add
> a column to the table, they cannot be connected automatically.
It cannot be done via SQL API at runtime.
The reason for that constraint is that this feature requires custom SQL syntax which is not SQL ANSI-99 obviously, and I don't think there are any plans to support this feature.
Perhaps, it makes sense to start a discussion on the dev list.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by "slava.koptilin" <sl...@gmail.com>.
Hello Cong,
> when we add a field to the first-level value object and add a column to
> the table dynamically, they can be connected automatically.
Yes, that is correct.
> So now the problem is when we add a field to the nested object and add a
> column to the table,
> they cannot be connected automatically.
It cannot be done via SQL API at runtime.
The reason for that constraint is that this feature requires custom SQL
syntax which is not SQL ANSI-99 obviously, and I don't think there are any
plans to support this feature.
Perhaps, it makes sense to start a discussion on the dev list.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
Hi,
I think we can map members in a nested object to table columns when we create the cache. And when we add a field to the first-level value object (which contains the nested object) and add a column to the table dynamically, they can be connected automatically. So now the problem is when we add a field to the nested object and add a column to the table, they cannot be connected automatically. May I know how this part is implemented in Ignite? Could you please create a ticket and fix this in the future?
Thanks,
Cong
-----Original Message-----
From: slava.koptilin [mailto:slava.koptilin@gmail.com]
Sent: 2018年6月21日 9:25
To: user@ignite.apache.org
Subject: RE: SQL cannot find data of new class definition
Hello,
> How should I write the "alter table Person" statement if I want to add
> a new member to class Address after the cache has been created?
I don't think that there is a way to do it for nested objects, unfortunately.
In that case, I think that you need to update your configuration [1] and restart the cluster.
[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-annotation-based-configuration
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by "slava.koptilin" <sl...@gmail.com>.
Hello,
> How should I write the "alter table Person" statement if I want to add a
> new member
> to class Address after the cache has been created?
I don't think that there is a way to do it for nested objects,
unfortunately.
In that case, I think that you need to update your configuration [1] and
restart the cluster.
[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-annotation-based-configuration
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
Thank you for your reply! This method works, but I have another question. If the Person class has a member which is another class, say Address. Class Address has two members int streetNo and String streetName. When I create the cache, I can use QueryEntity to map the members of Address to table columns/fields. How should I write the "alter table Person" statement if I want to add a new member to class Address after the cache has been created? Thank you!
-----Original Message-----
From: slava.koptilin [mailto:slava.koptilin@gmail.com]
Sent: 2018年6月20日 14:50
To: user@ignite.apache.org
Subject: RE: SQL cannot find data of new class definition
Hello,
> Can I add fields without restarting the cluster?
Yes, It can be done via DDL command, as Ilya Kasnacheev mentioned.
Let's assume that you created a cache:
CacheConfiguration cfg = new CacheConfiguration(PERSON_CACHE_NAME)
.setIndexedTypes(Long.class, Person.class);
IgniteCache cache = ignite.getOrCreateCache(cfg);
where the Person class has two fields 'id' and 'firstName'.
after that, you want to add a new field, for example, 'secondName'.
// please take a look for the details:
https://apacheignite-sql.readme.io/docs/alter-table
String ddl = "alter table Person add column secondName varchar";
// execute the DDL command
cache.query(new SqlFieldsQuery(ddl)).getAll();
// new field should be queryable
Iterator iter = cache.query(new SqlFieldsQuery("select secondName from Person")).getAll());
hope it helps.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by "slava.koptilin" <sl...@gmail.com>.
Hello,
> Can I add fields without restarting the cluster?
Yes, It can be done via DDL command, as Ilya Kasnacheev mentioned.
Let's assume that you created a cache:
CacheConfiguration cfg = new CacheConfiguration(PERSON_CACHE_NAME)
.setIndexedTypes(Long.class, Person.class);
IgniteCache cache = ignite.getOrCreateCache(cfg);
where the Person class has two fields 'id' and 'firstName'.
after that, you want to add a new field, for example, 'secondName'.
// please take a look for the details:
https://apacheignite-sql.readme.io/docs/alter-table
String ddl = "alter table Person add column secondName varchar";
// execute the DDL command
cache.query(new SqlFieldsQuery(ddl)).getAll();
// new field should be queryable
Iterator iter = cache.query(new SqlFieldsQuery("select secondName from
Person")).getAll());
hope it helps.
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
Can I add fields without restarting the cluster? My requirement is to do rolling upgrade.
From: Вячеслав Коптилин [mailto:slava.koptilin@gmail.com]
Sent: 2018年6月18日 17:35
To: user@ignite.apache.org
Subject: Re: SQL cannot find data of new class definition
Hello,
> I use BinaryObject in the first place because the document says BinaryObject “enables you to add and remove fields from objects of the same type”
Yes, you can dynamically add fields to BinaryObject using BinaryObjecyBuilder, but fields that you want to query have to be specified on node startup for example through QueryEntity.
Please take a look at this page: https://apacheignite.readme.io/v2.5/docs/indexes#queryentity-based-configuration
I would suggest specifying a new field via QueryEntity in XML configuration file and restart your cluster. I hope it helps.
Thanks!
пн, 18 июн. 2018 г. в 16:47, Cong Guo <co...@huawei.com>>:
Hi,
Does anyone have experience using both Cache and SQL interfaces at the same time? How do you solve the possible upgrade? Is my problem a bug for BinaryObject? Should I debug the ignite source code?
From: Cong Guo
Sent: 2018年6月15日 10:12
To: 'user@ignite.apache.org<ma...@ignite.apache.org>' <us...@ignite.apache.org>>
Subject: RE: SQL cannot find data of new class definition
I run the SQL query only after the cache size has changed. The new data should be already in the cache when I run the query.
From: Cong Guo
Sent: 2018年6月15日 10:01
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: RE: SQL cannot find data of new class definition
Hi,
Thank you for the reply. In my original test, I do not create a table using SQL. I just create a cache. I think a table using the value class name is created implicitely. I add the new field/column using ALTER TABLE before I put new data into the cache, but I still cannot find the data of the new class in the table with the class name.
It is easy to reproduce my original test. I use the Person class from ignite example.
In the old code:
CacheConfiguration<Long, Person> personCacheCfg = new CacheConfiguration<>(PERSON_CACHE_NAME);
personCacheCfg.setCacheMode(CacheMode.REPLICATED);
personCacheCfg.setQueryEntities(Arrays.asList(createPersonQueryEntity()));
try(IgniteCache<Long, Person> personCache = ignite.getOrCreateCache(personCacheCfg)){
// add some data here
Person p1 = new Person(…);
personCache.put(1L, p1);
// keep the node running and run the SQL query
}
private static QueryEntity createPersonQueryEntity() {
QueryEntity personEntity = new QueryEntity();
personEntity.setValueType(Person.class.getName());
personEntity.setKeyType(Long.class.getName());
LinkedHashMap<String, String> fields = new LinkedHashMap<>();
fields.put("id", Long.class.getName());
fields.put("orgId", Long.class.getName());
fields.put("firstName", String.class.getName());
fields.put("lastName", String.class.getName());
fields.put("resume", String.class.getName());
fields.put("salary", Double.class.getName());
personEntity.setFields(fields);
personEntity.setIndexes(Arrays.asList(
new QueryIndex("id"),
new QueryIndex("orgId")
));
return personEntity;
}
The SQL query is:
IgniteCache<BinaryObject, BinaryObject> binaryCache = personCache.withKeepBinary();
SqlFieldsQuery qry = new SqlFieldsQuery("select salary from Person");
QueryCursor<List<?>> answers = binaryCache.query(qry);
List<List<?>> salaryList = answers.getAll();
for(List<?> row : salaryList) {
Double salary = (Double)row.get(0);
System.out.println(salary);
}
In the new code:
I add a member to the Person class which is “private in addOn”.
try(IgniteCache<Long, Person> personCache = ignite.cache(PERSON_CACHE_NAME)){
// add the new data and then check the cache size
Person p2 = new Person(…);
personCache.put(2L, p2);
System.out.println("Size of the cache is: " + personCache.size(CachePeekMode.ALL));
}
I can only get the data of the old class P1 using the SQL query, but there is no error.
I use BinaryObject in the first place because the document says BinaryObject “enables you to add and remove fields from objects of the same type”
https://apacheignite.readme.io/docs/binary-marshaller
I can get the data of different class definitions using get(key), but I also need the SQL fields query.
IgniteCache<Long, BinaryObject> binaryCache = personCache.<Long, BinaryObject>withKeepBinary();
BinaryObject bObj = binaryCache.get(1L);
System.out.println(bObj.type().field("firstName").value(bObj) + " " + bObj.type().field("salary").value(bObj));
System.out.println("" + bObj.type().field("addON").value(bObj));
BinaryObject bObj2 = binaryCache.get(2L);
System.out.println(bObj2.type().field("firstName").value(bObj2) + " " + bObj2.type().field("salary").value(bObj2));
System.out.println("" + bObj2.type().field("addON").value(bObj2));
Thanks,
Cong
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: 2018年6月15日 9:37
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: SQL cannot find data of new class definition
Hello!
You can add fields to existing SQL-backed cache using ALTER TABLE ... ADD COLUMN command:
https://apacheignite-sql.readme.io/docs/alter-table
The recommendation for your use case, where the layout of dat1a is expected to change, is to just use SQL (DDL) defined tables and forget about BinaryObject's.
With regards to your original case, i.e., a different class definition: I could spend time debugging it if you had more info, but this approach is not recommended anyway.
Regards,
--
Ilya Kasnacheev
2018-06-15 16:29 GMT+03:00 Cong Guo <co...@huawei.com>>:
Hi all,
I am trying to use BinaryObject to support data of different class definitions in one cache. This is for my system upgrade. I first start a cache with data, and then launch a new node to join the cluster and put new data into the existing cache. The new data has a different class definition. I use the same class name, but add a member to the class. I can add the objects of this new class to the cache. The cache size changes and I can get both the new and old data using keys. However, when I use SQLFieldsQuery like “select id from myclassname” where id is a member exists in both the versions of classes, I can get only the old data. There is no error or exception. SQL just cannot find the data of the new class definition.
How can I use SQL queries to find both the new and old data? The new data is in the cache, but it seems not being in the table using my class name. Where is the new data? Is there a new table? If yes, what is the table name? I do not expect to see the new column using the old query. I just hope to see the old fields of new data using the old queries.
BTW, I use QueryEntity to set up fields and indexes in my codes. Does anyone has an example about how to add fields to existing cache dynamically? Thank you!
Re: SQL cannot find data of new class definition
Posted by Вячеслав Коптилин <sl...@gmail.com>.
Hello,
> I use BinaryObject in the first place because the document says
BinaryObject “enables you to add and remove fields from objects of the same
type”
Yes, you can dynamically add fields to BinaryObject using
BinaryObjecyBuilder, but fields that you want to query have to be specified
on node startup for example through QueryEntity.
Please take a look at this page:
https://apacheignite.readme.io/v2.5/docs/indexes#queryentity-based-configuration
I would suggest specifying a new field via QueryEntity in XML configuration
file and restart your cluster. I hope it helps.
Thanks!
пн, 18 июн. 2018 г. в 16:47, Cong Guo <co...@huawei.com>:
> Hi,
>
>
>
> Does anyone have experience using both Cache and SQL interfaces at the
> same time? How do you solve the possible upgrade? Is my problem a bug for
> BinaryObject? Should I debug the ignite source code?
>
>
>
> *From:* Cong Guo
> *Sent:* 2018年6月15日 10:12
> *To:* 'user@ignite.apache.org' <us...@ignite.apache.org>
> *Subject:* RE: SQL cannot find data of new class definition
>
>
>
> I run the SQL query only after the cache size has changed. The new data
> should be already in the cache when I run the query.
>
>
>
>
>
> *From:* Cong Guo
> *Sent:* 2018年6月15日 10:01
> *To:* user@ignite.apache.org
> *Subject:* RE: SQL cannot find data of new class definition
>
>
>
> Hi,
>
>
>
> Thank you for the reply. In my original test, I do not create a table
> using SQL. I just create a cache. I think a table using the value class
> name is created implicitely. I add the new field/column using ALTER TABLE
> before I put new data into the cache, but I still cannot find the data of
> the new class in the table with the class name.
>
>
>
> It is easy to reproduce my original test. I use the Person class from
> ignite example.
>
>
>
> In the old code:
>
>
>
> CacheConfiguration<Long, Person> personCacheCfg = new
> CacheConfiguration<>(PERSON_CACHE_NAME);
>
> personCacheCfg.setCacheMode(CacheMode.REPLICATED);
>
> personCacheCfg.setQueryEntities(Arrays.asList(createPersonQueryEntity()));
>
> try(IgniteCache<Long, Person> personCache =
> ignite.getOrCreateCache(personCacheCfg)){
>
> // add some data here
>
> Person p1 = new Person(…);
>
> personCache.put(1L, p1);
>
> // keep the node running and run the SQL query
>
> }
>
>
>
> private static QueryEntity createPersonQueryEntity() {
>
> QueryEntity personEntity = new
> QueryEntity();
>
>
> personEntity.setValueType(Person.class.getName());
>
>
> personEntity.setKeyType(Long.class.getName());
>
>
>
> LinkedHashMap<String, String> fields = new
> LinkedHashMap<>();
>
> fields.put("id", Long.class.getName());
>
> fields.put("orgId", Long.class.getName());
>
> fields.put("firstName",
> String.class.getName());
>
> fields.put("lastName",
> String.class.getName());
>
> fields.put("resume",
> String.class.getName());
>
> fields.put("salary",
> Double.class.getName());
>
> personEntity.setFields(fields);
>
>
>
> personEntity.setIndexes(Arrays.asList(
>
> new
> QueryIndex("id"),
>
> new
> QueryIndex("orgId")
>
> ));
>
>
>
> return personEntity;
>
> }
>
>
>
> The SQL query is:
>
> IgniteCache<BinaryObject, BinaryObject> binaryCache =
> personCache.withKeepBinary();
>
>
> SqlFieldsQuery qry = new SqlFieldsQuery("select salary from Person");
>
>
>
>
>
> QueryCursor<List<?>> answers = binaryCache.query(qry);
>
>
> List<List<?>> salaryList = answers.getAll();
>
>
> for(List<?> row : salaryList) {
>
>
> Double salary = (Double)row.get(0);
>
>
> System.out.println(salary);
>
> }
>
>
>
> In the new code:
>
>
>
> I add a member to the Person class which is “private in addOn”.
>
>
>
> try(IgniteCache<Long, Person> personCache =
> ignite.cache(PERSON_CACHE_NAME)){
>
> // add the new data and then check the cache size
>
> Person p2 = new Person(…);
>
> personCache.put(2L, p2);
>
> System.out.println("Size of the cache is: " +
> personCache.size(CachePeekMode.ALL));
>
> }
>
>
>
> I can only get the data of the old class P1 using the SQL query, but there
> is no error.
>
>
>
> I use BinaryObject in the first place because the document says
> BinaryObject “enables you to add and remove fields from objects of the
> same type”
>
>
>
> https://apacheignite.readme.io/docs/binary-marshaller
>
>
>
> I can get the data of different class definitions using get(key), but I
> also need the SQL fields query.
>
>
>
> IgniteCache<Long, BinaryObject> binaryCache = personCache.<Long,
> BinaryObject>withKeepBinary();
>
> BinaryObject bObj = binaryCache.get(1L);
>
> System.out.println(bObj.type().field("firstName").value(bObj) + " " +
> bObj.type().field("salary").value(bObj));
>
> System.out.println("" + bObj.type().field("addON").value(bObj));
>
>
>
> BinaryObject bObj2 = binaryCache.get(2L);
>
> System.out.println(bObj2.type().field("firstName").value(bObj2) + " " +
> bObj2.type().field("salary").value(bObj2));
>
> System.out.println("" + bObj2.type().field("addON").value(bObj2));
>
>
>
>
>
>
>
> Thanks,
>
> Cong
>
>
>
>
>
>
>
> *From:* Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com
> <il...@gmail.com>]
> *Sent:* 2018年6月15日 9:37
> *To:* user@ignite.apache.org
> *Subject:* Re: SQL cannot find data of new class definition
>
>
>
> Hello!
>
>
>
> You can add fields to existing SQL-backed cache using ALTER TABLE ... ADD
> COLUMN command:
>
> https://apacheignite-sql.readme.io/docs/alter-table
>
>
>
> The recommendation for your use case, where the layout of dat1a is
> expected to change, is to just use SQL (DDL) defined tables and forget
> about BinaryObject's.
>
>
>
> With regards to your original case, i.e., a different class definition: I
> could spend time debugging it if you had more info, but this approach is
> not recommended anyway.
>
>
>
> Regards,
>
>
> --
>
> Ilya Kasnacheev
>
>
>
> 2018-06-15 16:29 GMT+03:00 Cong Guo <co...@huawei.com>:
>
> Hi all,
>
>
>
> I am trying to use BinaryObject to support data of different class
> definitions in one cache. This is for my system upgrade. I first start a
> cache with data, and then launch a new node to join the cluster and put new
> data into the existing cache. The new data has a different class
> definition. I use the same class name, but add a member to the class. I can
> add the objects of this new class to the cache. The cache size changes and
> I can get both the new and old data using keys. However, when I use
> SQLFieldsQuery like “select id from myclassname” where id is a member
> exists in both the versions of classes, I can get only the old data. There
> is no error or exception. SQL just cannot find the data of the new class
> definition.
>
>
>
> How can I use SQL queries to find both the new and old data? The new data
> is in the cache, but it seems not being in the table using my class name.
> Where is the new data? Is there a new table? If yes, what is the table
> name? I do not expect to see the new column using the old query. I just
> hope to see the old fields of new data using the old queries.
>
>
>
> BTW, I use QueryEntity to set up fields and indexes in my codes. Does
> anyone has an example about how to add fields to existing cache
> dynamically? Thank you!
>
>
>
RE: SQL cannot find data of new class definition
Posted by Cong Guo <co...@huawei.com>.
Hi,
Does anyone have experience using both Cache and SQL interfaces at the same time? How do you solve the possible upgrade? Is my problem a bug for BinaryObject? Should I debug the ignite source code?
From: Cong Guo
Sent: 2018年6月15日 10:12
To: 'user@ignite.apache.org' <us...@ignite.apache.org>
Subject: RE: SQL cannot find data of new class definition
I run the SQL query only after the cache size has changed. The new data should be already in the cache when I run the query.
From: Cong Guo
Sent: 2018年6月15日 10:01
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: RE: SQL cannot find data of new class definition
Hi,
Thank you for the reply. In my original test, I do not create a table using SQL. I just create a cache. I think a table using the value class name is created implicitely. I add the new field/column using ALTER TABLE before I put new data into the cache, but I still cannot find the data of the new class in the table with the class name.
It is easy to reproduce my original test. I use the Person class from ignite example.
In the old code:
CacheConfiguration<Long, Person> personCacheCfg = new CacheConfiguration<>(PERSON_CACHE_NAME);
personCacheCfg.setCacheMode(CacheMode.REPLICATED);
personCacheCfg.setQueryEntities(Arrays.asList(createPersonQueryEntity()));
try(IgniteCache<Long, Person> personCache = ignite.getOrCreateCache(personCacheCfg)){
// add some data here
Person p1 = new Person(…);
personCache.put(1L, p1);
// keep the node running and run the SQL query
}
private static QueryEntity createPersonQueryEntity() {
QueryEntity personEntity = new QueryEntity();
personEntity.setValueType(Person.class.getName());
personEntity.setKeyType(Long.class.getName());
LinkedHashMap<String, String> fields = new LinkedHashMap<>();
fields.put("id", Long.class.getName());
fields.put("orgId", Long.class.getName());
fields.put("firstName", String.class.getName());
fields.put("lastName", String.class.getName());
fields.put("resume", String.class.getName());
fields.put("salary", Double.class.getName());
personEntity.setFields(fields);
personEntity.setIndexes(Arrays.asList(
new QueryIndex("id"),
new QueryIndex("orgId")
));
return personEntity;
}
The SQL query is:
IgniteCache<BinaryObject, BinaryObject> binaryCache = personCache.withKeepBinary();
SqlFieldsQuery qry = new SqlFieldsQuery("select salary from Person");
QueryCursor<List<?>> answers = binaryCache.query(qry);
List<List<?>> salaryList = answers.getAll();
for(List<?> row : salaryList) {
Double salary = (Double)row.get(0);
System.out.println(salary);
}
In the new code:
I add a member to the Person class which is “private in addOn”.
try(IgniteCache<Long, Person> personCache = ignite.cache(PERSON_CACHE_NAME)){
// add the new data and then check the cache size
Person p2 = new Person(…);
personCache.put(2L, p2);
System.out.println("Size of the cache is: " + personCache.size(CachePeekMode.ALL));
}
I can only get the data of the old class P1 using the SQL query, but there is no error.
I use BinaryObject in the first place because the document says BinaryObject “enables you to add and remove fields from objects of the same type”
https://apacheignite.readme.io/docs/binary-marshaller
I can get the data of different class definitions using get(key), but I also need the SQL fields query.
IgniteCache<Long, BinaryObject> binaryCache = personCache.<Long, BinaryObject>withKeepBinary();
BinaryObject bObj = binaryCache.get(1L);
System.out.println(bObj.type().field("firstName").value(bObj) + " " + bObj.type().field("salary").value(bObj));
System.out.println("" + bObj.type().field("addON").value(bObj));
BinaryObject bObj2 = binaryCache.get(2L);
System.out.println(bObj2.type().field("firstName").value(bObj2) + " " + bObj2.type().field("salary").value(bObj2));
System.out.println("" + bObj2.type().field("addON").value(bObj2));
Thanks,
Cong
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: 2018年6月15日 9:37
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: SQL cannot find data of new class definition
Hello!
You can add fields to existing SQL-backed cache using ALTER TABLE ... ADD COLUMN command:
https://apacheignite-sql.readme.io/docs/alter-table
The recommendation for your use case, where the layout of dat1a is expected to change, is to just use SQL (DDL) defined tables and forget about BinaryObject's.
With regards to your original case, i.e., a different class definition: I could spend time debugging it if you had more info, but this approach is not recommended anyway.
Regards,
--
Ilya Kasnacheev
2018-06-15 16:29 GMT+03:00 Cong Guo <co...@huawei.com>>:
Hi all,
I am trying to use BinaryObject to support data of different class definitions in one cache. This is for my system upgrade. I first start a cache with data, and then launch a new node to join the cluster and put new data into the existing cache. The new data has a different class definition. I use the same class name, but add a member to the class. I can add the objects of this new class to the cache. The cache size changes and I can get both the new and old data using keys. However, when I use SQLFieldsQuery like “select id from myclassname” where id is a member exists in both the versions of classes, I can get only the old data. There is no error or exception. SQL just cannot find the data of the new class definition.
How can I use SQL queries to find both the new and old data? The new data is in the cache, but it seems not being in the table using my class name. Where is the new data? Is there a new table? If yes, what is the table name? I do not expect to see the new column using the old query. I just hope to see the old fields of new data using the old queries.
BTW, I use QueryEntity to set up fields and indexes in my codes. Does anyone has an example about how to add fields to existing cache dynamically? Thank you!
Re: SQL cannot find data of new class definition
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You can add fields to existing SQL-backed cache using ALTER TABLE ... ADD
COLUMN command:
https://apacheignite-sql.readme.io/docs/alter-table
The recommendation for your use case, where the layout of dat1a is expected
to change, is to just use SQL (DDL) defined tables and forget about
BinaryObject's.
With regards to your original case, i.e., a different class definition: I
could spend time debugging it if you had more info, but this approach is
not recommended anyway.
Regards,
--
Ilya Kasnacheev
2018-06-15 16:29 GMT+03:00 Cong Guo <co...@huawei.com>:
> Hi all,
>
>
>
> I am trying to use BinaryObject to support data of different class
> definitions in one cache. This is for my system upgrade. I first start a
> cache with data, and then launch a new node to join the cluster and put new
> data into the existing cache. The new data has a different class
> definition. I use the same class name, but add a member to the class. I can
> add the objects of this new class to the cache. The cache size changes and
> I can get both the new and old data using keys. However, when I use
> SQLFieldsQuery like “select id from myclassname” where id is a member
> exists in both the versions of classes, I can get only the old data. There
> is no error or exception. SQL just cannot find the data of the new class
> definition.
>
>
>
> How can I use SQL queries to find both the new and old data? The new data
> is in the cache, but it seems not being in the table using my class name.
> Where is the new data? Is there a new table? If yes, what is the table
> name? I do not expect to see the new column using the old query. I just
> hope to see the old fields of new data using the old queries.
>
>
>
> BTW, I use QueryEntity to set up fields and indexes in my codes. Does
> anyone has an example about how to add fields to existing cache
> dynamically? Thank you!
>