You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Andrey Kornev <an...@hotmail.com> on 2016/01/08 18:02:07 UTC
IgniteBinary, POJOs and indexing
Hello, Ignite Community,
IgniteBinary javadocs mention the ability to dynamically change the structure of the "classes" without restarting the cluster. The javadocs even say that the new fields become automatically available to the SQL queries with no extra effort. It's all fine and dandy, but how can I tell Ignite that some of the new fields should also be indexed?
Using the example from the same javadocs, my binary object initially has two field A and B. I configure the cache entry metadata to index the field A and then create the cache. When later I change the structure by adding a new field C, how can I tell Ignite that I now want only fields B and C to be included in the index?
Related to above, is there any way to modify the indexes after the cache has been started? In this case I do not modify the structure of the class, but rather change which fields get indexed, the sorting properties, etc.
Finally, how about introducing new POJO classes at runtime (yes, I can do it -- I run in OSGi environment)? For example, at the cache creation time the cache metadata only had the POJOA class annotated to index its field "foo", and then later the user introduces a new POJOB class annotated to index its field "bar". Would POJOB start getting indexed automatically, or the user will be given the finger?
Any input will be very much appreciated!
Andrey
Re: IgniteBinary, POJOs and indexing
Posted by christos <ch...@gridgain.com>.
I agree that dynamic indexing at runtime is certainly useful.
I don't think persistent storage or supporting removing and updating of
indexes added at runtime is necessary - i think its more frequent use case
to be able to add than remove or update.
--
View this message in context: http://apache-ignite-developers.2346864.n4.nabble.com/IgniteBinary-POJOs-and-indexing-tp6447p6759.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
Re: IgniteBinary, POJOs and indexing
Posted by Sergi Vladykin <se...@gmail.com>.
As for dynamic caches this is usually ok, because application can use
method getOrCreateCache(). For dynamic indexes probably you don't want to
write all the initialization logic each time client tries to access cache.
Sergi
2016-01-11 14:07 GMT+03:00 Sergi Vladykin <se...@gmail.com>:
> I mean it would be quite weird if dynamically created index silently
> disappears at some point. This may be obvious for you but not for everyone.
> We can start with version which does not persist dynamic configuration but
> looking forward it must be designed with persistence in mind. Probably we
> can utilize some system cache and achieve persistence using store.
>
> Sergi
>
> 2016-01-10 19:51 GMT+03:00 Andrey Kornev <an...@hotmail.com>:
>
>> Sergi,
>>
>> Thanks for your reply.
>>
>> I didn't quite get the point about the "persistent configuration
>> storage". Could you please elaborate?
>>
>> From my point of view, dynamic index management requires something
>> similar to the SQL DDL, like create/drop index, alter index, etc. So in
>> addition to the indexing metadata in the cache config, Ignite could provide
>> an index management API: create, drop, alter, describe. It would be up to
>> the user to ensure that the indexes get recreated after the cluster is
>> restarted.
>>
>> Such approach is no different from the way Ignite currently handles
>> dynamically created caches -- there is no "persistent configuration
>> storage" to store the dynamic cache configs, and Ignite doesn't even try to
>> recreate them after a full cluster restart -- it's the use who does that,
>> either in the code or thru the configuration files.
>>
>> Thanks
>> Andrey
>>
>> > From: sergi.vladykin@gmail.com
>> > Date: Sat, 9 Jan 2016 13:10:13 +0300
>> > Subject: Re: IgniteBinary, POJOs and indexing
>> > To: dev@ignite.apache.org
>> >
>> > I don't think we can easily implement this feature. Because it means
>> that
>> > we need some kind of "persistent configuration storage", which will be
>> > consistent in all the scenarios of failures/restarts. So we have to
>> design
>> > this thing first keeping in mind that in the future we will store there
>> not
>> > only "dynamic indexes" but some other dynamic configuration as well.
>> >
>> > I don't think I will be able to work on it myself but since it is going
>> to
>> > be a general mechanism mostly unrelated to SQL we can ask someone else
>> to
>> > pick it up.
>> >
>> > Sergi
>> >
>> >
>> > 2016-01-08 21:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>> >
>> > > Hi Andrey,
>> > >
>> > > The answer right now is simple. Yes, you can create classes and add
>> fields
>> > > at runtime. No, you cannot change indexes that you have created in
>> cache
>> > > configuration.
>> > >
>> > > However, you are raising a very valid point. We should be able to
>> create
>> > > and update indexes dynamically. There is a ticket, IGNITE-735 [1],
>> that
>> > > has been sitting around for a while. I think we should reprioritize
>> it.
>> > >
>> > > Sergi, can you please comment on how hard this would be to implement?
>> > >
>> > > [1] https://issues.apache.org/jira/browse/IGNITE-735
>> > >
>> > > On Fri, Jan 8, 2016 at 9:02 AM, Andrey Kornev <
>> andrewkornev@hotmail.com>
>> > > wrote:
>> > >
>> > > > Hello, Ignite Community,
>> > > >
>> > > > IgniteBinary javadocs mention the ability to dynamically change the
>> > > > structure of the "classes" without restarting the cluster. The
>> javadocs
>> > > > even say that the new fields become automatically available to the
>> SQL
>> > > > queries with no extra effort. It's all fine and dandy, but how can
>> I tell
>> > > > Ignite that some of the new fields should also be indexed?
>> > > >
>> > > > Using the example from the same javadocs, my binary object
>> initially has
>> > > > two field A and B. I configure the cache entry metadata to index the
>> > > field
>> > > > A and then create the cache. When later I change the structure by
>> adding
>> > > a
>> > > > new field C, how can I tell Ignite that I now want only fields B
>> and C to
>> > > > be included in the index?
>> > > >
>> > > > Related to above, is there any way to modify the indexes after the
>> cache
>> > > > has been started? In this case I do not modify the structure of the
>> > > class,
>> > > > but rather change which fields get indexed, the sorting properties,
>> etc.
>> > > >
>> > > > Finally, how about introducing new POJO classes at runtime (yes, I
>> can do
>> > > > it -- I run in OSGi environment)? For example, at the cache
>> creation time
>> > > > the cache metadata only had the POJOA class annotated to index its
>> field
>> > > > "foo", and then later the user introduces a new POJOB class
>> annotated to
>> > > > index its field "bar". Would POJOB start getting indexed
>> automatically,
>> > > or
>> > > > the user will be given the finger?
>> > > >
>> > > > Any input will be very much appreciated!
>> > > > Andrey
>> > > >
>> > >
>>
>>
>
>
Re: IgniteBinary, POJOs and indexing
Posted by Sergi Vladykin <se...@gmail.com>.
I mean it would be quite weird if dynamically created index silently
disappears at some point. This may be obvious for you but not for everyone.
We can start with version which does not persist dynamic configuration but
looking forward it must be designed with persistence in mind. Probably we
can utilize some system cache and achieve persistence using store.
Sergi
2016-01-10 19:51 GMT+03:00 Andrey Kornev <an...@hotmail.com>:
> Sergi,
>
> Thanks for your reply.
>
> I didn't quite get the point about the "persistent configuration storage".
> Could you please elaborate?
>
> From my point of view, dynamic index management requires something similar
> to the SQL DDL, like create/drop index, alter index, etc. So in addition to
> the indexing metadata in the cache config, Ignite could provide an index
> management API: create, drop, alter, describe. It would be up to the user
> to ensure that the indexes get recreated after the cluster is restarted.
>
> Such approach is no different from the way Ignite currently handles
> dynamically created caches -- there is no "persistent configuration
> storage" to store the dynamic cache configs, and Ignite doesn't even try to
> recreate them after a full cluster restart -- it's the use who does that,
> either in the code or thru the configuration files.
>
> Thanks
> Andrey
>
> > From: sergi.vladykin@gmail.com
> > Date: Sat, 9 Jan 2016 13:10:13 +0300
> > Subject: Re: IgniteBinary, POJOs and indexing
> > To: dev@ignite.apache.org
> >
> > I don't think we can easily implement this feature. Because it means that
> > we need some kind of "persistent configuration storage", which will be
> > consistent in all the scenarios of failures/restarts. So we have to
> design
> > this thing first keeping in mind that in the future we will store there
> not
> > only "dynamic indexes" but some other dynamic configuration as well.
> >
> > I don't think I will be able to work on it myself but since it is going
> to
> > be a general mechanism mostly unrelated to SQL we can ask someone else to
> > pick it up.
> >
> > Sergi
> >
> >
> > 2016-01-08 21:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> >
> > > Hi Andrey,
> > >
> > > The answer right now is simple. Yes, you can create classes and add
> fields
> > > at runtime. No, you cannot change indexes that you have created in
> cache
> > > configuration.
> > >
> > > However, you are raising a very valid point. We should be able to
> create
> > > and update indexes dynamically. There is a ticket, IGNITE-735 [1],
> that
> > > has been sitting around for a while. I think we should reprioritize it.
> > >
> > > Sergi, can you please comment on how hard this would be to implement?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-735
> > >
> > > On Fri, Jan 8, 2016 at 9:02 AM, Andrey Kornev <
> andrewkornev@hotmail.com>
> > > wrote:
> > >
> > > > Hello, Ignite Community,
> > > >
> > > > IgniteBinary javadocs mention the ability to dynamically change the
> > > > structure of the "classes" without restarting the cluster. The
> javadocs
> > > > even say that the new fields become automatically available to the
> SQL
> > > > queries with no extra effort. It's all fine and dandy, but how can I
> tell
> > > > Ignite that some of the new fields should also be indexed?
> > > >
> > > > Using the example from the same javadocs, my binary object initially
> has
> > > > two field A and B. I configure the cache entry metadata to index the
> > > field
> > > > A and then create the cache. When later I change the structure by
> adding
> > > a
> > > > new field C, how can I tell Ignite that I now want only fields B and
> C to
> > > > be included in the index?
> > > >
> > > > Related to above, is there any way to modify the indexes after the
> cache
> > > > has been started? In this case I do not modify the structure of the
> > > class,
> > > > but rather change which fields get indexed, the sorting properties,
> etc.
> > > >
> > > > Finally, how about introducing new POJO classes at runtime (yes, I
> can do
> > > > it -- I run in OSGi environment)? For example, at the cache creation
> time
> > > > the cache metadata only had the POJOA class annotated to index its
> field
> > > > "foo", and then later the user introduces a new POJOB class
> annotated to
> > > > index its field "bar". Would POJOB start getting indexed
> automatically,
> > > or
> > > > the user will be given the finger?
> > > >
> > > > Any input will be very much appreciated!
> > > > Andrey
> > > >
> > >
>
>
RE: IgniteBinary, POJOs and indexing
Posted by Andrey Kornev <an...@hotmail.com>.
Sergi,
Thanks for your reply.
I didn't quite get the point about the "persistent configuration storage". Could you please elaborate?
>From my point of view, dynamic index management requires something similar to the SQL DDL, like create/drop index, alter index, etc. So in addition to the indexing metadata in the cache config, Ignite could provide an index management API: create, drop, alter, describe. It would be up to the user to ensure that the indexes get recreated after the cluster is restarted.
Such approach is no different from the way Ignite currently handles dynamically created caches -- there is no "persistent configuration storage" to store the dynamic cache configs, and Ignite doesn't even try to recreate them after a full cluster restart -- it's the use who does that, either in the code or thru the configuration files.
Thanks
Andrey
> From: sergi.vladykin@gmail.com
> Date: Sat, 9 Jan 2016 13:10:13 +0300
> Subject: Re: IgniteBinary, POJOs and indexing
> To: dev@ignite.apache.org
>
> I don't think we can easily implement this feature. Because it means that
> we need some kind of "persistent configuration storage", which will be
> consistent in all the scenarios of failures/restarts. So we have to design
> this thing first keeping in mind that in the future we will store there not
> only "dynamic indexes" but some other dynamic configuration as well.
>
> I don't think I will be able to work on it myself but since it is going to
> be a general mechanism mostly unrelated to SQL we can ask someone else to
> pick it up.
>
> Sergi
>
>
> 2016-01-08 21:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
>
> > Hi Andrey,
> >
> > The answer right now is simple. Yes, you can create classes and add fields
> > at runtime. No, you cannot change indexes that you have created in cache
> > configuration.
> >
> > However, you are raising a very valid point. We should be able to create
> > and update indexes dynamically. There is a ticket, IGNITE-735 [1], that
> > has been sitting around for a while. I think we should reprioritize it.
> >
> > Sergi, can you please comment on how hard this would be to implement?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-735
> >
> > On Fri, Jan 8, 2016 at 9:02 AM, Andrey Kornev <an...@hotmail.com>
> > wrote:
> >
> > > Hello, Ignite Community,
> > >
> > > IgniteBinary javadocs mention the ability to dynamically change the
> > > structure of the "classes" without restarting the cluster. The javadocs
> > > even say that the new fields become automatically available to the SQL
> > > queries with no extra effort. It's all fine and dandy, but how can I tell
> > > Ignite that some of the new fields should also be indexed?
> > >
> > > Using the example from the same javadocs, my binary object initially has
> > > two field A and B. I configure the cache entry metadata to index the
> > field
> > > A and then create the cache. When later I change the structure by adding
> > a
> > > new field C, how can I tell Ignite that I now want only fields B and C to
> > > be included in the index?
> > >
> > > Related to above, is there any way to modify the indexes after the cache
> > > has been started? In this case I do not modify the structure of the
> > class,
> > > but rather change which fields get indexed, the sorting properties, etc.
> > >
> > > Finally, how about introducing new POJO classes at runtime (yes, I can do
> > > it -- I run in OSGi environment)? For example, at the cache creation time
> > > the cache metadata only had the POJOA class annotated to index its field
> > > "foo", and then later the user introduces a new POJOB class annotated to
> > > index its field "bar". Would POJOB start getting indexed automatically,
> > or
> > > the user will be given the finger?
> > >
> > > Any input will be very much appreciated!
> > > Andrey
> > >
> >
Re: IgniteBinary, POJOs and indexing
Posted by Sergi Vladykin <se...@gmail.com>.
I don't think we can easily implement this feature. Because it means that
we need some kind of "persistent configuration storage", which will be
consistent in all the scenarios of failures/restarts. So we have to design
this thing first keeping in mind that in the future we will store there not
only "dynamic indexes" but some other dynamic configuration as well.
I don't think I will be able to work on it myself but since it is going to
be a general mechanism mostly unrelated to SQL we can ask someone else to
pick it up.
Sergi
2016-01-08 21:20 GMT+03:00 Dmitriy Setrakyan <ds...@apache.org>:
> Hi Andrey,
>
> The answer right now is simple. Yes, you can create classes and add fields
> at runtime. No, you cannot change indexes that you have created in cache
> configuration.
>
> However, you are raising a very valid point. We should be able to create
> and update indexes dynamically. There is a ticket, IGNITE-735 [1], that
> has been sitting around for a while. I think we should reprioritize it.
>
> Sergi, can you please comment on how hard this would be to implement?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-735
>
> On Fri, Jan 8, 2016 at 9:02 AM, Andrey Kornev <an...@hotmail.com>
> wrote:
>
> > Hello, Ignite Community,
> >
> > IgniteBinary javadocs mention the ability to dynamically change the
> > structure of the "classes" without restarting the cluster. The javadocs
> > even say that the new fields become automatically available to the SQL
> > queries with no extra effort. It's all fine and dandy, but how can I tell
> > Ignite that some of the new fields should also be indexed?
> >
> > Using the example from the same javadocs, my binary object initially has
> > two field A and B. I configure the cache entry metadata to index the
> field
> > A and then create the cache. When later I change the structure by adding
> a
> > new field C, how can I tell Ignite that I now want only fields B and C to
> > be included in the index?
> >
> > Related to above, is there any way to modify the indexes after the cache
> > has been started? In this case I do not modify the structure of the
> class,
> > but rather change which fields get indexed, the sorting properties, etc.
> >
> > Finally, how about introducing new POJO classes at runtime (yes, I can do
> > it -- I run in OSGi environment)? For example, at the cache creation time
> > the cache metadata only had the POJOA class annotated to index its field
> > "foo", and then later the user introduces a new POJOB class annotated to
> > index its field "bar". Would POJOB start getting indexed automatically,
> or
> > the user will be given the finger?
> >
> > Any input will be very much appreciated!
> > Andrey
> >
>
Re: IgniteBinary, POJOs and indexing
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Hi Andrey,
The answer right now is simple. Yes, you can create classes and add fields
at runtime. No, you cannot change indexes that you have created in cache
configuration.
However, you are raising a very valid point. We should be able to create
and update indexes dynamically. There is a ticket, IGNITE-735 [1], that
has been sitting around for a while. I think we should reprioritize it.
Sergi, can you please comment on how hard this would be to implement?
[1] https://issues.apache.org/jira/browse/IGNITE-735
On Fri, Jan 8, 2016 at 9:02 AM, Andrey Kornev <an...@hotmail.com>
wrote:
> Hello, Ignite Community,
>
> IgniteBinary javadocs mention the ability to dynamically change the
> structure of the "classes" without restarting the cluster. The javadocs
> even say that the new fields become automatically available to the SQL
> queries with no extra effort. It's all fine and dandy, but how can I tell
> Ignite that some of the new fields should also be indexed?
>
> Using the example from the same javadocs, my binary object initially has
> two field A and B. I configure the cache entry metadata to index the field
> A and then create the cache. When later I change the structure by adding a
> new field C, how can I tell Ignite that I now want only fields B and C to
> be included in the index?
>
> Related to above, is there any way to modify the indexes after the cache
> has been started? In this case I do not modify the structure of the class,
> but rather change which fields get indexed, the sorting properties, etc.
>
> Finally, how about introducing new POJO classes at runtime (yes, I can do
> it -- I run in OSGi environment)? For example, at the cache creation time
> the cache metadata only had the POJOA class annotated to index its field
> "foo", and then later the user introduces a new POJOB class annotated to
> index its field "bar". Would POJOB start getting indexed automatically, or
> the user will be given the finger?
>
> Any input will be very much appreciated!
> Andrey
>