You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ferry Syafei Sapei <fe...@googlemail.com> on 2015/12/01 10:08:25 UTC
Re: Continuous update Data Grid Cache
Thank you for your idea. Your solution should work.
May it be possible to perform the following operation?
- Create a new cache and isolate it from being replicated with the old cache.
- Populate data on new cache
- Shutdown old cache and remove the new cache isolation.
fsapei
> Am 30.11.2015 um 12:51 schrieb Denis Magda <dm...@gridgain.com>:
>
> Fsapei,
>
> Ignite neither supports a cache renaming nor has a notion of aliases for caches.
>
> In general you can support aliases-like functionality by extending Ignite's JDBC Driver implementation.
> The flow should look like this:
> - create a new cache and populate it;
> - redirect all the queries to the new cache. To achieve this the code has to parse a SQL query string and exchange an aliase name with a real cache name;
> - delete the old cache.
>
> Will this work for you?
>
> --
> Denis
>
> On 11/30/2015 2:22 PM, Ferry Syafei Sapei wrote:
>> Hi Denis,
>>
>> Using a database, we can import new data to new tables and then rename the table.
>>
>> In ElasticSearch, there is an alias. The client executes queries against index alias. On the background, the alias will point to the actual index. In my use case, I just need to create a new index and then update the alias’ reference to the new index. The client does not realize that the index has been changed, since it only communicates with the alias.
>>
>> Are there these kind of operations in apache ignite?
>>
>>
>> fsapei
>>
>>
>>> Am 30.11.2015 um 11:57 schrieb Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>>:
>>>
>>> Hi Fsapei,
>>>
>>> I don't think that it's feasible or at least easy to implement your approach without a downtime or changing cache name in queries once the second cache is fully propagated.
>>> How would you implement your use case if instead of an Ignite cache you would be using a data base?
>>>
>>> --
>>> Denis
>>>
>>> On 11/28/2015 11:50 AM, Ferry Syafei Sapei wrote:
>>>> Thank you for the reply.
>>>>
>>>> The client uses JDBC Driver to get the data from Ignite.
>>>>
>>>> The swap between new cache and old cache should be hidden from the client.
>>>>
>>>> Would it be possible to do this in Ignite?
>>>>
>>>> My first idea is to create a new cache and fill it with new data data. Afterwards, within a transaction, the old cache content will be deleted and all the content from new cache will be inserted to the old cache.
>>>>
>>>>
>>>>> Am 28.11.2015 um 03:35 schrieb Dmitriy Setrakyan < <ma...@apache.org>dsetrakyan@apache.org <ma...@apache.org>>:
>>>>>
>>>>> Hi Fsapei,
>>>>>
>>>>> Ignite supports dynamic caches, so you can create and destroy caches on the fly:
>>>>> https://apacheignite.readme.io/docs/data-grid#section-jcache <https://apacheignite.readme.io/docs/data-grid#section-jcache>
>>>>>
>>>>> To support your use case, you can:
>>>>>
>>>>> 1. create a new cache and start populating it
>>>>> 2. while the new cache is populated, the queries should go to the old cache.
>>>>> 3. when new cache is fully populated, route the queries to the new cache
>>>>> 4. delete the old cache
>>>>>
>>>>> Will this work?
>>>>>
>>>>> D.
>>>>>
>>>>> On Sat, Nov 28, 2015 at 3:38 AM, Ferry Syafei Sapei < <ma...@googlemail.com>ferry.sapei@googlemail.com <ma...@googlemail.com>> wrote:
>>>>> Hallo,
>>>>>
>>>>> I would like to use Ignite as Data Grid. The cache will be initially loaded with a lot of data (e.g. a CSV file with a size more than 10 GB).
>>>>>
>>>>> When a new CSV file is available, the current Cache content must be replaced with the new data from the new CSV. The old cache content must be completely deleted.
>>>>>
>>>>> The switch from old cache content to the new one must happen without downtime. When any SQL Clients execute query during the switch, they should still obtain any data from the old cache content, since the switch has not finished yet.
>>>>>
>>>>> Could somebody please advise me, how to continuously update the cache like the description above?
>>>>>
>>>>> Thank you.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> fsapei
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Denis Magda
>>> Lead Professional Services Engineer, GridGain Systems
>>> http://www.gridgain.com/ <http://www.gridgain.com/>
Re: Continuous update Data Grid Cache
Posted by Denis Magda <dm...@apache.org>.
Hi,
To avoid any downtime here I would recommend to load the new data set in the same cache and remove the old entries from there once the preloading is finished. This approach assumes that both old and new entries can co-exist in the cache.
—
Denis
> On Oct 23, 2017, at 7:13 AM, blackfield <ch...@maxpoint.com> wrote:
>
>
> The use case is similar to OP.
>
> We have a large table that we import/load from a dataset.
>
> This dataset is generated periodically such that we need to re-load this
> whole dataset to Ignite.
>
> If we re-load the new dataset against the same Ignite table, the users will
> be impacted during that window.
>
> We are looking for ways to minimize the impact, e.g. load the data to
> another table->perform swap, etc.
>
> Let me know if there are existing capabilities to minimize the above
> scenario.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous update Data Grid Cache
Posted by blackfield <ch...@maxpoint.com>.
The use case is similar to OP.
We have a large table that we import/load from a dataset.
This dataset is generated periodically such that we need to re-load this
whole dataset to Ignite.
If we re-load the new dataset against the same Ignite table, the users will
be impacted during that window.
We are looking for ways to minimize the impact, e.g. load the data to
another table->perform swap, etc.
Let me know if there are existing capabilities to minimize the above
scenario.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous update Data Grid Cache
Posted by Denis Magda <dm...@apache.org>.
Why do you need cache aliases? Ignite evolved tremendously since the time of this discussion. Most likely your task is solvable with already existing capabilities.
—
Denis
> On Oct 19, 2017, at 1:48 PM, blackfield <ch...@maxpoint.com> wrote:
>
> Resurrecting this old thread.
>
> What is the current recommendation for the OP use case?
>
> Does Ignite support/plan to support aliases for cache name?
>
> I think this is really great feature to have to minimize client
> interruption/pain to switch to different cache.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous update Data Grid Cache
Posted by blackfield <ch...@maxpoint.com>.
Resurrecting this old thread.
What is the current recommendation for the OP use case?
Does Ignite support/plan to support aliases for cache name?
I think this is really great feature to have to minimize client
interruption/pain to switch to different cache.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Continuous update Data Grid Cache
Posted by Denis Magda <dm...@gridgain.com>.
Hi fsapei,
Actually you need to extend functionality that is located in
org.apache.ignite.internal.jdbc2 package or org.apache.ignite.internal.jdbc
in case you work with a legacy version of Ignite JDBC Driver.
In particular you can start looking at
org.apache.ignite.internal.jdbc2.JdbcStatement.executeQuery() method.
BTW, Igniters I've mentioned that we introduced a notion of aliases for
query fields (QueryEntity.setAliases). Aren't we planning to support aliases
for cache names? This would significantly to support the case discussed in
this topic.
--
Denis
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Continuous-update-Data-Grid-Cache-tp2075p2186.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Continuous update Data Grid Cache
Posted by Ferry Syafei Sapei <fe...@googlemail.com>.
I am trying to implement the solution from Denis to parse an SQL query string and exchange the alias name with the real cache name.
In which class should I do the real processing above? Should I do it in IgniteH2Indexing class?
Thank you very much in advance.
Best regards,
fsapei
> Am 03.12.2015 um 00:10 schrieb vkulichenko <va...@gmail.com>:
>
> Fsapei,
>
> Automatic discovery process has nothing to do with caches, it's a protocol
> allowing several nodes to form a cluster. A cluster can have multiple
> caches, which are independent and isolated from each other. So, if I
> correctly understood your proposal, it will not work. I think you should go
> with the solution suggested by Denis.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Continuous-update-Data-Grid-Cache-tp2075p2120.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Continuous update Data Grid Cache
Posted by vkulichenko <va...@gmail.com>.
Fsapei,
Automatic discovery process has nothing to do with caches, it's a protocol
allowing several nodes to form a cluster. A cluster can have multiple
caches, which are independent and isolated from each other. So, if I
correctly understood your proposal, it will not work. I think you should go
with the solution suggested by Denis.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Continuous-update-Data-Grid-Cache-tp2075p2120.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Continuous update Data Grid Cache
Posted by Ferry Syafei Sapei <fe...@googlemail.com>.
What I mean by isolation is that they do not try to automatically discover each other. They only form a cluster manually when triggered.
> Am 01.12.2015 um 13:58 schrieb Denis Magda <dm...@gridgain.com>:
>
> fsapei,
>
> What do you mean under isolation between two different caches?
> All the caches live they own life and don't affect each other.
>
> --
> Denis
>
> On 12/1/2015 12:08 PM, Ferry Syafei Sapei wrote:
>> Thank you for your idea. Your solution should work.
>>
>> May it be possible to perform the following operation?
>> - Create a new cache and isolate it from being replicated with the old cache.
>> - Populate data on new cache
>> - Shutdown old cache and remove the new cache isolation.
>>
>> fsapei
>>
>>
>>> Am 30.11.2015 um 12:51 schrieb Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>>:
>>>
>>> Fsapei,
>>>
>>> Ignite neither supports a cache renaming nor has a notion of aliases for caches.
>>>
>>> In general you can support aliases-like functionality by extending Ignite's JDBC Driver implementation.
>>> The flow should look like this:
>>> - create a new cache and populate it;
>>> - redirect all the queries to the new cache. To achieve this the code has to parse a SQL query string and exchange an aliase name with a real cache name;
>>> - delete the old cache.
>>>
>>> Will this work for you?
>>>
>>> --
>>> Denis
>>>
>>> On 11/30/2015 2:22 PM, Ferry Syafei Sapei wrote:
>>>> Hi Denis,
>>>>
>>>> Using a database, we can import new data to new tables and then rename the table.
>>>>
>>>> In ElasticSearch, there is an alias. The client executes queries against index alias. On the background, the alias will point to the actual index. In my use case, I just need to create a new index and then update the alias’ reference to the new index. The client does not realize that the index has been changed, since it only communicates with the alias.
>>>>
>>>> Are there these kind of operations in apache ignite?
>>>>
>>>>
>>>> fsapei
>>>>
>>>>
>>>>> Am 30.11.2015 um 11:57 schrieb Denis Magda <dmagda@gridgain.com <ma...@gridgain.com>>:
>>>>>
>>>>> Hi Fsapei,
>>>>>
>>>>> I don't think that it's feasible or at least easy to implement your approach without a downtime or changing cache name in queries once the second cache is fully propagated.
>>>>> How would you implement your use case if instead of an Ignite cache you would be using a data base?
>>>>>
>>>>> --
>>>>> Denis
>>>>>
>>>>> On 11/28/2015 11:50 AM, Ferry Syafei Sapei wrote:
>>>>>> Thank you for the reply.
>>>>>>
>>>>>> The client uses JDBC Driver to get the data from Ignite.
>>>>>>
>>>>>> The swap between new cache and old cache should be hidden from the client.
>>>>>>
>>>>>> Would it be possible to do this in Ignite?
>>>>>>
>>>>>> My first idea is to create a new cache and fill it with new data data. Afterwards, within a transaction, the old cache content will be deleted and all the content from new cache will be inserted to the old cache.
>>>>>>
>>>>>>
>>>>>>> Am 28.11.2015 um 03:35 schrieb Dmitriy Setrakyan < <ma...@apache.org>dsetrakyan@apache.org <ma...@apache.org>>:
>>>>>>>
>>>>>>> Hi Fsapei,
>>>>>>>
>>>>>>> Ignite supports dynamic caches, so you can create and destroy caches on the fly:
>>>>>>> <https://apacheignite.readme.io/docs/data-grid#section-jcache>https://apacheignite.readme.io/docs/data-grid#section-jcache <https://apacheignite.readme.io/docs/data-grid#section-jcache>
>>>>>>>
>>>>>>> To support your use case, you can:
>>>>>>>
>>>>>>> 1. create a new cache and start populating it
>>>>>>> 2. while the new cache is populated, the queries should go to the old cache.
>>>>>>> 3. when new cache is fully populated, route the queries to the new cache
>>>>>>> 4. delete the old cache
>>>>>>>
>>>>>>> Will this work?
>>>>>>>
>>>>>>> D.
>>>>>>>
>>>>>>> On Sat, Nov 28, 2015 at 3:38 AM, Ferry Syafei Sapei < <ma...@googlemail.com>ferry.sapei@googlemail.com <ma...@googlemail.com>> wrote:
>>>>>>> Hallo,
>>>>>>>
>>>>>>> I would like to use Ignite as Data Grid. The cache will be initially loaded with a lot of data (e.g. a CSV file with a size more than 10 GB).
>>>>>>>
>>>>>>> When a new CSV file is available, the current Cache content must be replaced with the new data from the new CSV. The old cache content must be completely deleted.
>>>>>>>
>>>>>>> The switch from old cache content to the new one must happen without downtime. When any SQL Clients execute query during the switch, they should still obtain any data from the old cache content, since the switch has not finished yet.
>>>>>>>
>>>>>>> Could somebody please advise me, how to continuously update the cache like the description above?
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>> Best regards,
>>>>>>> fsapei
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Denis Magda
>>>>> Lead Professional Services Engineer, GridGain Systems
>>>>> http://www.gridgain.com/ <http://www.gridgain.com/>
>>
Re: Continuous update Data Grid Cache
Posted by Denis Magda <dm...@gridgain.com>.
fsapei,
What do you mean under isolation between two different caches?
All the caches live they own life and don't affect each other.
--
Denis
On 12/1/2015 12:08 PM, Ferry Syafei Sapei wrote:
> Thank you for your idea. Your solution should work.
>
> May it be possible to perform the following operation?
> - Create a new cache and isolate it from being replicated with the old
> cache.
> - Populate data on new cache
> - Shutdown old cache and remove the new cache isolation.
>
> fsapei
>
>
>> Am 30.11.2015 um 12:51 schrieb Denis Magda <dmagda@gridgain.com
>> <ma...@gridgain.com>>:
>>
>> Fsapei,
>>
>> Ignite neither supports a cache renaming nor has a notion of aliases
>> for caches.
>>
>> In general you can support aliases-like functionality by extending
>> Ignite's JDBC Driver implementation.
>> The flow should look like this:
>> - create a new cache and populate it;
>> - redirect all the queries to the new cache. To achieve this the code
>> has to parse a SQL query string and exchange an aliase name with a
>> real cache name;
>> - delete the old cache.
>>
>> Will this work for you?
>>
>> --
>> Denis
>>
>> On 11/30/2015 2:22 PM, Ferry Syafei Sapei wrote:
>>> Hi Denis,
>>>
>>> Using a database, we can import new data to new tables and then
>>> rename the table.
>>>
>>> In ElasticSearch, there is an alias. The client executes queries
>>> against index alias. On the background, the alias will point to the
>>> actual index. In my use case, I just need to create a new index and
>>> then update the alias’ reference to the new index. The client does
>>> not realize that the index has been changed, since it only
>>> communicates with the alias.
>>>
>>> Are there these kind of operations in apache ignite?
>>>
>>>
>>> fsapei
>>>
>>>
>>>> Am 30.11.2015 um 11:57 schrieb Denis Magda <dmagda@gridgain.com
>>>> <ma...@gridgain.com>>:
>>>>
>>>> Hi Fsapei,
>>>>
>>>> I don't think that it's feasible or at least easy to implement your
>>>> approach without a downtime or changing cache name in queries once
>>>> the second cache is fully propagated.
>>>> How would you implement your use case if instead of an Ignite cache
>>>> you would be using a data base?
>>>>
>>>> --
>>>> Denis
>>>>
>>>> On 11/28/2015 11:50 AM, Ferry Syafei Sapei wrote:
>>>>> Thank you for the reply.
>>>>>
>>>>> The client uses JDBC Driver to get the data from Ignite.
>>>>>
>>>>> The swap between new cache and old cache should be hidden from the
>>>>> client.
>>>>>
>>>>> Would it be possible to do this in Ignite?
>>>>>
>>>>> My first idea is to create a new cache and fill it with new data
>>>>> data. Afterwards, within a transaction, the old cache content will
>>>>> be deleted and all the content from new cache will be inserted to
>>>>> the old cache.
>>>>>
>>>>>
>>>>>> Am 28.11.2015 um 03:35 schrieb Dmitriy Setrakyan
>>>>>> <ds...@apache.org>:
>>>>>>
>>>>>> Hi Fsapei,
>>>>>>
>>>>>> Ignite supports dynamic caches, so you can create and destroy
>>>>>> caches on the fly:
>>>>>> https://apacheignite.readme.io/docs/data-grid#section-jcache
>>>>>>
>>>>>> To support your use case, you can:
>>>>>>
>>>>>> 1. create a new cache and start populating it
>>>>>> 2. while the new cache is populated, the queries should go to the
>>>>>> old cache.
>>>>>> 3. when new cache is fully populated, route the queries to the
>>>>>> new cache
>>>>>> 4. delete the old cache
>>>>>>
>>>>>> Will this work?
>>>>>>
>>>>>> D.
>>>>>>
>>>>>> On Sat, Nov 28, 2015 at 3:38 AM, Ferry Syafei Sapei
>>>>>> <fe...@googlemail.com> wrote:
>>>>>>
>>>>>> Hallo,
>>>>>>
>>>>>> I would like to use Ignite as Data Grid. The cache will be
>>>>>> initially loaded with a lot of data (e.g. a CSV file with a
>>>>>> size more than 10 GB).
>>>>>>
>>>>>> When a new CSV file is available, the current Cache content
>>>>>> must be replaced with the new data from the new CSV. The old
>>>>>> cache content must be completely deleted.
>>>>>>
>>>>>> The switch from old cache content to the new one must happen
>>>>>> without downtime. When any SQL Clients execute query during
>>>>>> the switch, they should still obtain any data from the old
>>>>>> cache content, since the switch has not finished yet.
>>>>>>
>>>>>> Could somebody please advise me, how to continuously update
>>>>>> the cache like the description above?
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>> Best regards,
>>>>>> fsapei
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Denis Magda
>>>> Lead Professional Services Engineer, GridGain Systems
>>>> http://www.gridgain.com/
>>>
>