You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Deepa Kolwalkar <de...@tcs.com> on 2018/08/05 14:32:36 UTC
what are the tehcniques to automatically detect changes in Multiple DBs and
automatically push them into Ignite Cache
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in Multiple DBs
and automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
Thanks Juan .. I will check on the same
From: "Juan Rodríguez Hortalá" <ju...@gmail.com>
To: user@ignite.apache.org
Date: 07-08-2018 08:44
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
That looks like something you could do with Kafka connect
https://www.confluent.io/product/connectors/ using the jdbc source and the
ignite sink. Just my 2 cents
On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar <de...@tcs.com>
wrote:
Thanks Prasad for your suggestions.
The Legacy sytems are on different platforms and some of them are products
.. so there is no way of implementing any custom logic in such products to
send update messages. The Legacy Systems remain a black-box to us with
only its DB which is accessible for viewing..
Regards
From: "Prasad Bhalerao" <pr...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 16:51
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
Can this back office legacy system send you a DB update message or can you
make this back office system to send you DB update message?
If yes then you can have the Id/primary key, DB operation and table name
in this DB update message.
In your application you use this information to refresh your cache using
read through mechanism.
Thanks,
Prasad
On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <de...@tcs.com>
wrote:
Thanks Denis.
But as I mentioned in earlier mail, the Caches are meant to be Read-Only
(only to be used by Microservices for fetching data).
The Databases are updated by backoffice legacy systems. Hence we cannot
do a Write-through to the DBs via the CacheStore API.
If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
to hear about any challenges/short-comings if any.
Regards
From: Denis Mekhanikov <dm...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 13:18
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then
you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data
into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in
Multiple DBs and automatically push them into Ignite Cache
Posted by Juan Rodríguez Hortalá <ju...@gmail.com>.
That looks like something you could do with Kafka connect
https://www.confluent.io/product/connectors/ using the jdbc source and the
ignite sink. Just my 2 cents
On Mon, Aug 6, 2018, 07:19 Deepa Kolwalkar <de...@tcs.com> wrote:
> Thanks Prasad for your suggestions.
>
> The Legacy sytems are on different platforms and some of them are products
> .. so there is no way of implementing any custom logic in such products to
> send update messages. The Legacy Systems remain a black-box to us with only
> its DB which is accessible for viewing..
>
> Regards
>
>
>
>
>
> From: "Prasad Bhalerao" <pr...@gmail.com>
> To: user@ignite.apache.org
> Date: 06-08-2018 16:51
> Subject: Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> ------------------------------
>
>
>
> Can this back office legacy system send you a DB update message or can you
> make this back office system to send you DB update message?
>
> If yes then you can have the Id/primary key, DB operation and table name
> in this DB update message.
>
> In your application you use this information to refresh your cache using
> read through mechanism.
>
> Thanks,
> Prasad
>
> On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <*deepa.kolwalkar@tcs.com*
> <de...@tcs.com>> wrote:
> Thanks Denis.
>
> But as I mentioned in earlier mail, the Caches are meant to be Read-Only
> (only to be used by Microservices for fetching data).
> The Databases are updated by backoffice legacy systems. Hence we cannot
> do a Write-through to the DBs via the CacheStore API.
>
> If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
> to hear about any challenges/short-comings if any.
>
> Regards
>
>
>
> From: Denis Mekhanikov <*dmekhanikov@gmail.com*
> <dm...@gmail.com>>
> To: *user@ignite.apache.org* <us...@ignite.apache.org>
> Date: 06-08-2018 13:18
> Subject: Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> ------------------------------
>
>
>
> There is no such feature in Ignite.
> If you know, how to subscribe for events in the external database, then
> you can implement this logic yourself.
> You just need to perform put into Ignite cache for every insert into the
> external DB.
>
> But the recommended way to do it is to perform writing on Ignite.
> Cache store with *write-through*
> <https://apacheignite.readme.io/v2.6/docs/3rd-party-store#section-read-through-and-write-through> enabled
> will take care of writing the data into the external DB.
>
> Denis
>
> вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <*deepa.kolwalkar@tcs.com*
> <de...@tcs.com>>:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> *http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html*
> <http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html>
> which suggests the following :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> <https://docs.gridgain.com/docs/goldengate-replication>
> ==============
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
Re: what are the tehcniques to automatically detect changes in Multiple DBs
and automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
Thanks Prasad for your suggestions.
The Legacy sytems are on different platforms and some of them are products
.. so there is no way of implementing any custom logic in such products to
send update messages. The Legacy Systems remain a black-box to us with
only its DB which is accessible for viewing..
Regards
From: "Prasad Bhalerao" <pr...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 16:51
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
Can this back office legacy system send you a DB update message or can you
make this back office system to send you DB update message?
If yes then you can have the Id/primary key, DB operation and table name
in this DB update message.
In your application you use this information to refresh your cache using
read through mechanism.
Thanks,
Prasad
On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <de...@tcs.com>
wrote:
Thanks Denis.
But as I mentioned in earlier mail, the Caches are meant to be Read-Only
(only to be used by Microservices for fetching data).
The Databases are updated by backoffice legacy systems. Hence we cannot
do a Write-through to the DBs via the CacheStore API.
If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
to hear about any challenges/short-comings if any.
Regards
From: Denis Mekhanikov <dm...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 13:18
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then
you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data
into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in
Multiple DBs and automatically push them into Ignite Cache
Posted by Prasad Bhalerao <pr...@gmail.com>.
Can this back office legacy system send you a DB update message or can you
make this back office system to send you DB update message?
If yes then you can have the Id/primary key, DB operation and table name in
this DB update message.
In your application you use this information to refresh your cache using
read through mechanism.
Thanks,
Prasad
On Mon, Aug 6, 2018, 3:02 PM Deepa Kolwalkar <de...@tcs.com>
wrote:
> Thanks Denis.
>
> But as I mentioned in earlier mail, the Caches are meant to be Read-Only
> (only to be used by Microservices for fetching data).
> The Databases are updated by backoffice legacy systems. Hence we cannot
> do a Write-through to the DBs via the CacheStore API.
>
> If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
> to hear about any challenges/short-comings if any.
>
> Regards
>
>
>
> From: Denis Mekhanikov <dm...@gmail.com>
> To: user@ignite.apache.org
> Date: 06-08-2018 13:18
> Subject: Re: what are the tehcniques to automatically detect
> changes in Multiple DBs and automatically push them into Ignite Cache
> ------------------------------
>
>
>
> There is no such feature in Ignite.
> If you know, how to subscribe for events in the external database, then
> you can implement this logic yourself.
> You just need to perform put into Ignite cache for every insert into the
> external DB.
>
> But the recommended way to do it is to perform writing on Ignite.
> Cache store with *write-through*
> <https://apacheignite.readme.io/v2.6/docs/3rd-party-store#section-read-through-and-write-through> enabled
> will take care of writing the data into the external DB.
>
> Denis
>
> вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <*deepa.kolwalkar@tcs.com*
> <de...@tcs.com>>:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> *http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html*
> <http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html>
> which suggests the following :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> <https://docs.gridgain.com/docs/goldengate-replication>
> ==============
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
Re: what are the tehcniques to automatically detect changes in Multiple DBs
and automatically push them into Ignite Cache
Posted by Deepa Kolwalkar <de...@tcs.com>.
Thanks Denis.
But as I mentioned in earlier mail, the Caches are meant to be Read-Only
(only to be used by Microservices for fetching data).
The Databases are updated by backoffice legacy systems. Hence we cannot
do a Write-through to the DBs via the CacheStore API.
If anyone has used the Gridgain GoldenGate Adapter, then we would be glad
to hear about any challenges/short-comings if any.
Regards
From: Denis Mekhanikov <dm...@gmail.com>
To: user@ignite.apache.org
Date: 06-08-2018 13:18
Subject: Re: what are the tehcniques to automatically detect
changes in Multiple DBs and automatically push them into Ignite Cache
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then
you can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through enabled will take care of writing the data
into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
We have a requirement where Changes to data from Multiple DBs need to be
periodically & automatically Pushed (not sure how) into various Ignite
Caches
Once the Data is available in the Ignite Caches, it will be persisted
using Ignite Native Persistence, so that in the event of a crash, the Data
can be loaded from Native Persistence.
The Caches will be used in read-only manner by Clients (Microservices) .
What is the Best technique for having Changes to data from Multiple DBs to
be automatically put into the Ignite Caches ?
While searching for this solution i came across this link :
http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
which suggests the following :
==============
Ignite does not provide such integration out of the box, however there
a commercial offering from GridGain for that:
https://docs.gridgain.com/docs/goldengate-replication
==============
Was wondering whether we still need to use GoldenGate for such
replications OR whether newer versions of Ignite are now supporting such
asynchronous sync-ups with underlying DB changes
Thanks
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
Re: what are the tehcniques to automatically detect changes in
Multiple DBs and automatically push them into Ignite Cache
Posted by Denis Mekhanikov <dm...@gmail.com>.
There is no such feature in Ignite.
If you know, how to subscribe for events in the external database, then you
can implement this logic yourself.
You just need to perform put into Ignite cache for every insert into the
external DB.
But the recommended way to do it is to perform writing on Ignite.
Cache store with write-through
<https://apacheignite.readme.io/v2.6/docs/3rd-party-store#section-read-through-and-write-through>
enabled
will take care of writing the data into the external DB.
Denis
вс, 5 авг. 2018 г. в 17:32, Deepa Kolwalkar <de...@tcs.com>:
> We have a requirement where Changes to data from Multiple DBs need to be
> periodically & automatically Pushed (not sure how) into various Ignite
> Caches
> Once the Data is available in the Ignite Caches, it will be persisted
> using Ignite Native Persistence, so that in the event of a crash, the Data
> can be loaded from Native Persistence.
> The Caches will be used in read-only manner by Clients (Microservices) .
>
> What is the Best technique for having Changes to data from Multiple DBs to
> be automatically put into the Ignite Caches ?
>
> While searching for this solution i came across this link :
>
> http://apache-ignite-users.70518.x6.nabble.com/Any-references-Syncing-Ignite-and-Oracle-DB-with-Oracle-GoldenGate-updates-from-DB-to-ignite-td20715.html
> which suggests the following :
> ==============
> Ignite does not provide such integration out of the box, however there
> a commercial offering from GridGain for that:
> *https://docs.gridgain.com/docs/goldengate-replication*
> <https://docs.gridgain.com/docs/goldengate-replication>
> ==============
>
> Was wondering whether we still need to use GoldenGate for such
> replications OR whether newer versions of Ignite are now supporting such
> asynchronous sync-ups with underlying DB changes
>
> Thanks
>
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>