You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "satyajit.mandal.barclays.com via user" <us...@ignite.apache.org> on 2024/06/12 06:52:07 UTC
Ignite .NET + CDC replication using Kafka
Hi Pavel,
How can we achieve replication from Ignite .NET cluster to another Ignite .NET cluster as backup using Kafka as middleware? This is needed for resiliency test for Ignite cluster where we shutdown one cluster and move all application service to backup cluster as part of resiliency testing.
Basically need to know which Ignite features supports to capture changes in caches so that we can write programs to capture those changes and stream those to Kafka using .NET libraries.
Found this doc but not sure how Ignite .NET fits into this ( CDC replication using Kafka)
https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions
Can you share some working examples to capture the changes happening to cache realtime using .NET?
Thanks
Satyajit
Barclays Execution Services Limited registered in England. Registered No. 1767980. Registered office: 1 Churchill Place, London, E14 5HP
Barclays Execution Services Limited provides support and administrative services across Barclays group. Barclays Execution Services Limited is an appointed representative of Barclays Bank UK plc and Barclays Bank plc. Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.
This email and any attachments are confidential and intended solely for the addressee and may also be privileged or exempt from disclosure under applicable law. If you are not the addressee, or have received this email in error, please notify the sender and immediately delete it and any attachments from your system. Do not copy, use, disclose or otherwise act on any part of this email or its attachments.
Internet communications are not guaranteed to be secure or virus-free. The Barclays group does not accept responsibility for any loss arising from unauthorised access to, or interference with, any internet communications by any third party, or from the transmission of any viruses. Replies to this email may be monitored by the Barclays group for operational or business reasons.
Any opinion or other information in this email or its attachments that does not relate to the business of the Barclays group is personal to the sender and is not given or endorsed by the Barclays group.
Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays.
Re: Ignite .NET + CDC replication using Kafka
Posted by Nikolay Izhikov <ni...@apache.org>.
CDC not related to Ignite.NET <http://ignite.net/>.
It captures changes that happens on storage (WAL) layer of Ignite.
So, all you need is to follow the examples from documentation and configure CDC replication.
> 12 июня 2024 г., в 14:02, Pavel Tupitsyn <pt...@apache.org> написал(а):
>
> Can you share some working examples to capture the changes happening to cache realtime using .NET?
>
Re: Ignite .NET + CDC replication using Kafka
Posted by Pavel Tupitsyn <pt...@apache.org>.
> Is this feature available in Ignite .NET ? Can you share the
sample example?
Yes, Continuous Query is available in Ignite.NET, sample code:
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs
RE: Ignite .NET + CDC replication using Kafka
Posted by "satyajit.mandal.barclays.com via user" <us...@ignite.apache.org>.
Thanks Pavel.
Is this feature available in Ignite .NET ? Can you share the sample example?
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Affinity.Rendezvous.ClusterNodeAttributeAffinityBackupFilter.html
Regards
Satyajit
From: Pavel Tupitsyn <pt...@apache.org>
Sent: Wednesday, June 12, 2024 4:39 PM
To: Mandal, Satyajit: IT (PUN) <sa...@barclays.com>
Subject: Re: Ignite .NET + CDC replication using Kafka
CAUTION: This email originated from outside our organisation - ptupitsyn@apache.org<ma...@apache.org> Do not click on links, open attachments, or respond unless you recognize the sender and can validate the content is safe.
Some vendors also provide cluster replication with native .NET APIs based on Ignite:
https://www.gridgain.com/docs/latest/administrators-guide/data-center-replication/configuring-replication<https://clicktime.symantec.com/15t5z5LCoSPzxZnd8HV1f?h=idSuwKpOy7WVXyTrMyN6I_jp2pG9Xj2Vk1v_yWPSoTM=&u=https://www.gridgain.com/docs/latest/administrators-guide/data-center-replication/configuring-replication>
On Wed, Jun 12, 2024 at 2:02 PM Pavel Tupitsyn <pt...@apache.org>> wrote:
Hi Satyajit,
CDC extension is not available directly in Ignite.NET, you can use a Compute job [1] or a Service [2] implemented in Java to access this functionality.
Alternatively, Continuous Query [3] provides a way to capture all changes in a specific cache. Example: [4]
[1] https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html#Apache_Ignite_Core_Compute_ICompute_ExecuteJavaTask__1_System_String_System_Object_<https://clicktime.symantec.com/15t5pQwdtD2p8g8n3AgiR?h=0OmzUAZhJ4oIYVK3YnnrK4Utc8FX4mfLzbzlFY_zyVw=&u=https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html%23Apache_Ignite_Core_Compute_ICompute_ExecuteJavaTask__1_System_String_System_Object_>
[2] https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Services.IServices.html<https://clicktime.symantec.com/15t5uF8vLpiQYcxhaj5s3?h=k1Ix2oO44D6JEfJZZylfedVJMd23dk5v3xOQ1LFGRJM=&u=https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Services.IServices.html>
[3] https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Query.Continuous.ContinuousQuery-2.html<https://clicktime.symantec.com/15t5jakMRbMDijJrVcHZo?h=rnbT0tB2FA_fBxze82QMiPgYEpJAV9mFZFsm0ZGDCIY=&u=https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Query.Continuous.ContinuousQuery-2.html>
[4] https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs<https://clicktime.symantec.com/15t5ZvMnWMz2tqf1QVVGZ?h=u43Fem4zddoJccj5n3cleHCQMjH1ZEAC0uTZQihPSl0=&u=https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs>
On Wed, Jun 12, 2024 at 9:52 AM <sa...@barclays.com>> wrote:
Hi Pavel,
How can we achieve replication from Ignite .NET cluster to another Ignite .NET cluster as backup using Kafka as middleware? This is needed for resiliency test for Ignite cluster where we shutdown one cluster and move all application service to backup cluster as part of resiliency testing.
Basically need to know which Ignite features supports to capture changes in caches so that we can write programs to capture those changes and stream those to Kafka using .NET libraries.
Found this doc but not sure how Ignite .NET fits into this ( CDC replication using Kafka)
https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions<https://clicktime.symantec.com/15t5ekZ4xyfdJnUvx3tRB?h=uLyfAFvoWOjEF9JvWuSOSIGaUYLH7IKGUW28dcFAg2M=&u=https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions>
Can you share some working examples to capture the changes happening to cache realtime using .NET?
Thanks
Satyajit
Barclays Execution Services Limited registered in England. Registered No. 1767980. Registered office: 1 Churchill Place, London, E14 5HP
Barclays Execution Services Limited provides support and administrative services across Barclays group. Barclays Execution Services Limited is an appointed representative of Barclays Bank UK plc and Barclays Bank plc. Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.
This email and any attachments are confidential and intended solely for the addressee and may also be privileged or exempt from disclosure under applicable law. If you are not the addressee, or have received this email in error, please notify the sender and immediately delete it and any attachments from your system. Do not copy, use, disclose or otherwise act on any part of this email or its attachments.
Internet communications are not guaranteed to be secure or virus-free. The Barclays group does not accept responsibility for any loss arising from unauthorised access to, or interference with, any internet communications by any third party, or from the transmission of any viruses. Replies to this email may be monitored by the Barclays group for operational or business reasons.
Any opinion or other information in this email or its attachments that does not relate to the business of the Barclays group is personal to the sender and is not given or endorsed by the Barclays group.
Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays.
Barclays Execution Services Limited registered in England. Registered No. 1767980. Registered office: 1 Churchill Place, London, E14 5HP
Barclays Execution Services Limited provides support and administrative services across Barclays group. Barclays Execution Services Limited is an appointed representative of Barclays Bank UK plc and Barclays Bank plc. Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.
This email and any attachments are confidential and intended solely for the addressee and may also be privileged or exempt from disclosure under applicable law. If you are not the addressee, or have received this email in error, please notify the sender and immediately delete it and any attachments from your system. Do not copy, use, disclose or otherwise act on any part of this email or its attachments.
Internet communications are not guaranteed to be secure or virus-free. The Barclays group does not accept responsibility for any loss arising from unauthorised access to, or interference with, any internet communications by any third party, or from the transmission of any viruses. Replies to this email may be monitored by the Barclays group for operational or business reasons.
Any opinion or other information in this email or its attachments that does not relate to the business of the Barclays group is personal to the sender and is not given or endorsed by the Barclays group.
Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays.
Re: Ignite .NET + CDC replication using Kafka
Posted by Pavel Tupitsyn <pt...@apache.org>.
Hi Satyajit,
CDC extension is not available directly in Ignite.NET, you can use a
Compute job [1] or a Service [2] implemented in Java to access this
functionality.
Alternatively, Continuous Query [3] provides a way to capture all changes
in a specific cache. Example: [4]
[1]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Compute.ICompute.html#Apache_Ignite_Core_Compute_ICompute_ExecuteJavaTask__1_System_String_System_Object_
[2]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Services.IServices.html
[3]
https://ignite.apache.org/releases/latest/dotnetdoc/api/Apache.Ignite.Core.Cache.Query.Continuous.ContinuousQuery-2.html
[4]
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Thick/Cache/QueryContinuous/Program.cs
On Wed, Jun 12, 2024 at 9:52 AM <sa...@barclays.com> wrote:
> Hi Pavel,
>
>
>
> How can we achieve replication from Ignite .NET cluster to
> another Ignite .NET cluster as backup using Kafka as middleware?
> This is needed for resiliency test for Ignite cluster where we
> shutdown one cluster and move all application service to backup
> cluster as part of resiliency testing.
>
>
>
> Basically need to know which Ignite features supports to capture
> changes in caches so that we can write programs to capture those
> changes and stream those to Kafka using .NET libraries.
>
>
>
> Found this doc but not sure how Ignite .NET fits into this ( CDC
> replication using Kafka)
>
>
> https://ignite.apache.org/docs/latest/extensions-and-integrations/change-data-capture-extensions
>
>
>
> Can you share some working examples to capture the changes
> happening to cache realtime using .NET?
>
>
>
> Thanks
>
> Satyajit
>
>
>
>
>
> Barclays Execution Services Limited registered in England. Registered No.
> 1767980. Registered office: 1 Churchill Place, London, E14 5HP
>
> Barclays Execution Services Limited provides support and administrative
> services across Barclays group. Barclays Execution Services Limited is an
> appointed representative of Barclays Bank UK plc and Barclays Bank plc.
> Barclays Bank UK plc and Barclays Bank plc are authorised by the Prudential
> Regulation Authority and regulated by the Financial Conduct Authority and
> the Prudential Regulation Authority.
>
> This email and any attachments are confidential and intended solely for
> the addressee and may also be privileged or exempt from disclosure under
> applicable law. If you are not the addressee, or have received this email
> in error, please notify the sender and immediately delete it and any
> attachments from your system. Do not copy, use, disclose or otherwise act
> on any part of this email or its attachments.
>
> Internet communications are not guaranteed to be secure or virus-free. The
> Barclays group does not accept responsibility for any loss arising from
> unauthorised access to, or interference with, any internet communications
> by any third party, or from the transmission of any viruses. Replies to
> this email may be monitored by the Barclays group for operational or
> business reasons.
>
> Any opinion or other information in this email or its attachments that
> does not relate to the business of the Barclays group is personal to the
> sender and is not given or endorsed by the Barclays group.
>
> Unless specifically indicated, this e-mail is not an offer to buy or sell
> or a solicitation to buy or sell any securities, investment products or
> other financial product or service, an official confirmation of any
> transaction, or an official statement of Barclays.
>