You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Devakumar J <ma...@gmail.com> on 2020/07/17 15:34:25 UTC
Load balanced CQ listener/Catch up missed events
Hi,
We have a setup of 3 server nodes and 2 client nodes and 50 caches around.
When we register CQ listener for cache operations from client node, both the
client nodes are getting notified.
1. Is there way to achieve load balanced manner? For e.g) In case of 2
events, event-1 to be received by
client-1 and event-2 to be received by client-2 and in-case of one
client node down and other node should
start receiving all the events?
2. In case of both node failure, is there a way to get only those missed
events?
Thanks,
Devakumar J
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Load balanced CQ listener/Catch up missed events
Posted by Denis Magda <dm...@apache.org>.
The client can create a queue, and the servers will keep items of that
queue. Those items will be stored in memory and disk tiers of the servers.
And, yes, physically the records will be persisted in the database folder
(that is under the work folder by default). Hope this helps.
-
Denis
On Fri, Jul 24, 2020 at 9:36 AM Devakumar J <ma...@gmail.com>
wrote:
> Hi Denis,
>
> Thanks for the reply.
>
> I mean to ask, is it possible to create distributed queue at client node
> and
> if so where it will be persisted when native persistence enabled.
>
> Should it be under server folder's work folder?
>
> Thanks,
> Devakumar J
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Load balanced CQ listener/Catch up missed events
Posted by Devakumar J <ma...@gmail.com>.
Hi Denis,
Thanks for the reply.
I mean to ask, is it possible to create distributed queue at client node and
if so where it will be persisted when native persistence enabled.
Should it be under server folder's work folder?
Thanks,
Devakumar J
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Load balanced CQ listener/Catch up missed events
Posted by Denis Magda <dm...@apache.org>.
1. It depends on your application needs. Most likely a single distributed
queue will work if it needs to be accessed by any remote producer or
consumer.
2. Yes, refer to the documentation from my previous email.
3. What do you mean under "storing a part of the client node"?
-
Denis
On Wed, Jul 22, 2020 at 9:28 AM Devakumar J <ma...@gmail.com>
wrote:
> Thanks Denis. I will explore further on this.
>
> Some more questions in distributed queue.
>
> 1. When i create a queue, should it be created at each server node
> separately or it should be created once cluster wide?
>
> 2) If cluster wide, can multiple server nodes publish to the same queue?
>
> 3) Also is there a way to create the queue at client side and store a part
> of client node itself?
>
> Thanks,
> Devakumar J
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Load balanced CQ listener/Catch up missed events
Posted by Devakumar J <ma...@gmail.com>.
Thanks Denis. I will explore further on this.
Some more questions in distributed queue.
1. When i create a queue, should it be created at each server node
separately or it should be created once cluster wide?
2) If cluster wide, can multiple server nodes publish to the same queue?
3) Also is there a way to create the queue at client side and store a part
of client node itself?
Thanks,
Devakumar J
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Load balanced CQ listener/Catch up missed events
Posted by Devakumar J <ma...@gmail.com>.
Any thoughts on this pls?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Load balanced CQ listener/Catch up missed events
Posted by Devakumar J <ma...@gmail.com>.
Thanks Denis. I will explore further on this.
Some more questions in distributed queue.
1. When i create a queue, should it be created at each server node
separately or it should be created once cluster wide?
2) If cluster wide, can multiple server nodes publish to the same queue?
3) Also is there a way to create the queue at client side and store a part
of client node itself?
Thanks,
Devakumar J
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Load balanced CQ listener/Catch up missed events
Posted by Denis Magda <dm...@apache.org>.
The queue can be a good alternative [1] but the application has to care
about faults handling. For example, if a client reads a message from the
queue, starts doing some job as the message prescribes and then fails in
the middle, then the job is incomplete and nobody else can pick it up. So,
you would need to track somewhere else that the message is in fact was
fully processed (just toggle the status).
[1]
https://apacheignite.readme.io/docs/queue-and-set#cache-queues-and-load-balancing
-
Denis
On Sat, Jul 18, 2020 at 12:42 AM Devakumar J <ma...@gmail.com>
wrote:
> Hi Denis,
>
> Thanks for your reply.
>
> I will explore the custom implementation of remote filter for load
> balancing.
>
> Also currently we dont use kafka in our tech stack, so we dont want to
> introduce it just for this purpose.
>
> I was also checking ignite events and distributed message queue as an
> alternative for Continuous Queries.
>
> Can we define setup like,
> 1. Define distributed message queue across the cluster.
> 2. Each server node publish the selective cache events to the queue.
> 3. Client nodes consume the queue.
>
> So in this case as per my understanding on distributed queue,
> a. Client nodes will get events in load balanced way.
> b. If any one client node down then other node will start consuming
> all the events.
> c. If both client nodes down, queue will still have the entries that
> are not yet consumed as it is persisted. So if client nodes come up, it
> will
> resume from where it left.
>
> Is this feasible or do you see any issue in setup in case of server node
> failure/ server node re-balancing.
>
> Thanks,
> Devakumar J
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Load balanced CQ listener/Catch up missed events
Posted by Devakumar J <ma...@gmail.com>.
Hi Denis,
Thanks for your reply.
I will explore the custom implementation of remote filter for load
balancing.
Also currently we dont use kafka in our tech stack, so we dont want to
introduce it just for this purpose.
I was also checking ignite events and distributed message queue as an
alternative for Continuous Queries.
Can we define setup like,
1. Define distributed message queue across the cluster.
2. Each server node publish the selective cache events to the queue.
3. Client nodes consume the queue.
So in this case as per my understanding on distributed queue,
a. Client nodes will get events in load balanced way.
b. If any one client node down then other node will start consuming
all the events.
c. If both client nodes down, queue will still have the entries that
are not yet consumed as it is persisted. So if client nodes come up, it will
resume from where it left.
Is this feasible or do you see any issue in setup in case of server node
failure/ server node re-balancing.
Thanks,
Devakumar J
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Load balanced CQ listener/Catch up missed events
Posted by Denis Magda <dm...@apache.org>.
Hi Devakumar,
I think it's feasible to implement such logic with custom remote filters:
https://apacheignite.readme.io/docs/continuous-queries#remote-filter
Also, an alternate option might be a partitioned message bus like Kafka.
Servers can send changes to a Kafka topic and the clients can consume the
messages on the other side in a load-balanced and fault-tolerant fashion.
-
Denis
On Fri, Jul 17, 2020 at 8:34 AM Devakumar J <ma...@gmail.com>
wrote:
> Hi,
>
> We have a setup of 3 server nodes and 2 client nodes and 50 caches around.
>
> When we register CQ listener for cache operations from client node, both
> the
> client nodes are getting notified.
>
> 1. Is there way to achieve load balanced manner? For e.g) In case of 2
> events, event-1 to be received by
> client-1 and event-2 to be received by client-2 and in-case of one
> client node down and other node should
> start receiving all the events?
>
> 2. In case of both node failure, is there a way to get only those missed
> events?
>
> Thanks,
> Devakumar J
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>