You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Apoorv Palkar <ap...@aol.com> on 2017/07/19 17:59:48 UTC

Using Redis or Zookeeper for Email Monitoring

Hey Dev,


I'm working on the email monitoring system for Airavata. Currently I'm trying to solve the problem of done emails coming before start emails and making the system scalable. Today, we have only one GFaC that handles the email monitoring system. As we are moving towards a microservices approach from the monolithic code we have, this email monitoring system also needs to adapt to these changes. Currently, in the gfac code, a concurrent hashmap is kept to keep track of start/end of emails using their respective experiment ID's. Instead of keeping the hashmap locally, we should keep it in a global state so in the future multiple GFaCs can handle the map. Supun has suggested to use Zookeeper for this as it has high avaliability and realibility. I was also thinking that since these experiment IDs are a key value pair, Redis would be a good option for such a use case. What do you guys think about each one. I understand airavata currently uses zookeeper, so development wise there seems to be an edge toward it. Would Redis be a good for such use case?




-- shoutout Marcus.

Re: Using Redis or Zookeeper for Email Monitoring

Posted by "Shenoy, Gourav Ganesh" <go...@indiana.edu>.
@Apoorv,

Do you have some architectural diagrams to proceed? If not, I would recommend sketching out some and sharing it with the group. We can then work on it to improve/enhance if needed.

Thanks and Regards,
Gourav Shenoy

From: "Pierce, Marlon" <ma...@iu.edu>
Reply-To: <de...@airavata.apache.org>
Date: Thursday, July 20, 2017 at 2:10 PM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Re: Using Redis or Zookeeper for Email Monitoring

Agreed. I think some architecture diagrams are in order.

Marlon

From: "Shenoy, Gourav Ganesh" <go...@indiana.edu>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Thursday, July 20, 2017 at 1:35 PM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Re: Using Redis or Zookeeper for Email Monitoring

Apoorv,

Good thought about making the email monitoring service scalable and reliable. But I was wondering if you considered the Helix angle? Helix inherently uses Zookeeper to manage state and the cluster as a whole. You might want to consider leveraging some of Helix’s data handling capabilities. Nothing against Redis, Zookeeper looks like a definite candidate to me just because of Helix.

Also, could you chalk out and share an architecture of how this email monitoring service would integrate with Airavata (assume a Helix architecture). Drawing an architecture would help you better understand the problem and even find smaller issues which you didn’t consider.

Good work so far!

Thanks and Regards,
Gourav Shenoy

From: Apoorv Palkar <ap...@aol.com>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Wednesday, July 19, 2017 at 1:59 PM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Using Redis or Zookeeper for Email Monitoring

Currently I'm trying to solve the problem of done emails coming before start emails and making the system scalable. Today, we have only one GFaC that handles the email monitoring system. As we are moving towards a microservices approach from the monolithic code we have, this email monitoring system also needs to adapt to these changes. Currently, in the gfac code, a concurrent hashmap is kept to keep track of start/end of emails using their respective experiment ID's. Instead of keeping the hashmap locally, we should keep it in a global state so in the future multiple GFaCs can handle the map. Supun has suggested to use Zookeeper for this as it has high avaliability and realibility. I was also thinking that since these experiment IDs are a key value pair, Redis would be a good option for such a use case. What do you guys think about each one. I understand airavata currently uses zookeeper, so development wise there seems to be an edge toward it. Would Redis be a good for such use case?

Re: Using Redis or Zookeeper for Email Monitoring

Posted by "Pierce, Marlon" <ma...@iu.edu>.
Agreed. I think some architecture diagrams are in order.  

 

Marlon

 

From: "Shenoy, Gourav Ganesh" <go...@indiana.edu>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Thursday, July 20, 2017 at 1:35 PM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Re: Using Redis or Zookeeper for Email Monitoring

 

Apoorv,

 

Good thought about making the email monitoring service scalable and reliable. But I was wondering if you considered the Helix angle? Helix inherently uses Zookeeper to manage state and the cluster as a whole. You might want to consider leveraging some of Helix’s data handling capabilities. Nothing against Redis, Zookeeper looks like a definite candidate to me just because of Helix. 

 

Also, could you chalk out and share an architecture of how this email monitoring service would integrate with Airavata (assume a Helix architecture). Drawing an architecture would help you better understand the problem and even find smaller issues which you didn’t consider.

 

Good work so far!

 

Thanks and Regards,

Gourav Shenoy 

 

From: Apoorv Palkar <ap...@aol.com>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Wednesday, July 19, 2017 at 1:59 PM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Using Redis or Zookeeper for Email Monitoring

 

Currently I'm trying to solve the problem of done emails coming before start emails and making the system scalable. Today, we have only one GFaC that handles the email monitoring system. As we are moving towards a microservices approach from the monolithic code we have, this email monitoring system also needs to adapt to these changes. Currently, in the gfac code, a concurrent hashmap is kept to keep track of start/end of emails using their respective experiment ID's. Instead of keeping the hashmap locally, we should keep it in a global state so in the future multiple GFaCs can handle the map. Supun has suggested to use Zookeeper for this as it has high avaliability and realibility. I was also thinking that since these experiment IDs are a key value pair, Redis would be a good option for such a use case. What do you guys think about each one. I understand airavata currently uses zookeeper, so development wise there seems to be an edge toward it. Would Redis be a good for such use case?


Re: Using Redis or Zookeeper for Email Monitoring

Posted by "Shenoy, Gourav Ganesh" <go...@indiana.edu>.
Apoorv,

Good thought about making the email monitoring service scalable and reliable. But I was wondering if you considered the Helix angle? Helix inherently uses Zookeeper to manage state and the cluster as a whole. You might want to consider leveraging some of Helix’s data handling capabilities. Nothing against Redis, Zookeeper looks like a definite candidate to me just because of Helix.

Also, could you chalk out and share an architecture of how this email monitoring service would integrate with Airavata (assume a Helix architecture). Drawing an architecture would help you better understand the problem and even find smaller issues which you didn’t consider.

Good work so far!

Thanks and Regards,
Gourav Shenoy

From: Apoorv Palkar <ap...@aol.com>
Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Date: Wednesday, July 19, 2017 at 1:59 PM
To: "dev@airavata.apache.org" <de...@airavata.apache.org>
Subject: Using Redis or Zookeeper for Email Monitoring

Currently I'm trying to solve the problem of done emails coming before start emails and making the system scalable. Today, we have only one GFaC that handles the email monitoring system. As we are moving towards a microservices approach from the monolithic code we have, this email monitoring system also needs to adapt to these changes. Currently, in the gfac code, a concurrent hashmap is kept to keep track of start/end of emails using their respective experiment ID's. Instead of keeping the hashmap locally, we should keep it in a global state so in the future multiple GFaCs can handle the map. Supun has suggested to use Zookeeper for this as it has high avaliability and realibility. I was also thinking that since these experiment IDs are a key value pair, Redis would be a good option for such a use case. What do you guys think about each one. I understand airavata currently uses zookeeper, so development wise there seems to be an edge toward it. Would Redis be a good for such use case?