Queries on MFT containerization

["Rastogi, Shivam" <sr...@iu.edu>]

Hi Team,


For our final project, we are trying to containerize and deploy MFT services in a service-mesh.


Could you please advise on the following.


1. Why did we use Consul in MFT, instead of Kafka or some other message broker? Trying to understand the advantage that Consul provides. As per our understanding, Consul is only used as a message broker(key value store)



2. As per our initial understanding, there are 5 services + 1 consul service in MFT. Do you think there will be any issues if we segregate the MFT code in 5 different docker containers + 1 container for the consul service? Service breakdown is given below.


     *   API service - Used by clients for submitting/fetching requests
     *   Controller - Communicates with Consul for kvs, accepts live agents
     *   Resource service - Fetching details of the resources(files) that need to be transferred
     *   Secret service - Storing/Fetching the credentials for different services
     *   Agent - Manages the transfer of files, connections with external services
     *   Consul - Used for communication between different services, using a key-value pair system (something similar to Kafka).



Please let us know your views.

Re: Queries on MFT containerization

[DImuthu Upeksha <di...@gmail.com>]

Hi Shivam,

I will reply inline

On Sun, Apr 19, 2020 at 12:06 PM Rastogi, Shivam <sr...@iu.edu> wrote:

> Hi Team,
>
>
> For our final project, we are trying to containerize and deploy MFT
> services in a service-mesh.
>
>
> Could you please advise on the following.
>
>
> 1. Why did we use Consul in MFT, instead of Kafka or some other message
> broker? Trying to understand the advantage that Consul provides. As per
> our understanding, Consul is only used as a message broker(key value store)
>
>
>
Consul is used for multiple use cases in MFT. Asynchronous messaging is
one. We use it for leader election and distributed locks as well. For
example, framework allows only one agent to connect to the framework with a
unique name using distributed locks. And in future we plan to provide the
leader election among multiple controllers through consul.


> 2. As per our initial understanding, there are 5 services + 1 consul
> service in MFT. Do you think there will be any issues if we segregate the
> MFT code in 5 different docker containers + 1 container for the consul
> service? Service breakdown is given below.
>
>
>
>    1. API service - Used by clients for submitting/fetching requests
>       2. Controller - Communicates with Consul for kvs, accepts live
>       agents
>       3. Resource service - Fetching details of the resources(files) that
>       need to be transferred
>       4. Secret service - Storing/Fetching the credentials for different
>       services
>       5. Agent - Manages the transfer of files, connections with external
>       services
>       6. Consul - Used for communication between different services,
>       using a key-value pair system (something similar to Kafka).
>
>
>
Your understanding is correct. However for Agents, identity is critical. So
if you are using kubernetes to deploy them, try to use stateful sets to
deploy agents

[1] https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

Please let us know your views.
>
>
>