[airavata-mft] Separating mft-agent module to a golang implementation

[Dinuka Desilva <l....@gmail.com>]

Hi,

I'm currently looking in to convert the existing java implementation of
agent module to golang considering better concurrency, performance and make
it more platform enabled. Before just starting up, I'm trying to
discuss which kind of changes has to be done on which end.

1) I believe we are considering to accommodate multiple agents available
per controller and each agent to support multiple transports. Which means,
it's a collection of agents connected to a one controller. I think this is
not done yet.

2) Currently my understanding according to the implementation is that the
agent module is kind of coupled with the api and controller. I feel that
agent has to be an independent module. Or am I missing something?

3) The agent being independent is required for the agent to be converted to
golang too.

4) I was also thinking whether the combination of transports available per
agent also should be independently installable. So, that each agent could
decide which transports are required. Plus they could introduce new
transport plugins. (A kind of an enhancement)

5) I'm just doubt about the "LOCAL" transport. how does it work with
multiple agents. According to the architecture where does the local storage
exist? Is it inside the agent?

Regards,
Dinuka

Re: [airavata-mft] Separating mft-agent module to a golang implementation

[Dinuka Desilva <l....@gmail.com>]

On Sun, Apr 19, 2020 at 9:22 PM DImuthu Upeksha <di...@gmail.com>
wrote:

> Hi Dinuka,
>
> I will reply inline
>
> On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva <l....@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm currently looking in to convert the existing java implementation of
>> agent module to golang considering better concurrency, performance and make
>> it more platform enabled. Before just starting up, I'm trying to
>> discuss which kind of changes has to be done on which end.
>>
>> 1) I believe we are considering to accommodate multiple agents available
>> per controller and each agent to support multiple transports. Which means,
>> it's a collection of agents connected to a one controller. I think this is
>> not done yet.
>>
>> There should be only one active controller for entire system. Controller
> is the master and agents are the workers. This follows the master slave
> architecture. Controller sends messages to agents when a transfer needs to
> be performed. And this is already in place.
>
Got it.

>
> 2) Currently my understanding according to the implementation is that the
>> agent module is kind of coupled with the api and controller. I feel that
>> agent has to be an independent module. Or am I missing something?
>>
>>
> Agent is not coupled with the controller. But it has a standard messaging
> API built on top of Consul. If controller needs to get some work done by
> the agent, it has to communicate in that particular messaging API. But
> there is no code level dependency between the controller and agents.
> However Agent has dependencies with Resource and Secret APIs to fetch
> metadata of the resources. This is also not with the Service but with the
> APIs.
>
I can see proto3 interfaces only for api, resource service and secret
service. And I can see all the agents are mapped to switches on core [2][3]
which talk to java dependencies. That's why I was thinking that way. Can
you reference me where the agent interfaces are.


>
>> 3) The agent being independent is required for the agent to be converted
>> to golang too.
>>
>>
> Yes agent has to be independent from the code level with other components
> of the framework but it should have API level dependencies to communicate
> with the rest of the system. These are implemented on top of protobuf for
> Resource and Secret services which can be easily built for golang and
> consul level messaging dependencies with the controller which can be
> fulfilled through a golang consul client or just following consul HTTP API.
>
Yes, resource service and secret service can be access via API. My question
is how can I make the agent written in golang connect to the core [2][3]
unless it's mapped via a REST api.

>
> 4) I was also thinking whether the combination of transports available per
>> agent also should be independently installable. So, that each agent could
>> decide which transports are required. Plus they could introduce new
>> transport plugins. (A kind of an enhancement)
>>
>
> +1 and this is a good feature to have in agents. Agent have a feature [1]
> to inform controller about supported protocols but this not still validated
> from the controller when submitting a transfer job. We need to implement
> this feature
>

>> 5) I'm just doubt about the "LOCAL" transport. how does it work with
>> multiple agents. According to the architecture where does the local storage
>> exist? Is it inside the agent?
>>
>
> Local resource is attached to the Agent. If and Agent receives a local
> resource transfer request, it looks at the local file system of the machine
> where it was installed.
>
Could we have such scenarios where the source or destination could be a
storage attached to an agent server storage? Just curious to know.

*References*
[1]
https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245

[2]
https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java#L29
[3]
https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/MetadataCollectorResolver.java#L31


>
>
>> Regards,
>> Dinuka
>>
>

Re: [airavata-mft] Separating mft-agent module to a golang implementation

[Dinuka Desilva <l....@gmail.com>]

On Sun, Apr 19, 2020 at 9:22 PM DImuthu Upeksha <di...@gmail.com>
wrote:

> Hi Dinuka,
>
> I will reply inline
>
> On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva <l....@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm currently looking in to convert the existing java implementation of
>> agent module to golang considering better concurrency, performance and make
>> it more platform enabled. Before just starting up, I'm trying to
>> discuss which kind of changes has to be done on which end.
>>
>> 1) I believe we are considering to accommodate multiple agents available
>> per controller and each agent to support multiple transports. Which means,
>> it's a collection of agents connected to a one controller. I think this is
>> not done yet.
>>
>> There should be only one active controller for entire system. Controller
> is the master and agents are the workers. This follows the master slave
> architecture. Controller sends messages to agents when a transfer needs to
> be performed. And this is already in place.
>
Got it.

>
> 2) Currently my understanding according to the implementation is that the
>> agent module is kind of coupled with the api and controller. I feel that
>> agent has to be an independent module. Or am I missing something?
>>
>>
> Agent is not coupled with the controller. But it has a standard messaging
> API built on top of Consul. If controller needs to get some work done by
> the agent, it has to communicate in that particular messaging API. But
> there is no code level dependency between the controller and agents.
> However Agent has dependencies with Resource and Secret APIs to fetch
> metadata of the resources. This is also not with the Service but with the
> APIs.
>
I can see proto3 interfaces only for api, resource service and secret
service. And I can see all the agents are mapped to switches on core [2][3]
which talk to java dependencies. That's why I was thinking that way. Can
you reference me where the agent interfaces are.


>
>> 3) The agent being independent is required for the agent to be converted
>> to golang too.
>>
>>
> Yes agent has to be independent from the code level with other components
> of the framework but it should have API level dependencies to communicate
> with the rest of the system. These are implemented on top of protobuf for
> Resource and Secret services which can be easily built for golang and
> consul level messaging dependencies with the controller which can be
> fulfilled through a golang consul client or just following consul HTTP API.
>
Yes, resource service and secret service can be access via API. My question
is how can I make the agent written in golang connect to the core [2][3]
unless it's mapped via a REST api.

>
> 4) I was also thinking whether the combination of transports available per
>> agent also should be independently installable. So, that each agent could
>> decide which transports are required. Plus they could introduce new
>> transport plugins. (A kind of an enhancement)
>>
>
> +1 and this is a good feature to have in agents. Agent have a feature [1]
> to inform controller about supported protocols but this not still validated
> from the controller when submitting a transfer job. We need to implement
> this feature
>

>> 5) I'm just doubt about the "LOCAL" transport. how does it work with
>> multiple agents. According to the architecture where does the local storage
>> exist? Is it inside the agent?
>>
>
> Local resource is attached to the Agent. If and Agent receives a local
> resource transfer request, it looks at the local file system of the machine
> where it was installed.
>
Could we have such scenarios where the source or destination could be a
storage attached to an agent server storage? Just curious to know.

*References*
[1]
https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245

[2]
https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java#L29
[3]
https://github.com/apache/airavata-mft/blob/master/core/src/main/java/org/apache/airavata/mft/core/MetadataCollectorResolver.java#L31


>
>
>> Regards,
>> Dinuka
>>
>

Re: [airavata-mft] Separating mft-agent module to a golang implementation

[DImuthu Upeksha <di...@gmail.com>]

Hi Dinuka,

I will reply inline

On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva <l....@gmail.com>
wrote:

> Hi,
>
> I'm currently looking in to convert the existing java implementation of
> agent module to golang considering better concurrency, performance and make
> it more platform enabled. Before just starting up, I'm trying to
> discuss which kind of changes has to be done on which end.
>
> 1) I believe we are considering to accommodate multiple agents available
> per controller and each agent to support multiple transports. Which means,
> it's a collection of agents connected to a one controller. I think this is
> not done yet.
>
> There should be only one active controller for entire system. Controller
is the master and agents are the workers. This follows the master slave
architecture. Controller sends messages to agents when a transfer needs to
be performed. And this is already in place.

2) Currently my understanding according to the implementation is that the
> agent module is kind of coupled with the api and controller. I feel that
> agent has to be an independent module. Or am I missing something?
>
>
Agent is not coupled with the controller. But it has a standard messaging
API built on top of Consul. If controller needs to get some work done by
the agent, it has to communicate in that particular messaging API. But
there is no code level dependency between the controller and agents.
However Agent has dependencies with Resource and Secret APIs to fetch
metadata of the resources. This is also not with the Service but with the
APIs.


> 3) The agent being independent is required for the agent to be converted
> to golang too.
>
>
Yes agent has to be independent from the code level with other components
of the framework but it should have API level dependencies to communicate
with the rest of the system. These are implemented on top of protobuf for
Resource and Secret services which can be easily built for golang and
consul level messaging dependencies with the controller which can be
fulfilled through a golang consul client or just following consul HTTP API.

4) I was also thinking whether the combination of transports available per
> agent also should be independently installable. So, that each agent could
> decide which transports are required. Plus they could introduce new
> transport plugins. (A kind of an enhancement)
>

+1 and this is a good feature to have in agents. Agent have a feature [1]
to inform controller about supported protocols but this not still validated
from the controller when submitting a transfer job. We need to implement
this feature

[1]
https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245


>
> 5) I'm just doubt about the "LOCAL" transport. how does it work with
> multiple agents. According to the architecture where does the local storage
> exist? Is it inside the agent?
>

Local resource is attached to the Agent. If and Agent receives a local
resource transfer request, it looks at the local file system of the machine
where it was installed.


> Regards,
> Dinuka
>

Re: [airavata-mft] Separating mft-agent module to a golang implementation

[DImuthu Upeksha <di...@gmail.com>]

Hi Dinuka,

I will reply inline

On Fri, Apr 17, 2020 at 12:37 PM Dinuka Desilva <l....@gmail.com>
wrote:

> Hi,
>
> I'm currently looking in to convert the existing java implementation of
> agent module to golang considering better concurrency, performance and make
> it more platform enabled. Before just starting up, I'm trying to
> discuss which kind of changes has to be done on which end.
>
> 1) I believe we are considering to accommodate multiple agents available
> per controller and each agent to support multiple transports. Which means,
> it's a collection of agents connected to a one controller. I think this is
> not done yet.
>
> There should be only one active controller for entire system. Controller
is the master and agents are the workers. This follows the master slave
architecture. Controller sends messages to agents when a transfer needs to
be performed. And this is already in place.

2) Currently my understanding according to the implementation is that the
> agent module is kind of coupled with the api and controller. I feel that
> agent has to be an independent module. Or am I missing something?
>
>
Agent is not coupled with the controller. But it has a standard messaging
API built on top of Consul. If controller needs to get some work done by
the agent, it has to communicate in that particular messaging API. But
there is no code level dependency between the controller and agents.
However Agent has dependencies with Resource and Secret APIs to fetch
metadata of the resources. This is also not with the Service but with the
APIs.


> 3) The agent being independent is required for the agent to be converted
> to golang too.
>
>
Yes agent has to be independent from the code level with other components
of the framework but it should have API level dependencies to communicate
with the rest of the system. These are implemented on top of protobuf for
Resource and Secret services which can be easily built for golang and
consul level messaging dependencies with the controller which can be
fulfilled through a golang consul client or just following consul HTTP API.

4) I was also thinking whether the combination of transports available per
> agent also should be independently installable. So, that each agent could
> decide which transports are required. Plus they could introduce new
> transport plugins. (A kind of an enhancement)
>

+1 and this is a good feature to have in agents. Agent have a feature [1]
to inform controller about supported protocols but this not still validated
from the controller when submitting a transfer job. We need to implement
this feature

[1]
https://github.com/apache/airavata-mft/blob/master/agent/src/main/java/org/apache/airavata/mft/agent/MFTAgent.java#L245


>
> 5) I'm just doubt about the "LOCAL" transport. how does it work with
> multiple agents. According to the architecture where does the local storage
> exist? Is it inside the agent?
>

Local resource is attached to the Agent. If and Agent receives a local
resource transfer request, it looks at the local file system of the machine
where it was installed.


> Regards,
> Dinuka
>