You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Praneeth Kumar Chityala <pr...@gmail.com> on 2023/08/14 18:58:57 UTC
CyberShuttle | Updates on orchestration agent and consul
Dear All,
I wanted to post an update of the orchestration agent and consul components
of the architecture.
Orchestration Agent:
- After multiple considerations came to a conclusion to use nomad as the
container orchestration agent
- Advantages of using nomad are:
- It can execute docker/container applications with simple json
configuration file
- It can also execute java applications (so we can use MFT without
containerizing it)
- It works with SDKs in many languages and also it has simple http
client which makes it web friendly agent
- Present orchestration agent could connect to Nomad, start/stop job,
fetch existing jobs
- Orchestration agent in available in cybershuttle-server -
https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
Consul (message passing):
- Inspired from the architecture of MFT, used consul as the main service
in communications from server to orchestration agent
- Advantages of consul in this use case:
- We can easily keep track of all available orchestrations agents
(service discovery and availability)
- We can control access of each orchestration agent with controlled
KV store access
- It makes our server and orchestration agent decoupled which makes
huge scope for horizontal scaling when required
- Consul configurations as in this PR -
https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
- ConsulClient of cybershuttle server in this PR -
https://github.com/cyber-shuttle/cybershuttle-server/pull/4
- As of now consul to server integration is completed
Next steps to complete the whole integration:
- Code integration of consul and orchestration agent
- Final end to end testing of operation from desktop to server to consul
to orchestration agent
Please feel free to comment or suggest.
Best Regards,
Praneeth Chityala
Re: CyberShuttle | Updates on orchestration agent and consul
Posted by Praneeth Kumar Chityala <pr...@gmail.com>.
Hi Suresh,
Sure, I will look at the requirements of the format for the journal and
write up the article as required for your and Dimuthu's review.
Best,
Praneeth
On Sun, Aug 27, 2023 at 8:19 AM Suresh Marru <sm...@apache.org> wrote:
> Hi Praneeth,
>
> This is very nice work. Can you consider submitting this as a journal
> article to Journal of Open Source Software - https://joss.theoj.org/
>
> Suresh
>
> On Aug 26, 2023, at 9:10 PM, Praneeth Kumar Chityala <
> praneethchityala44@gmail.com> wrote:
>
> Hi Suresh,
>
> Please find the link below to access the summary of the project.
>
> GSoC-2023-Praneeth_Chityala
> <https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment>
> (
> https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment
> )
>
> I will be working on Medium article and demo video over the next few weeks.
>
> Best,
> Praneeth
>
>
> On Fri, Aug 25, 2023 at 4:41 PM Suresh Marru <sm...@apache.org> wrote:
>
>> Also, both of you feel free to publish Medium articles on your work or
>> let me know if you want them to get highlighted in other Apache Blogs.
>>
>> Suresh
>>
>> On Aug 25, 2023, at 4:39 PM, Praneeth Kumar Chityala <
>> praneethchityala44@gmail.com> wrote:
>>
>> Hi Suresh,
>>
>> Sure, I am working on the last piece of integration of the orchestration
>> agent to consul. I will create a clear article and update it over the
>> weekend.
>>
>> Best,
>> Praneeth
>>
>> On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru <sm...@apache.org> wrote:
>>
>>> Hi Praneeth,
>>>
>>> Can you create a child page here and summarize your project -
>>> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>>>
>>> Suresh
>>>
>>> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala <
>>> praneethchityala44@gmail.com> wrote:
>>>
>>> Dear All,
>>>
>>> I wanted to post an update of the orchestration agent and consul
>>> components of the architecture.
>>>
>>> Orchestration Agent:
>>>
>>> - After multiple considerations came to a conclusion to use nomad as
>>> the container orchestration agent
>>> - Advantages of using nomad are:
>>> - It can execute docker/container applications with simple json
>>> configuration file
>>> - It can also execute java applications (so we can use MFT
>>> without containerizing it)
>>> - It works with SDKs in many languages and also it has simple
>>> http client which makes it web friendly agent
>>> - Present orchestration agent could connect to Nomad, start/stop
>>> job, fetch existing jobs
>>> - Orchestration agent in available in cybershuttle-server -
>>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>>>
>>> Consul (message passing):
>>>
>>> - Inspired from the architecture of MFT, used consul as the main
>>> service in communications from server to orchestration agent
>>> - Advantages of consul in this use case:
>>> - We can easily keep track of all available orchestrations agents
>>> (service discovery and availability)
>>> - We can control access of each orchestration agent with
>>> controlled KV store access
>>> - It makes our server and orchestration agent decoupled which
>>> makes huge scope for horizontal scaling when required
>>> - Consul configurations as in this PR -
>>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
>>> - ConsulClient of cybershuttle server in this PR -
>>> https://github.com/cyber-shuttle/cybershuttle-server/pull/4
>>> - As of now consul to server integration is completed
>>>
>>>
>>> Next steps to complete the whole integration:
>>>
>>> - Code integration of consul and orchestration agent
>>> - Final end to end testing of operation from desktop to server to
>>> consul to orchestration agent
>>>
>>>
>>> Please feel free to comment or suggest.
>>>
>>> Best Regards,
>>> Praneeth Chityala
>>>
>>>
>>>
>>
>
Re: CyberShuttle | Updates on orchestration agent and consul
Posted by Suresh Marru <sm...@apache.org>.
Hi Praneeth,
This is very nice work. Can you consider submitting this as a journal article to Journal of Open Source Software - https://joss.theoj.org/
Suresh
> On Aug 26, 2023, at 9:10 PM, Praneeth Kumar Chityala <pr...@gmail.com> wrote:
>
> Hi Suresh,
>
> Please find the link below to access the summary of the project.
>
> GSoC-2023-Praneeth_Chityala <https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment> (https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment)
>
> I will be working on Medium article and demo video over the next few weeks.
>
> Best,
> Praneeth
>
>
> On Fri, Aug 25, 2023 at 4:41 PM Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>> Also, both of you feel free to publish Medium articles on your work or let me know if you want them to get highlighted in other Apache Blogs.
>>
>> Suresh
>>
>>> On Aug 25, 2023, at 4:39 PM, Praneeth Kumar Chityala <praneethchityala44@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Hi Suresh,
>>>
>>> Sure, I am working on the last piece of integration of the orchestration agent to consul. I will create a clear article and update it over the weekend.
>>>
>>> Best,
>>> Praneeth
>>>
>>> On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>>>> Hi Praneeth,
>>>>
>>>> Can you create a child page here and summarize your project - https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>>>>
>>>> Suresh
>>>>
>>>>> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala <praneethchityala44@gmail.com <ma...@gmail.com>> wrote:
>>>>>
>>>>> Dear All,
>>>>>
>>>>> I wanted to post an update of the orchestration agent and consul components of the architecture.
>>>>>
>>>>> Orchestration Agent:
>>>>> After multiple considerations came to a conclusion to use nomad as the container orchestration agent
>>>>> Advantages of using nomad are:
>>>>> It can execute docker/container applications with simple json configuration file
>>>>> It can also execute java applications (so we can use MFT without containerizing it)
>>>>> It works with SDKs in many languages and also it has simple http client which makes it web friendly agent
>>>>> Present orchestration agent could connect to Nomad, start/stop job, fetch existing jobs
>>>>> Orchestration agent in available in cybershuttle-server - https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>>>>> Consul (message passing):
>>>>> Inspired from the architecture of MFT, used consul as the main service in communications from server to orchestration agent
>>>>> Advantages of consul in this use case:
>>>>> We can easily keep track of all available orchestrations agents (service discovery and availability)
>>>>> We can control access of each orchestration agent with controlled KV store access
>>>>> It makes our server and orchestration agent decoupled which makes huge scope for horizontal scaling when required
>>>>> Consul configurations as in this PR - https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
>>>>> ConsulClient of cybershuttle server in this PR - https://github.com/cyber-shuttle/cybershuttle-server/pull/4
>>>>> As of now consul to server integration is completed
>>>>>
>>>>> Next steps to complete the whole integration:
>>>>> Code integration of consul and orchestration agent
>>>>> Final end to end testing of operation from desktop to server to consul to orchestration agent
>>>>>
>>>>> Please feel free to comment or suggest.
>>>>>
>>>>> Best Regards,
>>>>> Praneeth Chityala
>>>>
>>
Re: CyberShuttle | Updates on orchestration agent and consul
Posted by Praneeth Kumar Chityala <pr...@gmail.com>.
Hi Suresh,
Please find the link below to access the summary of the project.
GSoC-2023-Praneeth_Chityala
<https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment>
(
https://cwiki.apache.org/confluence/display/AIRAVATA/MFT+agent+deployment+and+lifecycle+automation+in+distributed+environment
)
I will be working on Medium article and demo video over the next few weeks.
Best,
Praneeth
On Fri, Aug 25, 2023 at 4:41 PM Suresh Marru <sm...@apache.org> wrote:
> Also, both of you feel free to publish Medium articles on your work or let
> me know if you want them to get highlighted in other Apache Blogs.
>
> Suresh
>
> On Aug 25, 2023, at 4:39 PM, Praneeth Kumar Chityala <
> praneethchityala44@gmail.com> wrote:
>
> Hi Suresh,
>
> Sure, I am working on the last piece of integration of the orchestration
> agent to consul. I will create a clear article and update it over the
> weekend.
>
> Best,
> Praneeth
>
> On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Praneeth,
>>
>> Can you create a child page here and summarize your project -
>> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>>
>> Suresh
>>
>> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala <
>> praneethchityala44@gmail.com> wrote:
>>
>> Dear All,
>>
>> I wanted to post an update of the orchestration agent and consul
>> components of the architecture.
>>
>> Orchestration Agent:
>>
>> - After multiple considerations came to a conclusion to use nomad as
>> the container orchestration agent
>> - Advantages of using nomad are:
>> - It can execute docker/container applications with simple json
>> configuration file
>> - It can also execute java applications (so we can use MFT without
>> containerizing it)
>> - It works with SDKs in many languages and also it has simple http
>> client which makes it web friendly agent
>> - Present orchestration agent could connect to Nomad, start/stop job,
>> fetch existing jobs
>> - Orchestration agent in available in cybershuttle-server -
>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>>
>> Consul (message passing):
>>
>> - Inspired from the architecture of MFT, used consul as the main
>> service in communications from server to orchestration agent
>> - Advantages of consul in this use case:
>> - We can easily keep track of all available orchestrations agents
>> (service discovery and availability)
>> - We can control access of each orchestration agent with
>> controlled KV store access
>> - It makes our server and orchestration agent decoupled which
>> makes huge scope for horizontal scaling when required
>> - Consul configurations as in this PR -
>> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
>> - ConsulClient of cybershuttle server in this PR -
>> https://github.com/cyber-shuttle/cybershuttle-server/pull/4
>> - As of now consul to server integration is completed
>>
>>
>> Next steps to complete the whole integration:
>>
>> - Code integration of consul and orchestration agent
>> - Final end to end testing of operation from desktop to server to
>> consul to orchestration agent
>>
>>
>> Please feel free to comment or suggest.
>>
>> Best Regards,
>> Praneeth Chityala
>>
>>
>>
>
Re: CyberShuttle | Updates on orchestration agent and consul
Posted by Suresh Marru <sm...@apache.org>.
Also, both of you feel free to publish Medium articles on your work or let me know if you want them to get highlighted in other Apache Blogs.
Suresh
> On Aug 25, 2023, at 4:39 PM, Praneeth Kumar Chityala <pr...@gmail.com> wrote:
>
> Hi Suresh,
>
> Sure, I am working on the last piece of integration of the orchestration agent to consul. I will create a clear article and update it over the weekend.
>
> Best,
> Praneeth
>
> On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>> Hi Praneeth,
>>
>> Can you create a child page here and summarize your project - https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>>
>> Suresh
>>
>>> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala <praneethchityala44@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Dear All,
>>>
>>> I wanted to post an update of the orchestration agent and consul components of the architecture.
>>>
>>> Orchestration Agent:
>>> After multiple considerations came to a conclusion to use nomad as the container orchestration agent
>>> Advantages of using nomad are:
>>> It can execute docker/container applications with simple json configuration file
>>> It can also execute java applications (so we can use MFT without containerizing it)
>>> It works with SDKs in many languages and also it has simple http client which makes it web friendly agent
>>> Present orchestration agent could connect to Nomad, start/stop job, fetch existing jobs
>>> Orchestration agent in available in cybershuttle-server - https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>>> Consul (message passing):
>>> Inspired from the architecture of MFT, used consul as the main service in communications from server to orchestration agent
>>> Advantages of consul in this use case:
>>> We can easily keep track of all available orchestrations agents (service discovery and availability)
>>> We can control access of each orchestration agent with controlled KV store access
>>> It makes our server and orchestration agent decoupled which makes huge scope for horizontal scaling when required
>>> Consul configurations as in this PR - https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
>>> ConsulClient of cybershuttle server in this PR - https://github.com/cyber-shuttle/cybershuttle-server/pull/4
>>> As of now consul to server integration is completed
>>>
>>> Next steps to complete the whole integration:
>>> Code integration of consul and orchestration agent
>>> Final end to end testing of operation from desktop to server to consul to orchestration agent
>>>
>>> Please feel free to comment or suggest.
>>>
>>> Best Regards,
>>> Praneeth Chityala
>>
Re: CyberShuttle | Updates on orchestration agent and consul
Posted by Praneeth Kumar Chityala <pr...@gmail.com>.
Hi Suresh,
Sure, I am working on the last piece of integration of the orchestration
agent to consul. I will create a clear article and update it over the
weekend.
Best,
Praneeth
On Fri, Aug 25, 2023 at 4:25 PM Suresh Marru <sm...@apache.org> wrote:
> Hi Praneeth,
>
> Can you create a child page here and summarize your project -
> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
>
> Suresh
>
> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala <
> praneethchityala44@gmail.com> wrote:
>
> Dear All,
>
> I wanted to post an update of the orchestration agent and consul
> components of the architecture.
>
> Orchestration Agent:
>
> - After multiple considerations came to a conclusion to use nomad as
> the container orchestration agent
> - Advantages of using nomad are:
> - It can execute docker/container applications with simple json
> configuration file
> - It can also execute java applications (so we can use MFT without
> containerizing it)
> - It works with SDKs in many languages and also it has simple http
> client which makes it web friendly agent
> - Present orchestration agent could connect to Nomad, start/stop job,
> fetch existing jobs
> - Orchestration agent in available in cybershuttle-server -
> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
>
> Consul (message passing):
>
> - Inspired from the architecture of MFT, used consul as the main
> service in communications from server to orchestration agent
> - Advantages of consul in this use case:
> - We can easily keep track of all available orchestrations agents
> (service discovery and availability)
> - We can control access of each orchestration agent with controlled
> KV store access
> - It makes our server and orchestration agent decoupled which makes
> huge scope for horizontal scaling when required
> - Consul configurations as in this PR -
> https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
> - ConsulClient of cybershuttle server in this PR -
> https://github.com/cyber-shuttle/cybershuttle-server/pull/4
> - As of now consul to server integration is completed
>
>
> Next steps to complete the whole integration:
>
> - Code integration of consul and orchestration agent
> - Final end to end testing of operation from desktop to server to
> consul to orchestration agent
>
>
> Please feel free to comment or suggest.
>
> Best Regards,
> Praneeth Chityala
>
>
>
Re: CyberShuttle | Updates on orchestration agent and consul
Posted by Suresh Marru <sm...@apache.org>.
Hi Praneeth,
Can you create a child page here and summarize your project - https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+2023
Suresh
> On Aug 14, 2023, at 2:58 PM, Praneeth Kumar Chityala <pr...@gmail.com> wrote:
>
> Dear All,
>
> I wanted to post an update of the orchestration agent and consul components of the architecture.
>
> Orchestration Agent:
> After multiple considerations came to a conclusion to use nomad as the container orchestration agent
> Advantages of using nomad are:
> It can execute docker/container applications with simple json configuration file
> It can also execute java applications (so we can use MFT without containerizing it)
> It works with SDKs in many languages and also it has simple http client which makes it web friendly agent
> Present orchestration agent could connect to Nomad, start/stop job, fetch existing jobs
> Orchestration agent in available in cybershuttle-server - https://github.com/cyber-shuttle/cybershuttle-server/tree/main/orchestration-agent
> Consul (message passing):
> Inspired from the architecture of MFT, used consul as the main service in communications from server to orchestration agent
> Advantages of consul in this use case:
> We can easily keep track of all available orchestrations agents (service discovery and availability)
> We can control access of each orchestration agent with controlled KV store access
> It makes our server and orchestration agent decoupled which makes huge scope for horizontal scaling when required
> Consul configurations as in this PR - https://github.com/cyber-shuttle/cybershuttle-server/tree/main/consul
> ConsulClient of cybershuttle server in this PR - https://github.com/cyber-shuttle/cybershuttle-server/pull/4
> As of now consul to server integration is completed
>
> Next steps to complete the whole integration:
> Code integration of consul and orchestration agent
> Final end to end testing of operation from desktop to server to consul to orchestration agent
>
> Please feel free to comment or suggest.
>
> Best Regards,
> Praneeth Chityala