You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Raminder Singh <ra...@gmail.com> on 2014/10/28 20:32:14 UTC
Separate Thrift services- Code restructure
Hi All,
I am fixing AIRAVATA-1471 to create separate distributions for all the Thrift services in Airavata so that we can run all in separate JVMs and dockerize (www.docker.com) the servers. In this exercise, i found we don’t have client stubs for several components in separate artifacts like Orchestrator Client is part of Orcherstrator Service, GFAC client is part of Orcherstator-Core, Workflow server and client is part of Airavata API server and client. To be consistent with API server and reduce maven dependency tree, i am going to create airavata— <component>—stubs package and add the component client to that project. I need to move the code and changed dependencies etc. Please let me know if there are any objections. If not i will go ahead tomorrow and make the change and commit them after testing.
Thanks
Raminder
Re: Separate Thrift services- Code restructure
Posted by Suresh Marru <sm...@apache.org>.
Thanks Amila for chiming in and you hit it right. I do not think we should disrupt any fault tolerant architecture. When we discussed the FT, the focus was on single job submissions and the idea of Orchestrator sprung up as a higher level component to GFac. Following the same design, Workflow Engine needs to also sit behind Orchestrator.
The current question though is, should workflow engine be a separate service or have a simple java API which runs within Orchestrator server. Either way we keep it as a separate service or bundle it, I do not think it will not hinder andy FT strategies. But good reminder through, we need to walk through the scenarios and be definite.
Suresh
On Nov 5, 2014, at 12:32 PM, Amila Jayasekara <th...@gmail.com> wrote:
> Swiftly went through this thread. Seems you are going to merge workflow service and orchestrator (if not ignore my comment).
>
> I remember we putting lot of thoughts to fault tolerance aspects of Orchestrator and also workflow interpreter some time back. I do believe merging will effect the FT design we came up at that time (Hope those design notes are somewhere in the SGG repo) and you may need to rethink about FT aspects.
>
> Thanks
> -AJ
>
> On Thu, Oct 30, 2014 at 11:34 AM, Raminder Singh <ra...@gmail.com> wrote:
> We need to approach merging of Workflow server slightly different. Methods like registerWorflow, getWorkflow etc need to be part of API server so client like Xbaya need to call them remotely. I am going to get rid of workflow server and merge methods to Airavata API server and handle launch in Orchestrator.
>
> Thanks
> Raminder
>
>>>
>>> From: Shameera Rathnayaka [mailto:shameerainfo@gmail.com]
>>> Sent: Tuesday, October 28, 2014 3:56 PM
>>> To: dev
>>> Subject: Re: Separate Thrift services- Code restructure
>>>
>>> +1 for merging Workflow service with Orchestrator,
>>> we are not get any advantage by keeping those two as separate services.
>>>
>>> Thanks,
>>> Shameera.
>>>
>>> On Tue, Oct 28, 2014 at 4:35 PM, Raminderjeet Singh <ra...@gmail.com>> wrote:
>>> I need to move workflow sever/client out out API server to remove extra dependencies on workflow model. I am going to move workflow server and client to orchestrator server and can get rid of server part as next step.
>>>
>>> Thanks
>>> Raminder
>>>
>>> On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <sm...@apache.org>> wrote:
>>> + 1.
>>>
>>> I think we can leave out the workflow service and its probably best to embedded it with orchestrator, since there is so much overlap. So that leaves 3 services:
>>>
>>> API Server - Client
>>> Orchestrator Server -Client
>>> GFac Server - Client
>>>
>>> Suresh
>>>
>>> On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am fixing AIRAVATA-1471 to create separate distributions for all the Thrift services in Airavata so that we can run all in separate JVMs and dockerize (www.docker.com<http://www.docker.com>) the servers. In this exercise, i found we don’t have client stubs for several components in separate artifacts like Orchestrator Client is part of Orcherstrator Service, GFAC client is part of Orcherstator-Core, Workflow server and client is part of Airavata API server and client. To be consistent with API server and reduce maven dependency tree, i am going to create airavata— <component>—stubs package and add the component client to that project. I need to move the code and changed dependencies etc. Please let me know if there are any objections. If not i will go ahead tomorrow and make the change and commit them after testing.
>>>>
>>>> Thanks
>>>> Raminder
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Shameera Rathnayaka.
>>>
>>> email: shameera AT apache.org<http://apache.org> , shameerainfo AT gmail.com<http://gmail.com>
>>> Blog : http://shameerarathnayaka.blogspot.com/
>
>
Re: Separate Thrift services- Code restructure
Posted by Amila Jayasekara <th...@gmail.com>.
Swiftly went through this thread. Seems you are going to merge workflow
service and orchestrator (if not ignore my comment).
I remember we putting lot of thoughts to fault tolerance aspects of
Orchestrator and also workflow interpreter some time back. I do believe
merging will effect the FT design we came up at that time (Hope those
design notes are somewhere in the SGG repo) and you may need to rethink
about FT aspects.
Thanks
-AJ
On Thu, Oct 30, 2014 at 11:34 AM, Raminder Singh <ra...@gmail.com>
wrote:
> We need to approach merging of Workflow server slightly different. Methods
> like registerWorflow, getWorkflow etc need to be part of API server so
> client like Xbaya need to call them remotely. I am going to get rid of
> workflow server and merge methods to Airavata API server and handle launch
> in Orchestrator.
>
> Thanks
> Raminder
>
>
> From: Shameera Rathnayaka [mailto:shameerainfo@gmail.com
> <sh...@gmail.com>]
> Sent: Tuesday, October 28, 2014 3:56 PM
> To: dev
> Subject: Re: Separate Thrift services- Code restructure
>
> +1 for merging Workflow service with Orchestrator,
> we are not get any advantage by keeping those two as separate services.
>
> Thanks,
> Shameera.
>
> On Tue, Oct 28, 2014 at 4:35 PM, Raminderjeet Singh <
> raminderjsingh@gmail.com<mailto:raminderjsingh@gmail.com
> <ra...@gmail.com>>> wrote:
> I need to move workflow sever/client out out API server to remove extra
> dependencies on workflow model. I am going to move workflow server and
> client to orchestrator server and can get rid of server part as next step.
>
> Thanks
> Raminder
>
> On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <smarru@apache.org<
> mailto:smarru@apache.org <sm...@apache.org>>> wrote:
> + 1.
>
> I think we can leave out the workflow service and its probably best to
> embedded it with orchestrator, since there is so much overlap. So that
> leaves 3 services:
>
> API Server - Client
> Orchestrator Server -Client
> GFac Server - Client
>
> Suresh
>
> On Oct 28, 2014, at 3:32 PM, Raminder Singh <raminderjsingh@gmail.com<
> mailto:raminderjsingh@gmail.com <ra...@gmail.com>>> wrote:
>
> Hi All,
>
> I am fixing AIRAVATA-1471 to create separate distributions for all the
> Thrift services in Airavata so that we can run all in separate JVMs and
> dockerize (www.docker.com<http://www.docker.com>) the servers. In this
> exercise, i found we don’t have client stubs for several components in
> separate artifacts like Orchestrator Client is part of Orcherstrator
> Service, GFAC client is part of Orcherstator-Core, Workflow server and
> client is part of Airavata API server and client. To be consistent with API
> server and reduce maven dependency tree, i am going to create airavata—
> <component>—stubs package and add the component client to that project. I
> need to move the code and changed dependencies etc. Please let me know if
> there are any objections. If not i will go ahead tomorrow and make the
> change and commit them after testing.
>
> Thanks
> Raminder
>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org<http://apache.org> , shameerainfo AT
> gmail.com<http://gmail.com>
> Blog : http://shameerarathnayaka.blogspot.com/
>
>
>
Re: Separate Thrift services- Code restructure
Posted by Raminder Singh <ra...@gmail.com>.
We need to approach merging of Workflow server slightly different. Methods like registerWorflow, getWorkflow etc need to be part of API server so client like Xbaya need to call them remotely. I am going to get rid of workflow server and merge methods to Airavata API server and handle launch in Orchestrator.
Thanks
Raminder
>>
>> From: Shameera Rathnayaka [mailto:shameerainfo@gmail.com]
>> Sent: Tuesday, October 28, 2014 3:56 PM
>> To: dev
>> Subject: Re: Separate Thrift services- Code restructure
>>
>> +1 for merging Workflow service with Orchestrator,
>> we are not get any advantage by keeping those two as separate services.
>>
>> Thanks,
>> Shameera.
>>
>> On Tue, Oct 28, 2014 at 4:35 PM, Raminderjeet Singh <ra...@gmail.com>> wrote:
>> I need to move workflow sever/client out out API server to remove extra dependencies on workflow model. I am going to move workflow server and client to orchestrator server and can get rid of server part as next step.
>>
>> Thanks
>> Raminder
>>
>> On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <sm...@apache.org>> wrote:
>> + 1.
>>
>> I think we can leave out the workflow service and its probably best to embedded it with orchestrator, since there is so much overlap. So that leaves 3 services:
>>
>> API Server - Client
>> Orchestrator Server -Client
>> GFac Server - Client
>>
>> Suresh
>>
>> On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com>> wrote:
>>
>>> Hi All,
>>>
>>> I am fixing AIRAVATA-1471 to create separate distributions for all the Thrift services in Airavata so that we can run all in separate JVMs and dockerize (www.docker.com<http://www.docker.com>) the servers. In this exercise, i found we don’t have client stubs for several components in separate artifacts like Orchestrator Client is part of Orcherstrator Service, GFAC client is part of Orcherstator-Core, Workflow server and client is part of Airavata API server and client. To be consistent with API server and reduce maven dependency tree, i am going to create airavata— <component>—stubs package and add the component client to that project. I need to move the code and changed dependencies etc. Please let me know if there are any objections. If not i will go ahead tomorrow and make the change and commit them after testing.
>>>
>>> Thanks
>>> Raminder
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org<http://apache.org> , shameerainfo AT gmail.com<http://gmail.com>
>> Blog : http://shameerarathnayaka.blogspot.com/
Re: Separate Thrift services- Code restructure
Posted by Marlon Pierce <ma...@iu.edu>.
Hi Sudhakar,
The current code base is https://github.com/apache/airavata/tree/master.
To grab a local copy, use "git clone
https://github.com/apache/airavata.git". Note this is a mirror of the
Apache git repo, but that is suitable for read-only access to master and
for submitting patches and pull requests.
Marlon
On 10/28/14, 5:04 PM, Pamidighantam, Sudhakar V wrote:
> I am starting to look at Airavata with a background of having used an old snapshot in paramchem project.
> Is there a place I can go to learn the current organization of the development.
>
> Thanks,
> Sudhakar.
>
> From: Shameera Rathnayaka [mailto:shameerainfo@gmail.com]
> Sent: Tuesday, October 28, 2014 3:56 PM
> To: dev
> Subject: Re: Separate Thrift services- Code restructure
>
> +1 for merging Workflow service with Orchestrator,
> we are not get any advantage by keeping those two as separate services.
>
> Thanks,
> Shameera.
>
> On Tue, Oct 28, 2014 at 4:35 PM, Raminderjeet Singh <ra...@gmail.com>> wrote:
> I need to move workflow sever/client out out API server to remove extra dependencies on workflow model. I am going to move workflow server and client to orchestrator server and can get rid of server part as next step.
>
> Thanks
> Raminder
>
> On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <sm...@apache.org>> wrote:
> + 1.
>
> I think we can leave out the workflow service and its probably best to embedded it with orchestrator, since there is so much overlap. So that leaves 3 services:
>
> API Server - Client
> Orchestrator Server -Client
> GFac Server - Client
>
> Suresh
>
> On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com>> wrote:
>
>> Hi All,
>>
>> I am fixing AIRAVATA-1471 to create separate distributions for all the Thrift services in Airavata so that we can run all in separate JVMs and dockerize (www.docker.com<http://www.docker.com>) the servers. In this exercise, i found we don’t have client stubs for several components in separate artifacts like Orchestrator Client is part of Orcherstrator Service, GFAC client is part of Orcherstator-Core, Workflow server and client is part of Airavata API server and client. To be consistent with API server and reduce maven dependency tree, i am going to create airavata— <component>—stubs package and add the component client to that project. I need to move the code and changed dependencies etc. Please let me know if there are any objections. If not i will go ahead tomorrow and make the change and commit them after testing.
>>
>> Thanks
>> Raminder
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org<http://apache.org> , shameerainfo AT gmail.com<http://gmail.com>
> Blog : http://shameerarathnayaka.blogspot.com/
RE: Separate Thrift services- Code restructure
Posted by "Pamidighantam, Sudhakar V" <sp...@illinois.edu>.
I am starting to look at Airavata with a background of having used an old snapshot in paramchem project.
Is there a place I can go to learn the current organization of the development.
Thanks,
Sudhakar.
From: Shameera Rathnayaka [mailto:shameerainfo@gmail.com]
Sent: Tuesday, October 28, 2014 3:56 PM
To: dev
Subject: Re: Separate Thrift services- Code restructure
+1 for merging Workflow service with Orchestrator,
we are not get any advantage by keeping those two as separate services.
Thanks,
Shameera.
On Tue, Oct 28, 2014 at 4:35 PM, Raminderjeet Singh <ra...@gmail.com>> wrote:
I need to move workflow sever/client out out API server to remove extra dependencies on workflow model. I am going to move workflow server and client to orchestrator server and can get rid of server part as next step.
Thanks
Raminder
On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <sm...@apache.org>> wrote:
+ 1.
I think we can leave out the workflow service and its probably best to embedded it with orchestrator, since there is so much overlap. So that leaves 3 services:
API Server - Client
Orchestrator Server -Client
GFac Server - Client
Suresh
On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com>> wrote:
> Hi All,
>
> I am fixing AIRAVATA-1471 to create separate distributions for all the Thrift services in Airavata so that we can run all in separate JVMs and dockerize (www.docker.com<http://www.docker.com>) the servers. In this exercise, i found we don’t have client stubs for several components in separate artifacts like Orchestrator Client is part of Orcherstrator Service, GFAC client is part of Orcherstator-Core, Workflow server and client is part of Airavata API server and client. To be consistent with API server and reduce maven dependency tree, i am going to create airavata— <component>—stubs package and add the component client to that project. I need to move the code and changed dependencies etc. Please let me know if there are any objections. If not i will go ahead tomorrow and make the change and commit them after testing.
>
> Thanks
> Raminder
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org<http://apache.org> , shameerainfo AT gmail.com<http://gmail.com>
Blog : http://shameerarathnayaka.blogspot.com/
Re: Separate Thrift services- Code restructure
Posted by Shameera Rathnayaka <sh...@gmail.com>.
+1 for merging Workflow service with Orchestrator,
we are not get any advantage by keeping those two as separate services.
Thanks,
Shameera.
On Tue, Oct 28, 2014 at 4:35 PM, Raminderjeet Singh <
raminderjsingh@gmail.com> wrote:
> I need to move workflow sever/client out out API server to remove extra
> dependencies on workflow model. I am going to move workflow server and
> client to orchestrator server and can get rid of server part as next step.
>
> Thanks
> Raminder
>
> On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <sm...@apache.org> wrote:
>
>> + 1.
>>
>> I think we can leave out the workflow service and its probably best to
>> embedded it with orchestrator, since there is so much overlap. So that
>> leaves 3 services:
>>
>> API Server - Client
>> Orchestrator Server -Client
>> GFac Server - Client
>>
>> Suresh
>>
>> On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com>
>> wrote:
>>
>> > Hi All,
>> >
>> > I am fixing AIRAVATA-1471 to create separate distributions for all the
>> Thrift services in Airavata so that we can run all in separate JVMs and
>> dockerize (www.docker.com) the servers. In this exercise, i found we
>> don’t have client stubs for several components in separate artifacts like
>> Orchestrator Client is part of Orcherstrator Service, GFAC client is part
>> of Orcherstator-Core, Workflow server and client is part of Airavata API
>> server and client. To be consistent with API server and reduce maven
>> dependency tree, i am going to create airavata— <component>—stubs package
>> and add the component client to that project. I need to move the code and
>> changed dependencies etc. Please let me know if there are any objections.
>> If not i will go ahead tomorrow and make the change and commit them after
>> testing.
>> >
>> > Thanks
>> > Raminder
>>
>>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Separate Thrift services- Code restructure
Posted by Raminderjeet Singh <ra...@gmail.com>.
I need to move workflow sever/client out out API server to remove extra
dependencies on workflow model. I am going to move workflow server and
client to orchestrator server and can get rid of server part as next step.
Thanks
Raminder
On Tue, Oct 28, 2014 at 3:49 PM, Suresh Marru <sm...@apache.org> wrote:
> + 1.
>
> I think we can leave out the workflow service and its probably best to
> embedded it with orchestrator, since there is so much overlap. So that
> leaves 3 services:
>
> API Server - Client
> Orchestrator Server -Client
> GFac Server - Client
>
> Suresh
>
> On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I am fixing AIRAVATA-1471 to create separate distributions for all the
> Thrift services in Airavata so that we can run all in separate JVMs and
> dockerize (www.docker.com) the servers. In this exercise, i found we
> don’t have client stubs for several components in separate artifacts like
> Orchestrator Client is part of Orcherstrator Service, GFAC client is part
> of Orcherstator-Core, Workflow server and client is part of Airavata API
> server and client. To be consistent with API server and reduce maven
> dependency tree, i am going to create airavata— <component>—stubs package
> and add the component client to that project. I need to move the code and
> changed dependencies etc. Please let me know if there are any objections.
> If not i will go ahead tomorrow and make the change and commit them after
> testing.
> >
> > Thanks
> > Raminder
>
>
Re: Separate Thrift services- Code restructure
Posted by Suresh Marru <sm...@apache.org>.
+ 1.
I think we can leave out the workflow service and its probably best to embedded it with orchestrator, since there is so much overlap. So that leaves 3 services:
API Server - Client
Orchestrator Server -Client
GFac Server - Client
Suresh
On Oct 28, 2014, at 3:32 PM, Raminder Singh <ra...@gmail.com> wrote:
> Hi All,
>
> I am fixing AIRAVATA-1471 to create separate distributions for all the Thrift services in Airavata so that we can run all in separate JVMs and dockerize (www.docker.com) the servers. In this exercise, i found we don’t have client stubs for several components in separate artifacts like Orchestrator Client is part of Orcherstrator Service, GFAC client is part of Orcherstator-Core, Workflow server and client is part of Airavata API server and client. To be consistent with API server and reduce maven dependency tree, i am going to create airavata— <component>—stubs package and add the component client to that project. I need to move the code and changed dependencies etc. Please let me know if there are any objections. If not i will go ahead tomorrow and make the change and commit them after testing.
>
> Thanks
> Raminder
Re: Separate Thrift services- Code restructure
Posted by Lahiru Gunathilake <gl...@gmail.com>.
+1
On Tue, Oct 28, 2014 at 3:32 PM, Raminder Singh <ra...@gmail.com>
wrote:
> Hi All,
>
> I am fixing AIRAVATA-1471 to create separate distributions for all the
> Thrift services in Airavata so that we can run all in separate JVMs and
> dockerize (www.docker.com) the servers. In this exercise, i found we
> don’t have client stubs for several components in separate artifacts like
> Orchestrator Client is part of Orcherstrator Service, GFAC client is part
> of Orcherstator-Core, Workflow server and client is part of Airavata API
> server and client. To be consistent with API server and reduce maven
> dependency tree, i am going to create airavata— <component>—stubs package
> and add the component client to that project. I need to move the code and
> changed dependencies etc. Please let me know if there are any objections.
> If not i will go ahead tomorrow and make the change and commit them after
> testing.
>
> Thanks
> Raminder
--
Research Assistant
Science Gateways Group
Indiana University