You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Shameera Rathnayaka <sh...@gmail.com> on 2015/06/02 17:51:07 UTC
Re: Airavata 0.16 maven module organization
Hi Suresh,
After reading this mail, what i understood is, you are not going to solve
the problem we have, instead you are replacing one problem with another.
Why we need maven module refactoring in the first place is to reduce
unnecessary maven modules. What you are suggesting is replacing another set
of modules with existing one. Instead explaining our architecture using
maven modules we should use maven modules to manage project. let's use wiki
to explain our architectural design. According to above structures how many
maven modules will be there?
Instead of planing 2.0 or any other release goals lets live with what we
have now, let's talk about release 1.0 and then focus on 2.0 goals. For the
1.0 release do we need this level of categorization?
Thanks,
Shameera.
On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi All,
>
> While we let other thread on leaf modules going, here is a abstract
> suggestion for maven modules. Please critique:
>
> We need to ensure the directory structure used for code navigability
> should not created additional maven release artifacts. In 0.15 and earlier
> versions, we have been releasing these empty artifacts for no good reason.
>
> ├── airavata-api
> │ ├── airavata-api-interface-descriptions (these are the thrift IDL’s)
> │ ├── airavata-data-models (thrift generated data models, used by
> clients, components and api server)
> │ ├── airavata-api-java-stubs (thrift generated java stubs used by
> clients as well has servers for implementations)
> │ ├── airavata-api-server (api service handlers and startup classes)
> │ ├── api-security-manager (gsoc project to add API level oauth security)
> ├── clients (self explanatory client SDK’s - thrift generated code plus
> samples)
> │ ├── airavata-client-cpp-sdk
> │ ├── airavata-client-java-sdk
> │ ├── airavata-client-php-sdk
> │ ├── airavata-client-python-sdk
> │ ├── airavata-sample-examples
> │ └── airavata-xbaya-gui
> ├── distribution
> ├── integration-tests
>
> For component organization we can go any number of ways. The fundamental
> proposition I have is components are literally loosely coupled and live by
> themselves.
>
> One suggestion is to organize all component interface descriptions,
> services and clients into and then component implementations (which
> internally can have interfaces, implementations of these interfaces, util
> packages and so forth).
>
> ├── components
> │ ├── commons
> │ ├── component-interface-descriptions
> │ ├── component-services
> │ │ ├── credential-store-service
> │ │ ├── orchestrator-service
> │ │ ├── task-executor-service
> │ │ └── workflow-interpreter-service
> │ ├── component-clients
> │ │ ├── credential-store-client
> │ │ ├── orchestrator-client
> │ │ ├── task-executor-client
> │ │ ├── workflow-interpreter-client
> │ │ └── messaging
> │ ├── orchestrator-component-impl
> │ │ ├── orchestrator-interface
> │ │ └── orchestrator-impl….
> │ ├── registry
> │ │ ├── app-catalog
> │ │ ├── experiment-catalog
> │ │ └── resource-catalog
>
> Other suggestion is to organize by a component. Within the component it
> can also have the service interfaces and clients.
>
> This facilitates to have one more then one component implementing the same
> component interface. They are used based on usecases and stability. In no
> scenario we should have one component implementation depending on other
> component’s implementation. If there is really a common functionality we
> should have a module in commons used by both. At most the component should
> only have dependency to component's thrift clients or in case of registry
> and messaging appropriate java clients.
>
> This is again abstract, once we debate on the down sides and address them,
> we can make this concrete to cover it all.
>
> Cheers,
> Suresh
>
>
>
>
>
>
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Airavata 0.16 maven module organization
Posted by Suresh Marru <sm...@apache.org>.
On Jun 2, 2015, at 1:33 PM, Shameera Rathnayaka <sh...@gmail.com> wrote:
>
> Hi Suresh,
>
> On Tue, Jun 2, 2015 at 12:15 PM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
> Hi Shameera,
>
> Can you be concrete on what the problem you foresee and lets address it. If the problem is number if maven modules, is it really a problem?
>
> Problem is current maven dependency graph is complex and prone to have cyclic maven dependency. One reason for this is, both implementations and interfaces are reside in one maven module. It is clear that we need to separate it out.
This is a component level detail which needs to be handled within the components. We should not unnecessary proliferate these into airavata level. The current cyclic dependencies may be an oversight or lack of co-ordination at the component level. We just need to fix it. If a component has too many implementations, it’s probably worth considering to spin it off as a high level component. For instance, BES can be a component by itself implementing submitJob thrift CPI instead of implementing GFac interface. If it is also using a GridFTP file transfer module then it can be made into a library which both components use.
A simple example is the recent effort in discarding EC2 implementations. If it was residing in a component by itself and implementing a high level submitJob interface then it would have prevailed. It is important to preserve such components as they can attract developers to fix them. Currently we discard everything core developers do not support. This is not sustainable.
> Another reason is having large number of maven module which also make dependency graph complex. For this we need to reduce number of maven modules. And only have most required modules. So can you think only this problem and explain how your suggestion resolve this problem?
I do not have a good answer with a complex dependency graph. As long as it is navigable what is the issue? is IDE not able to handle it? the module organization I am proposing is a higherarcial and should make it manageable. If at all we think one repo should not have so many modules, we should fork into multiple repositories (components can safely sit in separate repos by themselves).
> I certainly not explaining architecture through maven. What I am trying to address for 1.0 with this is:
>
> * Lets say we have validation implemented by component A version 1.3. We have new validation implementation which confirms to AbstractOrchestrator then it becomes Component A version 1.4 (which will have two implementations and have some switching logic at the factory class level). Instead, we might also have a totally new component G which will have its own internal interface and implementation and released as Component G version 1 and so on.
>
> * Lets say we want to add a new job submission to NERSC resources. We could add a new host and proceed. Alternately we wanted to try out integration with NEWT (https://newt.nersc.gov/ <https://newt.nersc.gov/>). Instead of dovetailing the implementation to implement GFac interface, we then want to avoid adding BetterGFac or EvenBetterGfac interfaces to common core. We instead simply can develop a new component which implements submit job component interface. Internally it may have any implementations (programmed through Java interfaces and classes or scala or python or whatever).
>
> Could you please explain why this is not possible with suggested maven module refactoring?
As I explained on earlier thread, it pulls together component level implementation details (lower level java interfaces) into parent airavata core. This counters the conceptual architectural patten that there is no common core. But Airavata constitutes loosely composed components which are implementation agnostic to one another.
Hope this explains all your concerns,
Suresh
>
> Thanks,
> Shameera.
>
>
>
> So this organization is to facilitate components can evolve liberally without touching anything existing. In operational deployment we just figure out routing so at work queue level new component pick up jobs. At higher level, we can have the logic that if an experimental new component fails,it can put the job in the queue for a proven stable component and so on.
>
> The down side is we add maven modules in order of number of competent multiplied by 2 or 3. On a up side, these components are totally loosely coupled and development evolves rapidly and provides deployment flexibility.
>
> Suresh
>
>> On Jun 2, 2015, at 11:51 AM, Shameera Rathnayaka <shameerainfo@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hi Suresh,
>>
>> After reading this mail, what i understood is, you are not going to solve the problem we have, instead you are replacing one problem with another. Why we need maven module refactoring in the first place is to reduce unnecessary maven modules. What you are suggesting is replacing another set of modules with existing one. Instead explaining our architecture using maven modules we should use maven modules to manage project. let's use wiki to explain our architectural design. According to above structures how many maven modules will be there?
>>
>> Instead of planing 2.0 or any other release goals lets live with what we have now, let's talk about release 1.0 and then focus on 2.0 goals. For the 1.0 release do we need this level of categorization?
>>
>> Thanks,
>> Shameera.
>>
>> On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
>> Hi All,
>>
>> While we let other thread on leaf modules going, here is a abstract suggestion for maven modules. Please critique:
>>
>> We need to ensure the directory structure used for code navigability should not created additional maven release artifacts. In 0.15 and earlier versions, we have been releasing these empty artifacts for no good reason.
>>
>> ├── airavata-api
>> │ ├── airavata-api-interface-descriptions (these are the thrift IDL’s)
>> │ ├── airavata-data-models (thrift generated data models, used by clients, components and api server)
>> │ ├── airavata-api-java-stubs (thrift generated java stubs used by clients as well has servers for implementations)
>> │ ├── airavata-api-server (api service handlers and startup classes)
>> │ ├── api-security-manager (gsoc project to add API level oauth security)
>> ├── clients (self explanatory client SDK’s - thrift generated code plus samples)
>> │ ├── airavata-client-cpp-sdk
>> │ ├── airavata-client-java-sdk
>> │ ├── airavata-client-php-sdk
>> │ ├── airavata-client-python-sdk
>> │ ├── airavata-sample-examples
>> │ └── airavata-xbaya-gui
>> ├── distribution
>> ├── integration-tests
>>
>> For component organization we can go any number of ways. The fundamental proposition I have is components are literally loosely coupled and live by themselves.
>>
>> One suggestion is to organize all component interface descriptions, services and clients into and then component implementations (which internally can have interfaces, implementations of these interfaces, util packages and so forth).
>>
>> ├── components
>> │ ├── commons
>> │ ├── component-interface-descriptions
>> │ ├── component-services
>> │ │ ├── credential-store-service
>> │ │ ├── orchestrator-service
>> │ │ ├── task-executor-service
>> │ │ └── workflow-interpreter-service
>> │ ├── component-clients
>> │ │ ├── credential-store-client
>> │ │ ├── orchestrator-client
>> │ │ ├── task-executor-client
>> │ │ ├── workflow-interpreter-client
>> │ │ └── messaging
>> │ ├── orchestrator-component-impl
>> │ │ ├── orchestrator-interface
>> │ │ └── orchestrator-impl….
>> │ ├── registry
>> │ │ ├── app-catalog
>> │ │ ├── experiment-catalog
>> │ │ └── resource-catalog
>>
>> Other suggestion is to organize by a component. Within the component it can also have the service interfaces and clients.
>>
>> This facilitates to have one more then one component implementing the same component interface. They are used based on usecases and stability. In no scenario we should have one component implementation depending on other component’s implementation. If there is really a common functionality we should have a module in commons used by both. At most the component should only have dependency to component's thrift clients or in case of registry and messaging appropriate java clients.
>>
>> This is again abstract, once we debate on the down sides and address them, we can make this concrete to cover it all.
>>
>> Cheers,
>> Suresh
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com <http://gmail.com/>
>> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>
>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com <http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>
Re: Airavata 0.16 maven module organization
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Suresh,
On Tue, Jun 2, 2015 at 12:15 PM, Suresh Marru <sm...@apache.org> wrote:
> Hi Shameera,
>
> Can you be concrete on what the problem you foresee and lets address it.
> If the problem is number if maven modules, is it really a problem?
>
Problem is current maven dependency graph is complex and prone to have
cyclic maven dependency. One reason for this is, both implementations and
interfaces are reside in one maven module. It is clear that we need to
separate it out. Another reason is having large number of maven module
which also make dependency graph complex. For this we need to reduce number
of maven modules. And only have most required modules. So can you think
only this problem and explain how your suggestion resolve this problem?
>
> I certainly not explaining architecture through maven. What I am trying to
> address for 1.0 with this is:
>
> * Lets say we have validation implemented by component A version 1.3. We
> have new validation implementation which confirms to AbstractOrchestrator then
> it becomes Component A version 1.4 (which will have two implementations and
> have some switching logic at the factory class level). Instead, we might
> also have a totally new component G which will have its own internal
> interface and implementation and released as Component G version 1 and so
> on.
>
> * Lets say we want to add a new job submission to NERSC resources. We
> could add a new host and proceed. Alternately we wanted to try out
> integration with NEWT (https://newt.nersc.gov/). Instead of dovetailing
> the implementation to implement GFac interface, we then want to avoid
> adding BetterGFac or EvenBetterGfac interfaces to common core. We instead
> simply can develop a new component which implements submit job component
> interface. Internally it may have any implementations (programmed through
> Java interfaces and classes or scala or python or whatever).
>
Could you please explain why this is not possible with suggested maven
module refactoring?
Thanks,
Shameera.
>
> So this organization is to facilitate components can evolve liberally
> without touching anything existing. In operational deployment we just
> figure out routing so at work queue level new component pick up jobs. At
> higher level, we can have the logic that if an experimental new component
> fails,it can put the job in the queue for a proven stable component and so
> on.
>
> The down side is we add maven modules in order of number of competent
> multiplied by 2 or 3. On a up side, these components are totally loosely
> coupled and development evolves rapidly and provides deployment
> flexibility.
>
> Suresh
>
> On Jun 2, 2015, at 11:51 AM, Shameera Rathnayaka <sh...@gmail.com>
> wrote:
>
> Hi Suresh,
>
> After reading this mail, what i understood is, you are not going to solve
> the problem we have, instead you are replacing one problem with another.
> Why we need maven module refactoring in the first place is to reduce
> unnecessary maven modules. What you are suggesting is replacing another set
> of modules with existing one. Instead explaining our architecture using
> maven modules we should use maven modules to manage project. let's use wiki
> to explain our architectural design. According to above structures how many
> maven modules will be there?
>
> Instead of planing 2.0 or any other release goals lets live with what we
> have now, let's talk about release 1.0 and then focus on 2.0 goals. For the
> 1.0 release do we need this level of categorization?
>
> Thanks,
> Shameera.
>
> On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi All,
>>
>> While we let other thread on leaf modules going, here is a abstract
>> suggestion for maven modules. Please critique:
>>
>> We need to ensure the directory structure used for code navigability
>> should not created additional maven release artifacts. In 0.15 and earlier
>> versions, we have been releasing these empty artifacts for no good reason.
>>
>> ├── airavata-api
>> │ ├── airavata-api-interface-descriptions (these are the thrift IDL’s)
>> │ ├── airavata-data-models (thrift generated data models, used by
>> clients, components and api server)
>> │ ├── airavata-api-java-stubs (thrift generated java stubs used by
>> clients as well has servers for implementations)
>> │ ├── airavata-api-server (api service handlers and startup classes)
>> │ ├── api-security-manager (gsoc project to add API level oauth
>> security)
>> ├── clients (self explanatory client SDK’s - thrift generated code plus
>> samples)
>> │ ├── airavata-client-cpp-sdk
>> │ ├── airavata-client-java-sdk
>> │ ├── airavata-client-php-sdk
>> │ ├── airavata-client-python-sdk
>> │ ├── airavata-sample-examples
>> │ └── airavata-xbaya-gui
>> ├── distribution
>> ├── integration-tests
>>
>> For component organization we can go any number of ways. The fundamental
>> proposition I have is components are literally loosely coupled and live by
>> themselves.
>>
>> One suggestion is to organize all component interface descriptions,
>> services and clients into and then component implementations (which
>> internally can have interfaces, implementations of these interfaces, util
>> packages and so forth).
>>
>> ├── components
>> │ ├── commons
>> │ ├── component-interface-descriptions
>> │ ├── component-services
>> │ │ ├── credential-store-service
>> │ │ ├── orchestrator-service
>> │ │ ├── task-executor-service
>> │ │ └── workflow-interpreter-service
>> │ ├── component-clients
>> │ │ ├── credential-store-client
>> │ │ ├── orchestrator-client
>> │ │ ├── task-executor-client
>> │ │ ├── workflow-interpreter-client
>> │ │ └── messaging
>> │ ├── orchestrator-component-impl
>> │ │ ├── orchestrator-interface
>> │ │ └── orchestrator-impl….
>> │ ├── registry
>> │ │ ├── app-catalog
>> │ │ ├── experiment-catalog
>> │ │ └── resource-catalog
>>
>> Other suggestion is to organize by a component. Within the component it
>> can also have the service interfaces and clients.
>>
>> This facilitates to have one more then one component implementing the
>> same component interface. They are used based on usecases and stability. In
>> no scenario we should have one component implementation depending on other
>> component’s implementation. If there is really a common functionality we
>> should have a module in commons used by both. At most the component should
>> only have dependency to component's thrift clients or in case of registry
>> and messaging appropriate java clients.
>>
>> This is again abstract, once we debate on the down sides and address
>> them, we can make this concrete to cover it all.
>>
>> Cheers,
>> Suresh
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org , shameerainfo AT gmail.com
> Blog : http://shameerarathnayaka.blogspot.com/
>
>
>
--
Best Regards,
Shameera Rathnayaka.
email: shameera AT apache.org , shameerainfo AT gmail.com
Blog : http://shameerarathnayaka.blogspot.com/
Re: Airavata 0.16 maven module organization
Posted by Suresh Marru <sm...@apache.org>.
Hi Shameera,
Can you be concrete on what the problem you foresee and lets address it. If the problem is number if maven modules, is it really a problem?
I certainly not explaining architecture through maven. What I am trying to address for 1.0 with this is:
* Lets say we have validation implemented by component A version 1.3. We have new validation implementation which confirms to AbstractOrchestrator then it becomes Component A version 1.4 (which will have two implementations and have some switching logic at the factory class level). Instead, we might also have a totally new component G which will have its own internal interface and implementation and released as Component G version 1 and so on.
* Lets say we want to add a new job submission to NERSC resources. We could add a new host and proceed. Alternately we wanted to try out integration with NEWT (https://newt.nersc.gov/ <https://newt.nersc.gov/>). Instead of dovetailing the implementation to implement GFac interface, we then want to avoid adding BetterGFac or EvenBetterGfac interfaces to common core. We instead simply can develop a new component which implements submit job component interface. Internally it may have any implementations (programmed through Java interfaces and classes or scala or python or whatever).
So this organization is to facilitate components can evolve liberally without touching anything existing. In operational deployment we just figure out routing so at work queue level new component pick up jobs. At higher level, we can have the logic that if an experimental new component fails,it can put the job in the queue for a proven stable component and so on.
The down side is we add maven modules in order of number of competent multiplied by 2 or 3. On a up side, these components are totally loosely coupled and development evolves rapidly and provides deployment flexibility.
Suresh
> On Jun 2, 2015, at 11:51 AM, Shameera Rathnayaka <sh...@gmail.com> wrote:
>
> Hi Suresh,
>
> After reading this mail, what i understood is, you are not going to solve the problem we have, instead you are replacing one problem with another. Why we need maven module refactoring in the first place is to reduce unnecessary maven modules. What you are suggesting is replacing another set of modules with existing one. Instead explaining our architecture using maven modules we should use maven modules to manage project. let's use wiki to explain our architectural design. According to above structures how many maven modules will be there?
>
> Instead of planing 2.0 or any other release goals lets live with what we have now, let's talk about release 1.0 and then focus on 2.0 goals. For the 1.0 release do we need this level of categorization?
>
> Thanks,
> Shameera.
>
> On Tue, Jun 2, 2015 at 11:25 AM, Suresh Marru <smarru@apache.org <ma...@apache.org>> wrote:
> Hi All,
>
> While we let other thread on leaf modules going, here is a abstract suggestion for maven modules. Please critique:
>
> We need to ensure the directory structure used for code navigability should not created additional maven release artifacts. In 0.15 and earlier versions, we have been releasing these empty artifacts for no good reason.
>
> ├── airavata-api
> │ ├── airavata-api-interface-descriptions (these are the thrift IDL’s)
> │ ├── airavata-data-models (thrift generated data models, used by clients, components and api server)
> │ ├── airavata-api-java-stubs (thrift generated java stubs used by clients as well has servers for implementations)
> │ ├── airavata-api-server (api service handlers and startup classes)
> │ ├── api-security-manager (gsoc project to add API level oauth security)
> ├── clients (self explanatory client SDK’s - thrift generated code plus samples)
> │ ├── airavata-client-cpp-sdk
> │ ├── airavata-client-java-sdk
> │ ├── airavata-client-php-sdk
> │ ├── airavata-client-python-sdk
> │ ├── airavata-sample-examples
> │ └── airavata-xbaya-gui
> ├── distribution
> ├── integration-tests
>
> For component organization we can go any number of ways. The fundamental proposition I have is components are literally loosely coupled and live by themselves.
>
> One suggestion is to organize all component interface descriptions, services and clients into and then component implementations (which internally can have interfaces, implementations of these interfaces, util packages and so forth).
>
> ├── components
> │ ├── commons
> │ ├── component-interface-descriptions
> │ ├── component-services
> │ │ ├── credential-store-service
> │ │ ├── orchestrator-service
> │ │ ├── task-executor-service
> │ │ └── workflow-interpreter-service
> │ ├── component-clients
> │ │ ├── credential-store-client
> │ │ ├── orchestrator-client
> │ │ ├── task-executor-client
> │ │ ├── workflow-interpreter-client
> │ │ └── messaging
> │ ├── orchestrator-component-impl
> │ │ ├── orchestrator-interface
> │ │ └── orchestrator-impl….
> │ ├── registry
> │ │ ├── app-catalog
> │ │ ├── experiment-catalog
> │ │ └── resource-catalog
>
> Other suggestion is to organize by a component. Within the component it can also have the service interfaces and clients.
>
> This facilitates to have one more then one component implementing the same component interface. They are used based on usecases and stability. In no scenario we should have one component implementation depending on other component’s implementation. If there is really a common functionality we should have a module in commons used by both. At most the component should only have dependency to component's thrift clients or in case of registry and messaging appropriate java clients.
>
> This is again abstract, once we debate on the down sides and address them, we can make this concrete to cover it all.
>
> Cheers,
> Suresh
>
>
>
>
>
>
>
>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> email: shameera AT apache.org <http://apache.org/> , shameerainfo AT gmail.com <http://gmail.com/>
> Blog : http://shameerarathnayaka.blogspot.com/ <http://shameerarathnayaka.blogspot.com/>