Build and distribution modularity

[Jeremy Boynes <jb...@apache.org>]

We've talked about this in the past on several threads but I would  
like to bring this to conclusion. We've had a couple of issues  
recently where bits of the build were failing due to environmental  
issues (e.g. java.net repo availability, xerces versions on some JRE)  
and we are about to start adding more dependencies with the  
integration of the Axis2 module, the JSON modules, web containers,  
etc. which will make this worse. I want to try and sum up here where  
we are at and make some proposals for going forward.

==========

Proposal 1: Support independent build and distributions at the top level
Rationale: Users interested in different technologies such as SCA and  
SDO do not want to have to build all of them together. This also  
gives a false impression that there are strong dependencies between  
them.

What this means: Structure the build such that someone can check out  
any directory under "java" and build it on its own. For example, if I  
check out just "sdo" I would expect the build to work from that  
directory. All dependencies (e.g. on spec) would be resolved through  
the mvn repository.

This will require each technology to upload unstable artifacts (aka  
SNAPSHOTs) to the mvn repository on some periodic basis. These are  
not releases and do not need to be voted on.

Each technology will produce their own distribution. This may be a  
user-installable distro (e.g. for sca or sdo) or may just be a set of  
jars to be published in the mvn repo (e.g. for spec).

==========

Proposal 2: Break sca down into runtime libraries, distributions and  
extensions
Rationale: SCA involves a lot of different technologies and runs in a  
large set of environments that are not applicable to all users. We  
will not be able to keep all of this in sync all the time.

What this means: The runtime is structured as a set of libraries that  
provide the fabric for wiring services together. These libraries are  
used by a number of host environments to create running SCA  
containers (e.g. the launcher uses them to provide a command-line  
client environment, the servlet launcher uses them to provide a  
webapp based environment, the test harness provides an environment  
around JUnit, ...). The interface to those libraries is defined by  
the spi and proposed api modules. Modifications to the implementation  
of those libraries should not be visible to the hosts or extensions  
that use them. We can build and distribute (through mvn) these  
libraries separately from the things that are using them.

Users want distributions to be something that they can download,  
install and use for some purpose. We will build distributions with  
such a purpose in mind and include within them everything that a user  
would need to achieve it. For example, one purpose might be a client- 
side install so we would include within that the minimal runtime plus  
commonly used client bindings (e.g. web services); another purpose  
might be for building web applications so we would produce a  
distribution aimed to support web server functions (which may or may  
not include a server platform); another might be as a service  
intermediary which may have a very rich set of bindings.

All distributions will support some form of extension mechanism that  
allows users to add functionality by installing extension modules.  
These modules will be released and distributed separately. Some may  
be included by default in some distributions (e.g. web services).

==========

Proposal 3: It should be easy to build a distribution (including  
everything that is in it).
Rationale: We want to be able to build these things easily.

What this means: In the root of the sca project, the pom file will  
contain profiles that contain all the modules needed for a particular  
distribution. Each profile will build and package everything that  
needs to go into the distribution bundle - the runtime, all bundled  
extensions and the actual distribution bundle.

==========

Proposal 4: We need a test suite for each distribution that mirrors  
user experience
Rationale: When a user installs a distribution it's helpful if it works.

What this means: Each distribution has a test suite that tests the  
kind of things that a user would do with it (based on what its  
purpose was). The suite works by taking a distribution package,  
installing it, installing test applications and exercising them. We  
run these tests in the kind of environments that users have.

We can't do this every time we build as not everyone will have access  
to every environment. Instead we use a CI framework such as Continuum  
that can build or download a distro and exercise it on a variety of  
platforms. Any failures get converted to high-priority JIRA issues.

This is essentially an enhanced version of the old testing/tomcat  
environment - one that gets run frequently, automatically and in  
multiple environments.

==========

I realize these are high-level proposals - more a statement of  
principles really - but I wanted to capture the thinking behind the  
proposal. If we think this is the right way to go then we can dig  
into the detail needed to make it happen.

--
Jeremy


---------------------------------------------------------------------
To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: tuscany-dev-help@ws.apache.org

Re: Build and distribution modularity

[Jim Marino <jm...@myromatours.com>]

On Aug 1, 2006, at 6:07 PM, Ken Tam wrote:

>> Proposal 1: Support independent build and distributions at the top  
>> level
>> Rationale: Users interested in different technologies such as SCA and
>> SDO do not want to have to build all of them together. This also
>> gives a false impression that there are strong dependencies between
>> them.
>>
>> What this means: Structure the build such that someone can check out
>> any directory under "java" and build it on its own. For example, if I
>> check out just "sdo" I would expect the build to work from that
>> directory. All dependencies (e.g. on spec) would be resolved through
>> the mvn repository.
>>
>> This will require each technology to upload unstable artifacts (aka
>> SNAPSHOTs) to the mvn repository on some periodic basis. These are
>> not releases and do not need to be voted on.
>>
>> Each technology will produce their own distribution. This may be a
>> user-installable distro (e.g. for sca or sdo) or may just be a set of
>> jars to be published in the mvn repo (e.g. for spec).
>
> +1 on independent top-level build/distros, we already have no small
> amount of confusion in the community at large regarding dependencies
> between SCA & SDO.
>
>> Proposal 2: Break sca down into runtime libraries, distributions and
>> extensions
>> Rationale: SCA involves a lot of different technologies and runs in a
>> large set of environments that are not applicable to all users. We
>> will not be able to keep all of this in sync all the time.
>>
>> What this means: The runtime is structured as a set of libraries that
>> provide the fabric for wiring services together. These libraries are
>> used by a number of host environments to create running SCA
>> containers (e.g. the launcher uses them to provide a command-line
>> client environment, the servlet launcher uses them to provide a
>> webapp based environment, the test harness provides an environment
>> around JUnit, ...). The interface to those libraries is defined by
>> the spi and proposed api modules. Modifications to the implementation
>> of those libraries should not be visible to the hosts or extensions
>> that use them. We can build and distribute (through mvn) these
>> libraries separately from the things that are using them.
>
> In principal I think I agree with the intentions here, but I'm not so
> clear on what it really means in practice.  For example:
>
>> The interface to those libraries is defined by
>> the spi and proposed api modules. Modifications to the implementation
>> of those libraries should not be visible to the hosts or extensions
>> that use them.
>
> As I understand it, today this is definitely not true -- hosts such as
> the Launcher and SCATestCase are strongly coupled to implementation
> classes coming out of the runtime libraries via their system.scdls.
> It seems like changing this would imply that the the runtime libraries
> would include a "core" set of system SCDL definitions that would be
> leveraged by all hosts, and any host that wanted finer-grained control
> would continue to remain tightly coupled?
>
I'm not sure there is an easy way around this. Most of the  
dependencies are on things like loaders and builders. If we keep  
these in system composite scdls, there is at least some level of  
loose coupling since the dependency is on the composite component and  
not its implementation (i.e. loaders and builders).  I think we want  
to keep things as flexible as possible since a custom launcher may  
just want to prune everything down, perhaps not even having loaders  
and instead using some other config mechanism.

That said, I still think we leak classes from core into some of the  
extension modules (particularly around wires) so we may want to keep  
an eye on that and create some kind of test harness for extension  
builders to stop this from happening.

> Also not sure how this would work for extensions -- since so many of
> the SPI elements are abstract classes and not just interfaces, it
> seems it would be hard to keep impl changes in such classes from being
> visible to extensions.
>
>>
>> Users want distributions to be something that they can download,
>> install and use for some purpose. We will build distributions with
>> such a purpose in mind and include within them everything that a user
>> would need to achieve it. For example, one purpose might be a client-
>> side install so we would include within that the minimal runtime plus
>> commonly used client bindings (e.g. web services); another purpose
>> might be for building web applications so we would produce a
>> distribution aimed to support web server functions (which may or may
>> not include a server platform); another might be as a service
>> intermediary which may have a very rich set of bindings.
>>
>> All distributions will support some form of extension mechanism that
>> allows users to add functionality by installing extension modules.
>> These modules will be released and distributed separately. Some may
>> be included by default in some distributions (e.g. web services).
>
> +1 to this philosophy of distros.  In particular, I think we want to
> have a system where getting and installing extensions is super-easy a
> la maven plugins or ant tasks -- no one really worries about whether
> extensions to those programs "ship" with the default distro, since
> it's so easy to add them afterwards.  Ant's earlier concern that not
> being in a distro might make an extension "2nd class" is definitely
> something we want to avoid, but I think the right way to do that is to
> make post-facto inclusion really easy rather than including more
> extensions by default.
Yea, I think this is really key. Something like Eclipse plugins. As  
we expand the number of extensions, the resulting combinations will  
make having distributions for every one unmanageable. Having an easy  
mechanism to download on demand will provide a good solution to this.  
Taking Eclipse as an example again, I think the risk of making  
something second class may be attenuated by a having a small core  
distro with a good download system and good documentation/publicity  
for the extension. For example, WTP is not in the Eclipse core but  
doesn't seem second class.

I'm also concerned about forcing users to deal with a technology  
monolith. It's really important we allow choice with Tuscany and  
allow people progressively "buy into" whatever combination of  
technologies they need. Spring has been really successful with this  
approach. You can mix and match different pieces, including parts of  
Spring itself, and they don't force users to disable things they  
don't want. With Spring, people can use Spring WebFlow or they can  
use whatever other UI technology such as Struts, WebWork, or  
Tapestry. Likewise for the data tier, people can use Spring's JDBC  
abstraction layer or Hibernate, iBatis, or Kodo.

Regarding Tuscany, someone may want to use DAS in a J2EE web app  
without SCA. Or someone may want to use Celtix, JAXB and JPA with SCA  
as opposed to Axis, SDO and DAS. They shouldn't be forced to download  
and disable the pieces they do not need.

>
>> Proposal 3: It should be easy to build a distribution (including
>> everything that is in it).
>> Rationale: We want to be able to build these things easily.
>
> +1, how can anyone argue with "making something easy" :)  I need to
> spend more time understanding what's possible here before I have a lot
> more to say..
>
>> Proposal 4: We need a test suite for each distribution that mirrors
>> user experience
>> Rationale: When a user installs a distribution it's helpful if it  
>> works.
>> ...
>> We can't do this every time we build as not everyone will have access
>> to every environment. Instead we use a CI framework such as Continuum
>> that can build or download a distro and exercise it on a variety of
>> platforms. Any failures get converted to high-priority JIRA issues.
>
> Last time I was involved in this sort of thing around Beehive, the ASF
> did not really support any kind of CI infrastructure for its projects.
> Has that changed?  Might we persuade a company or two to pony up a
> machine on the internet?  I really like CI, though most of my
> experience is with cruise control rather than continuum.
I thought they may have added this. There's also TeamCity - OpenJPA  
is using this, hosted by BEA, and it looks pretty cool. Maybe we  
could check into this?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: tuscany-dev-help@ws.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: tuscany-dev-help@ws.apache.org

CI, was: Build and distribution modularity

[Jeremy Boynes <jb...@apache.org>]

Splitting into a different thread

On Aug 1, 2006, at 6:07 PM, Ken Tam wrote:
>> Proposal 4: We need a test suite for each distribution that mirrors
>> user experience
>> Rationale: When a user installs a distribution it's helpful if it  
>> works.
>> ...
>> We can't do this every time we build as not everyone will have access
>> to every environment. Instead we use a CI framework such as Continuum
>> that can build or download a distro and exercise it on a variety of
>> platforms. Any failures get converted to high-priority JIRA issues.
>
> Last time I was involved in this sort of thing around Beehive, the ASF
> did not really support any kind of CI infrastructure for its projects.
> Has that changed?  Might we persuade a company or two to pony up a
> machine on the internet?  I really like CI, though most of my
> experience is with cruise control rather than continuum.

I spoke with Dims in the spring about doing nightly builds using the  
zone available to the webservices project and he was OK with it. The  
only issue was that publishing to the snapshot repo required  
someone's private key to be stored on the machine.

Raymond got the build running under Continuum on his desktop so it is  
feasible. We may be able to use some of the GBuild infrastructure  
from Geronimo. I'm willing to open up a dedicated server for folk to  
use as well.

--
Jeremy


---------------------------------------------------------------------
To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: tuscany-dev-help@ws.apache.org

Re: Build and distribution modularity

[Ken Tam <ke...@gmail.com>]

> Proposal 1: Support independent build and distributions at the top level
> Rationale: Users interested in different technologies such as SCA and
> SDO do not want to have to build all of them together. This also
> gives a false impression that there are strong dependencies between
> them.
>
> What this means: Structure the build such that someone can check out
> any directory under "java" and build it on its own. For example, if I
> check out just "sdo" I would expect the build to work from that
> directory. All dependencies (e.g. on spec) would be resolved through
> the mvn repository.
>
> This will require each technology to upload unstable artifacts (aka
> SNAPSHOTs) to the mvn repository on some periodic basis. These are
> not releases and do not need to be voted on.
>
> Each technology will produce their own distribution. This may be a
> user-installable distro (e.g. for sca or sdo) or may just be a set of
> jars to be published in the mvn repo (e.g. for spec).

+1 on independent top-level build/distros, we already have no small
amount of confusion in the community at large regarding dependencies
between SCA & SDO.

> Proposal 2: Break sca down into runtime libraries, distributions and
> extensions
> Rationale: SCA involves a lot of different technologies and runs in a
> large set of environments that are not applicable to all users. We
> will not be able to keep all of this in sync all the time.
>
> What this means: The runtime is structured as a set of libraries that
> provide the fabric for wiring services together. These libraries are
> used by a number of host environments to create running SCA
> containers (e.g. the launcher uses them to provide a command-line
> client environment, the servlet launcher uses them to provide a
> webapp based environment, the test harness provides an environment
> around JUnit, ...). The interface to those libraries is defined by
> the spi and proposed api modules. Modifications to the implementation
> of those libraries should not be visible to the hosts or extensions
> that use them. We can build and distribute (through mvn) these
> libraries separately from the things that are using them.

In principal I think I agree with the intentions here, but I'm not so
clear on what it really means in practice.  For example:

> The interface to those libraries is defined by
> the spi and proposed api modules. Modifications to the implementation
> of those libraries should not be visible to the hosts or extensions
> that use them.

As I understand it, today this is definitely not true -- hosts such as
the Launcher and SCATestCase are strongly coupled to implementation
classes coming out of the runtime libraries via their system.scdls.
It seems like changing this would imply that the the runtime libraries
would include a "core" set of system SCDL definitions that would be
leveraged by all hosts, and any host that wanted finer-grained control
would continue to remain tightly coupled?

Also not sure how this would work for extensions -- since so many of
the SPI elements are abstract classes and not just interfaces, it
seems it would be hard to keep impl changes in such classes from being
visible to extensions.

>
> Users want distributions to be something that they can download,
> install and use for some purpose. We will build distributions with
> such a purpose in mind and include within them everything that a user
> would need to achieve it. For example, one purpose might be a client-
> side install so we would include within that the minimal runtime plus
> commonly used client bindings (e.g. web services); another purpose
> might be for building web applications so we would produce a
> distribution aimed to support web server functions (which may or may
> not include a server platform); another might be as a service
> intermediary which may have a very rich set of bindings.
>
> All distributions will support some form of extension mechanism that
> allows users to add functionality by installing extension modules.
> These modules will be released and distributed separately. Some may
> be included by default in some distributions (e.g. web services).

+1 to this philosophy of distros.  In particular, I think we want to
have a system where getting and installing extensions is super-easy a
la maven plugins or ant tasks -- no one really worries about whether
extensions to those programs "ship" with the default distro, since
it's so easy to add them afterwards.  Ant's earlier concern that not
being in a distro might make an extension "2nd class" is definitely
something we want to avoid, but I think the right way to do that is to
make post-facto inclusion really easy rather than including more
extensions by default.

> Proposal 3: It should be easy to build a distribution (including
> everything that is in it).
> Rationale: We want to be able to build these things easily.

+1, how can anyone argue with "making something easy" :)  I need to
spend more time understanding what's possible here before I have a lot
more to say..

> Proposal 4: We need a test suite for each distribution that mirrors
> user experience
> Rationale: When a user installs a distribution it's helpful if it works.
>...
> We can't do this every time we build as not everyone will have access
> to every environment. Instead we use a CI framework such as Continuum
> that can build or download a distro and exercise it on a variety of
> platforms. Any failures get converted to high-priority JIRA issues.

Last time I was involved in this sort of thing around Beehive, the ASF
did not really support any kind of CI infrastructure for its projects.
 Has that changed?  Might we persuade a company or two to pony up a
machine on the internet?  I really like CI, though most of my
experience is with cruise control rather than continuum.

---------------------------------------------------------------------
To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: tuscany-dev-help@ws.apache.org

Re: Build and distribution modularity

[Kevin Williams <ke...@qwest.net>]

Maybe we should take a conservative approach initially (and this is 
probably something similar to #3, #4).  As long as milestones are not 
too infrequent then I think we can release the three subprojects on the 
same schedule.   Users should be able to easily pull SCA , SDO and DAS 
components independently from a distribution.

Also, it does not seem too much an inconvenience for those interested in 
working from the head to pull and build the entire project.

--Kevin




Rick wrote:

> In theory I like what you're saying Jeremy, but in reality I'm kinding 
> of siding with Ant concerns.  Not sure how it will scale as we add may 
> extensions and all the possible combinations.  Also all the distros 
> add administration, documentation, issue tracking etc. I certainly can 
> see a separate SDO and possibly a separate DAS distro as I can see the 
> use case for that and it easily explained to the end user why they are 
> a separate distro for them.  With regard to SCA I think a goal of 
> building them separately is good,  having some test that just run with 
> each extension is good.  But maybe for the SCA distro it would be 
> easier to have the full always the full distro and some how configure 
> to only deploy and run the pieces and parts the user wants? ant elder 
> wrote:
>
>> I like proposals 3 and 4 - it should be easy to build a distribution 
>> and we
>> need to test it properly - +1!  And having some sort of daily build 
>> would be
>> great whatever ends up happening here.
>>
>> In general the idea of having something more than a single monolithic
>> distribution sounds good, I've a few concerns (version 
>> incompatibilities,
>> hard for newbies to choose a distribution, some things may become 2nd 
>> class
>> parts of the Tuscany) which I'll leave till there's more detail, but 
>> one big
>> problem I can see is that having lots of separate module/distribution
>> releases is will make it _really_ hard to get all the release votes 
>> through
>> the incubator PMC. Wont that be a bit unworkable while we're still
>> incubating?
>>
>>   ...ant
>>
>> On 7/29/06, Jeremy Boynes <jb...@apache.org> wrote:
>>
>>>
>>> We've talked about this in the past on several threads but I would
>>> like to bring this to conclusion. We've had a couple of issues
>>> recently where bits of the build were failing due to environmental
>>> issues (e.g. java.net repo availability, xerces versions on some JRE)
>>> and we are about to start adding more dependencies with the
>>> integration of the Axis2 module, the JSON modules, web containers,
>>> etc. which will make this worse. I want to try and sum up here where
>>> we are at and make some proposals for going forward.
>>>
>>> ==========
>>>
>>> Proposal 1: Support independent build and distributions at the top 
>>> level
>>> Rationale: Users interested in different technologies such as SCA and
>>> SDO do not want to have to build all of them together. This also
>>> gives a false impression that there are strong dependencies between
>>> them.
>>>
>>> What this means: Structure the build such that someone can check out
>>> any directory under "java" and build it on its own. For example, if I
>>> check out just "sdo" I would expect the build to work from that
>>> directory. All dependencies (e.g. on spec) would be resolved through
>>> the mvn repository.
>>>
>>> This will require each technology to upload unstable artifacts (aka
>>> SNAPSHOTs) to the mvn repository on some periodic basis. These are
>>> not releases and do not need to be voted on.
>>>
>>> Each technology will produce their own distribution. This may be a
>>> user-installable distro (e.g. for sca or sdo) or may just be a set of
>>> jars to be published in the mvn repo (e.g. for spec).
>>>
>>> ==========
>>>
>>> Proposal 2: Break sca down into runtime libraries, distributions and
>>> extensions
>>> Rationale: SCA involves a lot of different technologies and runs in a
>>> large set of environments that are not applicable to all users. We
>>> will not be able to keep all of this in sync all the time.
>>>
>>> What this means: The runtime is structured as a set of libraries that
>>> provide the fabric for wiring services together. These libraries are
>>> used by a number of host environments to create running SCA
>>> containers (e.g. the launcher uses them to provide a command-line
>>> client environment, the servlet launcher uses them to provide a
>>> webapp based environment, the test harness provides an environment
>>> around JUnit, ...). The interface to those libraries is defined by
>>> the spi and proposed api modules. Modifications to the implementation
>>> of those libraries should not be visible to the hosts or extensions
>>> that use them. We can build and distribute (through mvn) these
>>> libraries separately from the things that are using them.
>>>
>>> Users want distributions to be something that they can download,
>>> install and use for some purpose. We will build distributions with
>>> such a purpose in mind and include within them everything that a user
>>> would need to achieve it. For example, one purpose might be a client-
>>> side install so we would include within that the minimal runtime plus
>>> commonly used client bindings (e.g. web services); another purpose
>>> might be for building web applications so we would produce a
>>> distribution aimed to support web server functions (which may or may
>>> not include a server platform); another might be as a service
>>> intermediary which may have a very rich set of bindings.
>>>
>>> All distributions will support some form of extension mechanism that
>>> allows users to add functionality by installing extension modules.
>>> These modules will be released and distributed separately. Some may
>>> be included by default in some distributions (e.g. web services).
>>>
>>> ==========
>>>
>>> Proposal 3: It should be easy to build a distribution (including
>>> everything that is in it).
>>> Rationale: We want to be able to build these things easily.
>>>
>>> What this means: In the root of the sca project, the pom file will
>>> contain profiles that contain all the modules needed for a particular
>>> distribution. Each profile will build and package everything that
>>> needs to go into the distribution bundle - the runtime, all bundled
>>> extensions and the actual distribution bundle.
>>>
>>> ==========
>>>
>>> Proposal 4: We need a test suite for each distribution that mirrors
>>> user experience
>>> Rationale: When a user installs a distribution it's helpful if it 
>>> works.
>>>
>>> What this means: Each distribution has a test suite that tests the
>>> kind of things that a user would do with it (based on what its
>>> purpose was). The suite works by taking a distribution package,
>>> installing it, installing test applications and exercising them. We
>>> run these tests in the kind of environments that users have.
>>>
>>> We can't do this every time we build as not everyone will have access
>>> to every environment. Instead we use a CI framework such as Continuum
>>> that can build or download a distro and exercise it on a variety of
>>> platforms. Any failures get converted to high-priority JIRA issues.
>>>
>>> This is essentially an enhanced version of the old testing/tomcat
>>> environment - one that gets run frequently, automatically and in
>>> multiple environments.
>>>
>>> ==========
>>>
>>> I realize these are high-level proposals - more a statement of
>>> principles really - but I wanted to capture the thinking behind the
>>> proposal. If we think this is the right way to go then we can dig
>>> into the detail needed to make it happen.
>>>
>>> -- 
>>> Jeremy
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
>>> For additional commands, e-mail: tuscany-dev-help@ws.apache.org
>>>
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: tuscany-dev-help@ws.apache.org
>
>
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: tuscany-dev-help@ws.apache.org

Re: Build and distribution modularity

[Rick <cr...@gmail.com>]

In theory I like what you're saying Jeremy, but in reality I'm kinding 
of siding with Ant concerns.  Not sure how it will scale as we add may 
extensions and all the possible combinations.  Also all the distros add 
administration, documentation, issue tracking etc. I certainly can see a 
separate SDO and possibly a separate DAS distro as I can see the use 
case for that and it easily explained to the end user why they are a 
separate distro for them.  With regard to SCA I think a goal of building 
them separately is good,  having some test that just run with each 
extension is good.  But maybe for the SCA distro it would be easier to 
have the full always the full distro and some how configure to only 
deploy and run the pieces and parts the user wants? 
ant elder wrote:
> I like proposals 3 and 4 - it should be easy to build a distribution 
> and we
> need to test it properly - +1!  And having some sort of daily build 
> would be
> great whatever ends up happening here.
>
> In general the idea of having something more than a single monolithic
> distribution sounds good, I've a few concerns (version incompatibilities,
> hard for newbies to choose a distribution, some things may become 2nd 
> class
> parts of the Tuscany) which I'll leave till there's more detail, but 
> one big
> problem I can see is that having lots of separate module/distribution
> releases is will make it _really_ hard to get all the release votes 
> through
> the incubator PMC. Wont that be a bit unworkable while we're still
> incubating?
>
>   ...ant
>
> On 7/29/06, Jeremy Boynes <jb...@apache.org> wrote:
>>
>> We've talked about this in the past on several threads but I would
>> like to bring this to conclusion. We've had a couple of issues
>> recently where bits of the build were failing due to environmental
>> issues (e.g. java.net repo availability, xerces versions on some JRE)
>> and we are about to start adding more dependencies with the
>> integration of the Axis2 module, the JSON modules, web containers,
>> etc. which will make this worse. I want to try and sum up here where
>> we are at and make some proposals for going forward.
>>
>> ==========
>>
>> Proposal 1: Support independent build and distributions at the top level
>> Rationale: Users interested in different technologies such as SCA and
>> SDO do not want to have to build all of them together. This also
>> gives a false impression that there are strong dependencies between
>> them.
>>
>> What this means: Structure the build such that someone can check out
>> any directory under "java" and build it on its own. For example, if I
>> check out just "sdo" I would expect the build to work from that
>> directory. All dependencies (e.g. on spec) would be resolved through
>> the mvn repository.
>>
>> This will require each technology to upload unstable artifacts (aka
>> SNAPSHOTs) to the mvn repository on some periodic basis. These are
>> not releases and do not need to be voted on.
>>
>> Each technology will produce their own distribution. This may be a
>> user-installable distro (e.g. for sca or sdo) or may just be a set of
>> jars to be published in the mvn repo (e.g. for spec).
>>
>> ==========
>>
>> Proposal 2: Break sca down into runtime libraries, distributions and
>> extensions
>> Rationale: SCA involves a lot of different technologies and runs in a
>> large set of environments that are not applicable to all users. We
>> will not be able to keep all of this in sync all the time.
>>
>> What this means: The runtime is structured as a set of libraries that
>> provide the fabric for wiring services together. These libraries are
>> used by a number of host environments to create running SCA
>> containers (e.g. the launcher uses them to provide a command-line
>> client environment, the servlet launcher uses them to provide a
>> webapp based environment, the test harness provides an environment
>> around JUnit, ...). The interface to those libraries is defined by
>> the spi and proposed api modules. Modifications to the implementation
>> of those libraries should not be visible to the hosts or extensions
>> that use them. We can build and distribute (through mvn) these
>> libraries separately from the things that are using them.
>>
>> Users want distributions to be something that they can download,
>> install and use for some purpose. We will build distributions with
>> such a purpose in mind and include within them everything that a user
>> would need to achieve it. For example, one purpose might be a client-
>> side install so we would include within that the minimal runtime plus
>> commonly used client bindings (e.g. web services); another purpose
>> might be for building web applications so we would produce a
>> distribution aimed to support web server functions (which may or may
>> not include a server platform); another might be as a service
>> intermediary which may have a very rich set of bindings.
>>
>> All distributions will support some form of extension mechanism that
>> allows users to add functionality by installing extension modules.
>> These modules will be released and distributed separately. Some may
>> be included by default in some distributions (e.g. web services).
>>
>> ==========
>>
>> Proposal 3: It should be easy to build a distribution (including
>> everything that is in it).
>> Rationale: We want to be able to build these things easily.
>>
>> What this means: In the root of the sca project, the pom file will
>> contain profiles that contain all the modules needed for a particular
>> distribution. Each profile will build and package everything that
>> needs to go into the distribution bundle - the runtime, all bundled
>> extensions and the actual distribution bundle.
>>
>> ==========
>>
>> Proposal 4: We need a test suite for each distribution that mirrors
>> user experience
>> Rationale: When a user installs a distribution it's helpful if it works.
>>
>> What this means: Each distribution has a test suite that tests the
>> kind of things that a user would do with it (based on what its
>> purpose was). The suite works by taking a distribution package,
>> installing it, installing test applications and exercising them. We
>> run these tests in the kind of environments that users have.
>>
>> We can't do this every time we build as not everyone will have access
>> to every environment. Instead we use a CI framework such as Continuum
>> that can build or download a distro and exercise it on a variety of
>> platforms. Any failures get converted to high-priority JIRA issues.
>>
>> This is essentially an enhanced version of the old testing/tomcat
>> environment - one that gets run frequently, automatically and in
>> multiple environments.
>>
>> ==========
>>
>> I realize these are high-level proposals - more a statement of
>> principles really - but I wanted to capture the thinking behind the
>> proposal. If we think this is the right way to go then we can dig
>> into the detail needed to make it happen.
>>
>> -- 
>> Jeremy
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
>> For additional commands, e-mail: tuscany-dev-help@ws.apache.org
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: tuscany-dev-help@ws.apache.org

Re: Build and distribution modularity

[ant elder <an...@gmail.com>]

I like proposals 3 and 4 - it should be easy to build a distribution and we
need to test it properly - +1!  And having some sort of daily build would be
great whatever ends up happening here.

In general the idea of having something more than a single monolithic
distribution sounds good, I've a few concerns (version incompatibilities,
hard for newbies to choose a distribution, some things may become 2nd class
parts of the Tuscany) which I'll leave till there's more detail, but one big
problem I can see is that having lots of separate module/distribution
releases is will make it _really_ hard to get all the release votes through
the incubator PMC. Wont that be a bit unworkable while we're still
incubating?

   ...ant

On 7/29/06, Jeremy Boynes <jb...@apache.org> wrote:
>
> We've talked about this in the past on several threads but I would
> like to bring this to conclusion. We've had a couple of issues
> recently where bits of the build were failing due to environmental
> issues (e.g. java.net repo availability, xerces versions on some JRE)
> and we are about to start adding more dependencies with the
> integration of the Axis2 module, the JSON modules, web containers,
> etc. which will make this worse. I want to try and sum up here where
> we are at and make some proposals for going forward.
>
> ==========
>
> Proposal 1: Support independent build and distributions at the top level
> Rationale: Users interested in different technologies such as SCA and
> SDO do not want to have to build all of them together. This also
> gives a false impression that there are strong dependencies between
> them.
>
> What this means: Structure the build such that someone can check out
> any directory under "java" and build it on its own. For example, if I
> check out just "sdo" I would expect the build to work from that
> directory. All dependencies (e.g. on spec) would be resolved through
> the mvn repository.
>
> This will require each technology to upload unstable artifacts (aka
> SNAPSHOTs) to the mvn repository on some periodic basis. These are
> not releases and do not need to be voted on.
>
> Each technology will produce their own distribution. This may be a
> user-installable distro (e.g. for sca or sdo) or may just be a set of
> jars to be published in the mvn repo (e.g. for spec).
>
> ==========
>
> Proposal 2: Break sca down into runtime libraries, distributions and
> extensions
> Rationale: SCA involves a lot of different technologies and runs in a
> large set of environments that are not applicable to all users. We
> will not be able to keep all of this in sync all the time.
>
> What this means: The runtime is structured as a set of libraries that
> provide the fabric for wiring services together. These libraries are
> used by a number of host environments to create running SCA
> containers (e.g. the launcher uses them to provide a command-line
> client environment, the servlet launcher uses them to provide a
> webapp based environment, the test harness provides an environment
> around JUnit, ...). The interface to those libraries is defined by
> the spi and proposed api modules. Modifications to the implementation
> of those libraries should not be visible to the hosts or extensions
> that use them. We can build and distribute (through mvn) these
> libraries separately from the things that are using them.
>
> Users want distributions to be something that they can download,
> install and use for some purpose. We will build distributions with
> such a purpose in mind and include within them everything that a user
> would need to achieve it. For example, one purpose might be a client-
> side install so we would include within that the minimal runtime plus
> commonly used client bindings (e.g. web services); another purpose
> might be for building web applications so we would produce a
> distribution aimed to support web server functions (which may or may
> not include a server platform); another might be as a service
> intermediary which may have a very rich set of bindings.
>
> All distributions will support some form of extension mechanism that
> allows users to add functionality by installing extension modules.
> These modules will be released and distributed separately. Some may
> be included by default in some distributions (e.g. web services).
>
> ==========
>
> Proposal 3: It should be easy to build a distribution (including
> everything that is in it).
> Rationale: We want to be able to build these things easily.
>
> What this means: In the root of the sca project, the pom file will
> contain profiles that contain all the modules needed for a particular
> distribution. Each profile will build and package everything that
> needs to go into the distribution bundle - the runtime, all bundled
> extensions and the actual distribution bundle.
>
> ==========
>
> Proposal 4: We need a test suite for each distribution that mirrors
> user experience
> Rationale: When a user installs a distribution it's helpful if it works.
>
> What this means: Each distribution has a test suite that tests the
> kind of things that a user would do with it (based on what its
> purpose was). The suite works by taking a distribution package,
> installing it, installing test applications and exercising them. We
> run these tests in the kind of environments that users have.
>
> We can't do this every time we build as not everyone will have access
> to every environment. Instead we use a CI framework such as Continuum
> that can build or download a distro and exercise it on a variety of
> platforms. Any failures get converted to high-priority JIRA issues.
>
> This is essentially an enhanced version of the old testing/tomcat
> environment - one that gets run frequently, automatically and in
> multiple environments.
>
> ==========
>
> I realize these are high-level proposals - more a statement of
> principles really - but I wanted to capture the thinking behind the
> proposal. If we think this is the right way to go then we can dig
> into the detail needed to make it happen.
>
> --
> Jeremy
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tuscany-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: tuscany-dev-help@ws.apache.org
>
>