Sling component structure

[Jukka Zitting <ju...@gmail.com>]

Hi,

As far as I've understood, the plan going forward is to have separate
release cycles for all Sling components. Does this mean that each
component would get their own trunk,branches,tags structure, or do we
still keep everything inside a single trunk and perhaps use
subdirectories within the existing branches and tags directories?

How will inter-component dependencies be managed? Currently many
components in the trunk depend on the released 2.0.2 versions, so you
end up with a bit confusing setup where the build generates one
version of one component but downloads another version as a dependency
for another component. Perhaps that's just OK.

Do we have the release management capacity to manage dozens of
separate release cycles? I mean, even if we did just a single release
per year for a component, we'd still be releasing stuff almost weekly.

To simplify things, would it make sense to make the components more
coarse-grained? For example, would it be bad to merge all the commons
components into a single utility component?

Also, do all the components need to be bundles or would it make more
sense for them to be normal jars that bundles that require that
functionality just embed? The way I see it (and I may well be wrong)
is for bundles to be used for coarse grained and loosely coupled
services. I for example don't see why the commons/json component
should be a separate bundle and not an internal dependency of whatever
bundle that requires JSON functionality.

BR,

Jukka Zitting

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi,


Jukka Zitting schrieb:
> Hi,
> 
> On Fri, Jun 27, 2008 at 1:44 PM, Felix Meschberger <fm...@gmail.com> wrote:
>> Yep. And a second one would be to separate them into multiple bundles
>> covering mutiple areas of Sling, probably.
> 
> My call for reducing the number of bundles seems to have failed miserably. ;-)

;-)

Regards
Felix

Re: Sling component structure

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Fri, Jun 27, 2008 at 1:44 PM, Felix Meschberger <fm...@gmail.com> wrote:
> Yep. And a second one would be to separate them into multiple bundles
> covering mutiple areas of Sling, probably.

My call for reducing the number of bundles seems to have failed miserably. ;-)

BR,

Jukka Zitting

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi

Bertrand Delacretaz schrieb:
> On Fri, Jun 27, 2008 at 12:35 PM, Felix Meschberger <fm...@gmail.com> wrote:
> 
>> Bertrand Delacretaz schrieb:
>>> ...Why would having tests create more pain than not having tests?
>> Because of non-production dependencies you include (e.g. JUnit etc)....
> 
> Ok, that can be an issue, agreed.
> 
>> ...how about this approach: We keep the testing and production support
>> separate. We distribute special testing support (bundles). People may wish
>> to install and use the to integration test our and their bundles. If the are
>> satisfied, they remove the testing bundles again, and we are done....
> 
> Agreed, having separate testing bundles looks like the best way to solve this.
> 
> A good first use-case for this would be to extract the integration
> tests from the launchpad/webapp module into a separate testing bundle,
> and in a second step create a simple UI to run those from within
> Sling, without requiring Maven.

Yep. And a second one would be to separate them into multiple bundles 
covering mutiple areas of Sling, probably.

Regards
Felix

Re: Sling component structure

[Bertrand Delacretaz <bd...@apache.org>]

On Fri, Jun 27, 2008 at 12:35 PM, Felix Meschberger <fm...@gmail.com> wrote:

> Bertrand Delacretaz schrieb:
>> ...Why would having tests create more pain than not having tests?
>
> Because of non-production dependencies you include (e.g. JUnit etc)....

Ok, that can be an issue, agreed.

> ...how about this approach: We keep the testing and production support
> separate. We distribute special testing support (bundles). People may wish
> to install and use the to integration test our and their bundles. If the are
> satisfied, they remove the testing bundles again, and we are done....

Agreed, having separate testing bundles looks like the best way to solve this.

A good first use-case for this would be to extract the integration
tests from the launchpad/webapp module into a separate testing bundle,
and in a second step create a simple UI to run those from within
Sling, without requiring Maven.

-Bertrand

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi,

Bertrand Delacretaz schrieb:
> On Fri, Jun 27, 2008 at 12:22 PM, Felix Meschberger <fm...@gmail.com> wrote:
> 
>> Bertrand Delacretaz schrieb:
> 
>>> ...Having a list of testable bundles under /system/self-test, with a user
>>> interface to run the tests, would make it much easier to test various
>>> combinations of bundles....
> 
>> ...While I fully agree with the need for tests and more tests and integration
>> tests etc, I do not have the best of all gut feelings including tests into
>> released binaries....
> 
> Allow me to insist ;-)

Sure ;-)

> 
> Of course this can be dangerous and this is new territory. But the way
> we're working with OSGi is new territory as well, so we need to adjust
> our ways of working.
> 
>> ...Tests belong to development/QA and the source and should not be distributed
>> with the actual binary. At the end of the day, this might create more pain
>> than relieve....
> 
> Why would having tests create more pain than not having tests?

Because of non-production dependencies you include (e.g. JUnit etc).

> 
>> ...What we _must_ really care for is to ensure each bundle release is backwards
>> compatible - at least for the minor and micro release numbers....
> 
> Yes, but how are you going to do that except by testing the various
> combinations of bundles?
> 
> As OSGi allows people to mix and match bundles as they like, I don't
> see another way than allowing people to test their own mix themselves.

Point is, that we programm to the API ! This is key. If we can test our 
own bundles such that we can confidently say, they implement the API 
correctly we are more than halfway done.

Integration testing is important, yes. But if we have good unit tests 
proving the isolated API implementation is correct, integration testing 
is not that important anymore (it used to be in times when there were 
_no_ unit tests).

But how about this approach: We keep the testing and production support 
separate. We distribute special testing support (bundles). People may 
wish to install and use the to integration test our and their bundles. 
If the are satisfied, they remove the testing bundles again, and we are 
done.

WDYT ?

Regards
Felix

> 
> -Bertrand
> 

Re: Sling component structure

[Bertrand Delacretaz <bd...@apache.org>]

On Fri, Jun 27, 2008 at 12:22 PM, Felix Meschberger <fm...@gmail.com> wrote:

> Bertrand Delacretaz schrieb:

>> ...Having a list of testable bundles under /system/self-test, with a user
>> interface to run the tests, would make it much easier to test various
>> combinations of bundles....

> ...While I fully agree with the need for tests and more tests and integration
> tests etc, I do not have the best of all gut feelings including tests into
> released binaries....

Allow me to insist ;-)

Of course this can be dangerous and this is new territory. But the way
we're working with OSGi is new territory as well, so we need to adjust
our ways of working.

> ...Tests belong to development/QA and the source and should not be distributed
> with the actual binary. At the end of the day, this might create more pain
> than relieve....

Why would having tests create more pain than not having tests?

> ...What we _must_ really care for is to ensure each bundle release is backwards
> compatible - at least for the minor and micro release numbers....

Yes, but how are you going to do that except by testing the various
combinations of bundles?

As OSGi allows people to mix and match bundles as they like, I don't
see another way than allowing people to test their own mix themselves.

-Bertrand

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi,

Bertrand Delacretaz schrieb:
> I think there's only one thing missing for this to really work: we
> should have a way of including automated tests in bundles, that can be
> run in the actual Sling instance where the bundle is installed.
> 
> Having a list of testable bundles under /system/self-test, with a user
> interface to run the tests, would make it much easier to test various
> combinations of bundles. Shouldn't be too hard to implement if we
> consider a test suite as an OSGi service.

While I fully agree with the need for tests and more tests and 
integration tests etc, I do not have the best of all gut feelings 
including tests into released binaries.

Tests belong to development/QA and the source and should not be 
distributed with the actual binary. At the end of the day, this might 
create more pain than relieve.

What we _must_ really care for is to ensure each bundle release is 
backwards compatible - at least for the minor and micro release numbers.

We will have to take care of how to handle major release updates, which 
allow for backwards-compatibility breaking.

Regards
Felix

Re: Sling component structure

[Bertrand Delacretaz <bd...@apache.org>]

On Fri, Jun 27, 2008 at 12:27 PM, Felix Meschberger <fm...@gmail.com> wrote:
> Bertrand Delacretaz schrieb:

>>... By default, Sling users are expected to use the "big" Sling releases,...

> ...I am not sure, we should or can say "Sling users are expected to use the
> "big" Sling releases". We can certainly say, first-time users should use....

Agreed, that's what I tried to say with "by default", but first-time
users is more precise.

> ...For more experienced users, updating per bundle is much more convenient. For
> this we should have the OSGi bundle repository (as I already said), which
> enables this simple update....

Agree as well...assuming the resulting combinations of bundles are testable ;-)

-Bertrand

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi,

Bertrand Delacretaz schrieb:
> On Thu, Jun 26, 2008 at 9:32 AM, Jukka Zitting <ju...@gmail.com> wrote:
> 
>> ...As far as I've understood, the plan going forward is to have separate
>> release cycles for all Sling components....
> 
> By default, Sling users are expected to use the "big" Sling releases,
> for example what's under [1] for the current release, which is also
> available as app and webapp binaries.

I am not sure, we should or can say "Sling users are expected to use the 
"big" Sling releases". We can certainly say, first-time users should use.

I envision the big releases to just be convience to the uninitied -- not 
all users are uninitiated ;-)

For more experienced users, updating per bundle is much more convenient. 
For this we should have the OSGi bundle repository (as I already said), 
which enables this simple update.

For example: If you update your Sling instance to the latest Felix 
WebConsole and Felix bundlerepository snapshots you may go to the OSGi 
Repository page and add the repository:

    http://people.apache.org/~fmeschbe/repository.xml

This is a prototype master repository which referrs to a prototype 
Apache Felix repository and the Sling repository. After adding this 
repository you get a list of all bundles provided by these repositories 
which are either not currently installed or which are updates of 
currently installed bundles.

Likewise on the bundles page you should see the "Update" buttons of 
bundles with more recent version(s) in the repository being active.

Regards
Felix

Re: Sling component structure

[Bertrand Delacretaz <bd...@apache.org>]

On Thu, Jun 26, 2008 at 9:32 AM, Jukka Zitting <ju...@gmail.com> wrote:

> ...As far as I've understood, the plan going forward is to have separate
> release cycles for all Sling components....

Here's the way I see it, and I think this matches in a large part
what's been discussed in this thread.

Any Sling bundle can be released independently, when the committers
agree that that makes sense.

Some bundles belong together, and when releasing one it makes sense to
release all others from the same group. I guess these groups will be
invented as we go, and we can use component names in JIRA to keep
track of what's what.

By default, Sling users are expected to use the "big" Sling releases,
for example what's under [1] for the current release, which is also
available as app and webapp binaries.

Mixing and matching bundles released in between "big" releases is ok
if people assume the risks that go with that. Our integration tests
help checking the compatibility of bundles, but people are expected to
add their own tests (or better, contribute them to the project) to
verify this.

We could use the word "global release" for the big releases, and
"bundles release" when releasing individual or groups of bundles.

I think there's only one thing missing for this to really work: we
should have a way of including automated tests in bundles, that can be
run in the actual Sling instance where the bundle is installed.

Having a list of testable bundles under /system/self-test, with a user
interface to run the tests, would make it much easier to test various
combinations of bundles. Shouldn't be too hard to implement if we
consider a test suite as an OSGi service.

-Bertrand

[1] http://svn.apache.org/repos/asf/incubator/sling/tags/sling-3-incubator-source-release/

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi,

Jukka Zitting schrieb:
> Hi,
> 
> On Thu, Jun 26, 2008 at 12:25 PM, Felix Meschberger <fm...@gmail.com> wrote:
>> I do not envision weekly releases of single components. But there may well
>> be one or two releases (or more) per month, yes. We also have this over at
>> Felix and it works quite well.
> 
> OK. I'm just wondering how it looks to a potential end user.
> 
> Looking at the Felix download and news pages, it's quite difficult to
> determine whether an end user needs to care about all that. A release
> is not just an artifact pushed to the Maven repository, but also
> related stuff like release notes and relevant documentation updates.
> Each release should come at least with enough information to tell a
> user whether they should upgrade and what the effects will be if they
> do.

Agreed. Release notes will certainly help. But these may be 
semi-automated, I am sure.

Regarding end-user experience: Our current download pages has exactly 
three downloadables: the standalone app, the web app and the source. We 
might add individually released bundles in the future. But I don't 
think, this is good. Read on ;-)

> 
>> I would only consider merging if it would make sense on a code-wise basis.
>> Merging to "simplify" release management is IMHO wrong and bad.
> 
> How about merging to simplify end user experience?

We have this merging already and these are the apps we distribute today. 
we might add other bundlings in the future in Deployment Packages.

> Unless we can
> somehow automate upgrades (wouldn't it be cool if bundles and all the
> dependencies were automatically downloaded from the Maven
> repository... :-), our users could well end up manually managing a
> complex dependency tree. At least we should document the component
> dependencies somewhere.

Oh yeah, this _is_ possible ! There is the so-called OSGi bundle 
repositories -- OBR for short. The Sling applications come pre-installed 
with support for these repositories by means of the Felix Web Console 
and Felix bundlerepository bundles.

If you go to the bundles list of the web console you might notice the 
generally grayed-out "Update" buttons. If you happen to have an 
appropriate OBR registered (go to the OSGi Repository page) providing 
updates for one or more bundles, you see the button activated. Simply 
clicking on such a button will upgrade the given bundle -- and 
automatically also upgrade any required dependencies.

For now we have the OBR for the first Sling release as a test bed at 
[1]. This will be expanded and be coordinated with similar efforst in 
the Apache Felix project with the ultimate goal of providing OBRs for 
ease of use.

> 
>> In the end, we should make everything a bundle. This enables all use cases,
>> import and embedding. Because, remember: A bundle basically is just a JAR
>> with special manifest headers.
> 
> The bigger difference is how they are presented to the user in a
> running system. For example, on the web console, does it make sense
> that for example commons-collections can be individually removed or
> upgraded? How aware the end user should be of all the inter-component
> dependencies? IMHO, the fewer dependencies the user needs to worry
> about the better.

In the current web console you can see for each bundle, what imports are 
active and what exports and which bundle is actually importing from the 
bundle you are looking at.

So you can quickly decide which bundle will be touched when removing a 
bundle. When upgrading a single bundle, the depending bundles will 
generally quite happily work with the more recent version.

> 
> For example, say I have a component that uses some fancy new Map
> implementation in a recent commons-collections release. When a user
> deploys my component, it would be nice if she didn't need to also
> worry about the commons-collections version. IMHO the same reasoning
> applies also to things like commons/json.

Beware of mixing terms: What is a component here ? Is it a Declarative 
Services Component ? In this case it is deployed as part of a bundle, 
which might have an import to the commons-collections library. if you 
deploy the bundle through OBR and the OBR has the newer library 
available, it will automatically be deployed together with the new 
component bundle.

If the updated commons-collections library is not available the new 
bundle will not resolve -- or is not deployable through OBR due to 
missing dependencies.

Hope this helps.

Regards
Felix

[1] http://people.apache.org/~fmeschbe/sling-repository.xml

Re: Sling component structure

[Carsten Ziegeler <cz...@apache.org>]

Jukka Zitting wrote:
> Hi,
> 
> On Thu, Jun 26, 2008 at 12:25 PM, Felix Meschberger <fm...@gmail.com> wrote:
>> I do not envision weekly releases of single components. But there may well
>> be one or two releases (or more) per month, yes. We also have this over at
>> Felix and it works quite well.
> 
> OK. I'm just wondering how it looks to a potential end user.
> 
Potential end users should use the "full" download, which is either the 
launchpad app or webapp. We will release new versions of the launchpad 
several times a year and these will include the latest bundles (there 
might be exceptions to that rule of course).

> 
> How about merging to simplify end user experience? Unless we can
> somehow automate upgrades (wouldn't it be cool if bundles and all the
> dependencies were automatically downloaded from the Maven
> repository... :-), our users could well end up manually managing a
> complex dependency tree. At least we should document the component
> dependencies somewhere.
> 
> 
> For example, say I have a component that uses some fancy new Map
> implementation in a recent commons-collections release. When a user
> deploys my component, it would be nice if she didn't need to also
> worry about the commons-collections version. IMHO the same reasoning
> applies also to things like commons/json.
> 
OSGi has a nice feature called OBR - this is a repository where we will 
register each and every new bundle release. The user can then invoke the 
webconsole and see which updates are available and update whatever he 
wants. The OBR takes care of dependencies and updates other required 
bundles as well.
Setting up such an OBR repository is easy: it's just an xml document at 
a well-known location which points to the jars in the maven repository.
We're planning to do this very shortly :)

Carsten

-- 
Carsten Ziegeler
cziegeler@apache.org

Re: Sling component structure

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Thu, Jun 26, 2008 at 12:25 PM, Felix Meschberger <fm...@gmail.com> wrote:
> I do not envision weekly releases of single components. But there may well
> be one or two releases (or more) per month, yes. We also have this over at
> Felix and it works quite well.

OK. I'm just wondering how it looks to a potential end user.

Looking at the Felix download and news pages, it's quite difficult to
determine whether an end user needs to care about all that. A release
is not just an artifact pushed to the Maven repository, but also
related stuff like release notes and relevant documentation updates.
Each release should come at least with enough information to tell a
user whether they should upgrade and what the effects will be if they
do.

> I would only consider merging if it would make sense on a code-wise basis.
> Merging to "simplify" release management is IMHO wrong and bad.

How about merging to simplify end user experience? Unless we can
somehow automate upgrades (wouldn't it be cool if bundles and all the
dependencies were automatically downloaded from the Maven
repository... :-), our users could well end up manually managing a
complex dependency tree. At least we should document the component
dependencies somewhere.

> In the end, we should make everything a bundle. This enables all use cases,
> import and embedding. Because, remember: A bundle basically is just a JAR
> with special manifest headers.

The bigger difference is how they are presented to the user in a
running system. For example, on the web console, does it make sense
that for example commons-collections can be individually removed or
upgraded? How aware the end user should be of all the inter-component
dependencies? IMHO, the fewer dependencies the user needs to worry
about the better.

For example, say I have a component that uses some fancy new Map
implementation in a recent commons-collections release. When a user
deploys my component, it would be nice if she didn't need to also
worry about the commons-collections version. IMHO the same reasoning
applies also to things like commons/json.

BR,

Jukka Zitting

Re: Sling component structure

[Felix Meschberger <fm...@gmail.com>]

Hi Jukka,

Thanks for your thoughts on this issue. Please find my comments inline.

Jukka Zitting schrieb:
> Hi,
> 
> As far as I've understood, the plan going forward is to have separate
> release cycles for all Sling components. Does this mean that each
> component would get their own trunk,branches,tags structure, or do we
> still keep everything inside a single trunk and perhaps use
> subdirectories within the existing branches and tags directories?

First off: No Branches here ;-) [ unless really required, of course ]

Second: We have a single trunk. For each release we create a tag in the 
single tags folder we have. If we discover, that we get too many tags in 
the single tag folder, we might start creating subfolders in the tags 
folder (I would say along the lines of the structure in the trunk).

> 
> How will inter-component dependencies be managed? Currently many
> components in the trunk depend on the released 2.0.2 versions, so you
> end up with a bit confusing setup where the build generates one
> version of one component but downloads another version as a dependency
> for another component. Perhaps that's jst OK.

Basically, each component should refer to the minimum version required 
of any other component and it should refer to a released version of said 
component. Of course, if there is a dependency on an unreleased feature 
-- e.g. the jcr/resource project implementing new unreleased API of the 
api project -- a SNAPSHOT dependency is ok until release.

This ensures the maven bundle plugin will create correct version tags 
for the Import-Package headers.

In addition, for all exported packages/classes, we will have to take 
care to be backwards compatible, of course.

Versions of components are _not_ synchronized by intent. Just as we do 
not synchronize with any external component we depend on. This 
synchronization is not possible and not desired. All actual runtime 
dependencies are declared in the Import-Package headers.

> 
> Do we have the release management capacity to manage dozens of
> separate release cycles? I mean, even if we did just a single release
> per year for a component, we'd still be releasing stuff almost weekly.

I do not envision weekly releases of single components. But there may 
well be one or two releases (or more) per month, yes. We also have this 
over at Felix and it works quite well.

The point in this is, that we automate as much as possible and that 
there is not a single release manager. It is rather the case, that 
people requiring a release generally cut the release and conduct the 
complete procedure. We might codify this further by requiring such a 
process to be always lead by a (P)PMC member.

> 
> To simplify things, would it make sense to make the components more
> coarse-grained? For example, would it be bad to merge all the commons
> components into a single utility component?

I would only consider merging if it would make sense on a code-wise 
basis. Merging to "simplify" release management is IMHO wrong and bad. 
Because in the end we might have to cut a release of a component due to 
a single contained part but another part of the component may not be 
ready for release yet and we end up in a real mess.

> 
> Also, do all the components need to be bundles or would it make more
> sense for them to be normal jars that bundles that require that
> functionality just embed? The way I see it (and I may well be wrong)
> is for bundles to be used for coarse grained and loosely coupled
> services. I for example don't see why the commons/json component
> should be a separate bundle and not an internal dependency of whatever
> bundle that requires JSON functionality.

Good point. I think this question cannot be answered once and for all. 
It needs to be decided on a case-by-case basis. After all, it is a 
decision of the bundle creater/maintainer which mechanism to use.

Basically, some bundles may just provide services and other bundles may 
provide library functionality and still other bundles may just provide 
API. As soon as two bundles need to share data, they need to share the 
types of the data. This may not be a big issue for JSON but may be a 
bigger issue for others. YMMV, as always.

In the end, we should make everything a bundle. This enables all use 
cases, import and embedding. Because, remember: A bundle basically is 
just a JAR with special manifest headers.

Hope this helps.

After all, it may probably be a good thing to start a wiki page on these 
issues to come up with some kind of release concept to which people may 
refer in case of uncertainties.

Regards
Felix