3.0 Compatibility With 2.x [WAS Re: [crypto] Repackage?]

[Robert Burrell Donkin <ro...@gmail.com>]

On Tue, Nov 18, 2008 at 9:19 AM, Stefano Bagnara <ap...@bago.org> wrote:
> Robert Burrell Donkin ha scritto:
>> On Mon, Nov 17, 2008 at 8:35 AM, Stefano Bagnara <ap...@bago.org> wrote:

<snip>

>>> I'm fine with repackaging, but we should remember that this will mean
>>> that we need many <mailetpackages>/<matcherpackages> in our config, and
>>> that we won't provide backward compatibility for config.xml unless we
>>> add some hardcoded hack or we add a compatibility layer with
>>> matchers/mailets in the old places.

AIUI packaging is important for OSGi. splitting packages across jars
is not good. so i think it's worthwhile thinking about repackaging.

we're going to need new mailet loading mechanisms in any case. for
example, the sieve mailet needs an IoC mailet loader (eg spring).

<snip>

>>> I don't know anymore if we are still on a backward compatible config.xml
>>> or if we decided to abandon that goal.
>>
>> IMHO more detail is needed about what a commitment means in detail
>>
>> are we committing to:
>>
>> 1. maintain compatibility with 2.3 or with earlier versions of 3.0?
>> 2. maintain compatibility with custom configurations?
>
> What we tried to commit in past was being able to start james trunk with
> most james 2.3 user's *config.xml*. This meant adding many hacks every
> time we changed the granularity of components in order to keep the old
> conf working, or almost working.

ok

so: we're considering backwards compatibility for the config.xml (not
assembly etc) against the 2.3 user config.xml

AIUI no schema exists for config.xml so there isn't any clear way to
know exactly what this commitment would mean. perhaps we need to
create a clear schema for 2.3.

> IMHO either we keep this and really try to run james 2.3 config.xml or
> we completely give up with phoenix style config.xml and introduce a
> whole new method to configure. Having a similar config method with an
> incompatible "runtime" will make people to try to use their config (even
> if we explain what is compatible and what is not) and we'll spend too
> much time in user support.

i see your point. on the other hand, not offering an upgrade path for
existing users seems a little wrong.

i would really like to be able to load the configuration from more
than one document. i find a single configuration file inconvenient.

> Furthermore IMO we should keep (or provide support for) *bidirectional*
> compatibility for the storage layers (db/dbfile/file for
> mail/spool/users repository). This is a must to let people try the new
> code and eventually revert to their old installation if something does
> not work as expected. As a system administrator I'll think thrice before
> upgrading something that change my data format. We still have people
> trying to use james 2.2.0 !?!?! and it is a PITA to support these people
> (at least for me as I used james 2.2.0 more than 3 years ago the last time).

if this is required for every possible combination of stores then this
would be a big commitment

if one standard store were picked then it might be feasible to offer
bidirectional compatibility just to that

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: 3.0 Compatibility With 2.x [WAS Re: [crypto] Repackage?]

[Robert Burrell Donkin <ro...@gmail.com>]

On Wed, Nov 19, 2008 at 9:10 AM, Stefano Bagnara <ap...@bago.org> wrote:
> Robert Burrell Donkin ha scritto:
>> On Tue, Nov 18, 2008 at 9:19 AM, Stefano Bagnara <ap...@bago.org> wrote:
>>> Robert Burrell Donkin ha scritto:
>>>> On Mon, Nov 17, 2008 at 8:35 AM, Stefano Bagnara <ap...@bago.org> wrote:
>>
>> <snip>
>>
>>>>> I'm fine with repackaging, but we should remember that this will mean
>>>>> that we need many <mailetpackages>/<matcherpackages> in our config, and
>>>>> that we won't provide backward compatibility for config.xml unless we
>>>>> add some hardcoded hack or we add a compatibility layer with
>>>>> matchers/mailets in the old places.
>>
>> AIUI packaging is important for OSGi. splitting packages across jars
>> is not good. so i think it's worthwhile thinking about repackaging.
>>
>> we're going to need new mailet loading mechanisms in any case. for
>> example, the sieve mailet needs an IoC mailet loader (eg spring).
>>
>> <snip>
>>
>>>>> I don't know anymore if we are still on a backward compatible config.xml
>>>>> or if we decided to abandon that goal.
>>>> IMHO more detail is needed about what a commitment means in detail
>>>>
>>>> are we committing to:
>>>>
>>>> 1. maintain compatibility with 2.3 or with earlier versions of 3.0?
>>>> 2. maintain compatibility with custom configurations?
>>> What we tried to commit in past was being able to start james trunk with
>>> most james 2.3 user's *config.xml*. This meant adding many hacks every
>>> time we changed the granularity of components in order to keep the old
>>> conf working, or almost working.
>>
>> ok
>>
>> so: we're considering backwards compatibility for the config.xml (not
>> assembly etc) against the 2.3 user config.xml
>>
>> AIUI no schema exists for config.xml so there isn't any clear way to
>> know exactly what this commitment would mean. perhaps we need to
>> create a clear schema for 2.3.
>
> Every mailet has its own properties and xml tags. So there is no easy
> way to write a schema.

IIRC XML Schema and RELAX support this

> Talking about config.xml compatibility means IMHO having compatibility
> for the default configuration and the variations we propose in comments.
> IMO 99% of users only changes values for already existing tags without
> altering the tag structure of the config.xml (excluding the
> mailet/matcher configuration).

yes

ISTM that there are a few separate issues here:

1. the processor/mailet/matcher configuration which IHMO should be
delegated completely to a loader. i would happily model this with
complete freedom (allow any elements)

2. the main administrative configuration of the main services (the 99%
as above) needs to be fixed and maintained. i think we need to create
and version a schema then deprecate anything that we don't propose to
maintain going forward.

3. the configuration/assembly stuff (eg fetchmail, SMTP handler
chains) which is ATM mostly imported as entities. i would prefer to
see these factored out of the main configuration. each particular
service implementation would then maintain it's own particular
configuration.

>>> IMHO either we keep this and really try to run james 2.3 config.xml or
>>> we completely give up with phoenix style config.xml and introduce a
>>> whole new method to configure. Having a similar config method with an
>>> incompatible "runtime" will make people to try to use their config (even
>>> if we explain what is compatible and what is not) and we'll spend too
>>> much time in user support.
>>
>> i see your point. on the other hand, not offering an upgrade path for
>> existing users seems a little wrong.
>
> Upgrade path is a cost. Having the "upgrade path" as a show stopper for
> the next release could mean decrease probability to have a new release
> anytime. IMO 3.0 new features. If, at that time, users need an upgrade
> path we'll evaluate what kind of upgrade path is needed (automatic tool,
> vs documentation) and maybe release a 3.01/3.1 including the upgrade tools.

milestones have been successful for other projects. (they build
attention and momentum, and allow feedback from the sort of expert
users who are easy to recruit as developers.) my recommendation would
be a road map including a milestone every quarter next year starting
in January. as feature previews, no upgrade paths would be needed
initially.

>> i would really like to be able to load the configuration from more
>> than one document. i find a single configuration file inconvenient.
>
> This is not a technical issue: phoenix can load each component
> configuration from its own file. I don't have a preference on the
> configuration style ATM, but I remember some PMC was against splitting
> the config in multiple files. If you have something in mind please make
> an example so we have something to evaluate and compare pro/cons.

as a james user, having fetchmail (and IMAP) information in
james.config is a PITA.  i would much prefer to store them elsewhere
(for example in a JCR). i would much rather maintain minimal
administrative details in the config (port numbers and so on) with
assembly and other details elsewhere.

as a james developer, insisting that all configuration has to go in
james.config is a PITA. for example, developing and maintaining
complete avalon configuration bindings for activeMQ (and JPA and JCR
and IMAP) is a lot of work. ActiveMQ has good support for spring. i
would much prefer to use spring to configure an ActiveMQ based spool.

so, rather than importing the fetchmail configuration as an XML
entity, i would like to be able to give an url in the james.config and
load the details configuration from there.

so, rather than having to create avalon bindings for ActiveMQ, i would
like to be able to specify some basic admin parameters in the james
configuration and then load and configure the component using spring
based on an assembly configuration outside james.config.

>>> Furthermore IMO we should keep (or provide support for) *bidirectional*
>>> compatibility for the storage layers (db/dbfile/file for
>>> mail/spool/users repository). This is a must to let people try the new
>>> code and eventually revert to their old installation if something does
>>> not work as expected. As a system administrator I'll think thrice before
>>> upgrading something that change my data format. We still have people
>>> trying to use james 2.2.0 !?!?! and it is a PITA to support these people
>>> (at least for me as I used james 2.2.0 more than 3 years ago the last time).
>>
>> if this is required for every possible combination of stores then this
>> would be a big commitment
>
> As long as we have a bridge to current
> MailRepository/SpoolRepository/UsersRepository interface it should work.
> The fact is that not only users but also developers will not upgrade if
> they don't know they can start the old tool if something goes wrong.
> I, for example, won't be able to test a new version in production if
> this requirement is not met. The trunk from some months ago did work
> fine on the same data as 2.3.1 (I don't know what is the current status).

(this makes me think even more than james really needs an OSGi
micro-kernel but do we really need to roll our own...)

i would like to make a clean, clear break by repackaging copies of the
key APIs. the older (james 2.x APIs) could be retained and
undeprecated. particular implementation may choose to implement one or
both.

>> if one standard store were picked then it might be feasible to offer
>> bidirectional compatibility just to that
>
> Unfortunately file,db and dbfile are both used equally as far as I can
> tell supporting users. While I see the cost of supporting backward
> compatibility I don't see really a gain in dropping part of them.

do you mean bi-direction for the storage type (file <-> file, db<->db
but not file<->db) or between each combination of types?

> BTW this discussion belong to when some concrete proposal will be made
> about new storage types (or new interfaces to the storage). There is no
> need to discuss about hypothetical changes. (I remember 3 years ago we
> discussed about a new storage type.. everyone, of course, forgot what we
> discussed as it did not happen soon).

do you mean new storage types following the existing model? (if so,
please explain why discussion is needed.)

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: 3.0 Compatibility With 2.x [WAS Re: [crypto] Repackage?]

[Stefano Bagnara <ap...@bago.org>]

Robert Burrell Donkin ha scritto:
> On Tue, Nov 18, 2008 at 9:19 AM, Stefano Bagnara <ap...@bago.org> wrote:
>> Robert Burrell Donkin ha scritto:
>>> On Mon, Nov 17, 2008 at 8:35 AM, Stefano Bagnara <ap...@bago.org> wrote:
> 
> <snip>
> 
>>>> I'm fine with repackaging, but we should remember that this will mean
>>>> that we need many <mailetpackages>/<matcherpackages> in our config, and
>>>> that we won't provide backward compatibility for config.xml unless we
>>>> add some hardcoded hack or we add a compatibility layer with
>>>> matchers/mailets in the old places.
> 
> AIUI packaging is important for OSGi. splitting packages across jars
> is not good. so i think it's worthwhile thinking about repackaging.
> 
> we're going to need new mailet loading mechanisms in any case. for
> example, the sieve mailet needs an IoC mailet loader (eg spring).
> 
> <snip>
> 
>>>> I don't know anymore if we are still on a backward compatible config.xml
>>>> or if we decided to abandon that goal.
>>> IMHO more detail is needed about what a commitment means in detail
>>>
>>> are we committing to:
>>>
>>> 1. maintain compatibility with 2.3 or with earlier versions of 3.0?
>>> 2. maintain compatibility with custom configurations?
>> What we tried to commit in past was being able to start james trunk with
>> most james 2.3 user's *config.xml*. This meant adding many hacks every
>> time we changed the granularity of components in order to keep the old
>> conf working, or almost working.
> 
> ok
> 
> so: we're considering backwards compatibility for the config.xml (not
> assembly etc) against the 2.3 user config.xml
> 
> AIUI no schema exists for config.xml so there isn't any clear way to
> know exactly what this commitment would mean. perhaps we need to
> create a clear schema for 2.3.

Every mailet has its own properties and xml tags. So there is no easy
way to write a schema.
Talking about config.xml compatibility means IMHO having compatibility
for the default configuration and the variations we propose in comments.
IMO 99% of users only changes values for already existing tags without
altering the tag structure of the config.xml (excluding the
mailet/matcher configuration).

>> IMHO either we keep this and really try to run james 2.3 config.xml or
>> we completely give up with phoenix style config.xml and introduce a
>> whole new method to configure. Having a similar config method with an
>> incompatible "runtime" will make people to try to use their config (even
>> if we explain what is compatible and what is not) and we'll spend too
>> much time in user support.
> 
> i see your point. on the other hand, not offering an upgrade path for
> existing users seems a little wrong.

Upgrade path is a cost. Having the "upgrade path" as a show stopper for
the next release could mean decrease probability to have a new release
anytime. IMO 3.0 new features. If, at that time, users need an upgrade
path we'll evaluate what kind of upgrade path is needed (automatic tool,
vs documentation) and maybe release a 3.01/3.1 including the upgrade tools.

> i would really like to be able to load the configuration from more
> than one document. i find a single configuration file inconvenient.

This is not a technical issue: phoenix can load each component
configuration from its own file. I don't have a preference on the
configuration style ATM, but I remember some PMC was against splitting
the config in multiple files. If you have something in mind please make
an example so we have something to evaluate and compare pro/cons.

>> Furthermore IMO we should keep (or provide support for) *bidirectional*
>> compatibility for the storage layers (db/dbfile/file for
>> mail/spool/users repository). This is a must to let people try the new
>> code and eventually revert to their old installation if something does
>> not work as expected. As a system administrator I'll think thrice before
>> upgrading something that change my data format. We still have people
>> trying to use james 2.2.0 !?!?! and it is a PITA to support these people
>> (at least for me as I used james 2.2.0 more than 3 years ago the last time).
> 
> if this is required for every possible combination of stores then this
> would be a big commitment

As long as we have a bridge to current
MailRepository/SpoolRepository/UsersRepository interface it should work.
The fact is that not only users but also developers will not upgrade if
they don't know they can start the old tool if something goes wrong.
I, for example, won't be able to test a new version in production if
this requirement is not met. The trunk from some months ago did work
fine on the same data as 2.3.1 (I don't know what is the current status).

> if one standard store were picked then it might be feasible to offer
> bidirectional compatibility just to that

Unfortunately file,db and dbfile are both used equally as far as I can
tell supporting users. While I see the cost of supporting backward
compatibility I don't see really a gain in dropping part of them.

BTW this discussion belong to when some concrete proposal will be made
about new storage types (or new interfaces to the storage). There is no
need to discuss about hypothetical changes. (I remember 3 years ago we
discussed about a new storage type.. everyone, of course, forgot what we
discussed as it did not happen soon).

Stefano

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org