Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

We're really close now to being able to ship a James 3 release. It's
frustrating that work's still needed on the final assembly (of the
application from the built server components). It looks likely that
we're going to face ongoing maintenance issues (in this area) until
improved tooling is developed.

ATM the assembly is included with the server modules, and released
with them. Logically, though, assembly is a downstream process. Why
not factor out assembly into a separate product (apache-james, say ;-)
which is a downstream consumer of the server components?

I think that this would allow us to start cutting 3.0.x releases of
the server components immediately. The assembled application would be
released when it's ready without the need to freeze server trunk. I
think this this change would save me a lot of time in the short term,
and (by moving towards continuous delivery) improve quality in the
long term.

Opinions? Objections?

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

On Thu, Jun 30, 2011 at 9:19 AM, Norman Maurer <no...@apache.org> wrote:
> Hi Robert,
>
> I don't get why this should improve things. IMHO its just another thing
> which needs care, we already have to many projects for our current devs ..
> Maybe you can give some more background..

Auditing and maintaining the final application is something that's
going to need care however it's done. (It really takes me too much
time and I'm going to need to start building tools to automate the
process.)

Bundling application assembly within the server components project
introduces a conflict between

* the need to develop these components
* the need to have a fixed target to manually audit and review the
final application assembled

plus a very slow development cycle for work on the assembly descriptor.

It would be much easier and quicker for me to audit and maintain the
legal stuff if I could work on a fixed dependency set (using released
server components) in an independent project isolated from the server
components.

And by introducing this unnecessary coupling, any problem in either
the assembly or the library components means that neither can be
released.

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

On Thu, Jun 30, 2011 at 7:28 PM, Norman Maurer
<no...@googlemail.com> wrote:
> Ok then go ahead.... I just want to get the beta out soon enough

So do I :-)

I'm started work on the assembly but I'm out of time for tonight. Feel
free to disable the assembly stuff in server trunk. As soon as that's
done, you should be able to cut a release for the server components.

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Norman Maurer <no...@googlemail.com>]

Ok then go ahead.... I just want to get the beta out soon enough

Bye
Norman
2011/6/30, Robert Burrell Donkin <ro...@gmail.com>:
> On Thu, Jun 30, 2011 at 1:29 PM, Norman Maurer <no...@apache.org> wrote:
>> Am 30.06.2011 14:27, schrieb Eric Charles:
>>>
>>> On 30/06/11 14:19, Norman Maurer wrote:
>>> (snip)
>>>>>
>>>>> The main issues for me are a slow development cycle caused by
>>>>> unnecessary integration and targeting a moving target. IMO eliminating
>>>>> this bottleneck will save more time than the cost of cutting the
>>>>> additional release. Once we have a smooth pipeline, if this call turns
>>>>> out to be wrong, we can just move the assembly back in.
>>>>>
>>>>> Robert
>>>>
>>>> I'm still not 100 % conviced.. Why not just freeze trunk for a few days
>>>> ?
>>>>
>>>
>>> Or at least freeze pom's dependencies/plugins declarations.
>>
>> Yeah should be good enough..
>
> In general, freezing is just working around a needless bottleneck in
> our release pipeline. The best approach is to fix the bottleneck by
> lengthening the pipe. Taking the application assembly out creates two
> project that will be easy to release from one that's proving
> difficult.
>
> In particular, manual auditing takes me quite a while. If I were fully
> recovered, this might not be such an issue but ATM my productivity is
> very sensitive. Freezing would allow me to audit and create
> appropriate manual legal stuff but this would need to be repeated
> before each release and again when checking the release. At my current
> capacity level, I think this is likely to be a significant drain on my
> resources and slow everyone else down.
>
> Pushing assembly downstream from the server would allow me to check
> and release the assembled application. I'm pretty sure that we'll get
> better throughput this way but if it's not working for us, it would be
> easy enough to move the assembly back as a module of server.
>
> Robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

On Thu, Jun 30, 2011 at 1:29 PM, Norman Maurer <no...@apache.org> wrote:
> Am 30.06.2011 14:27, schrieb Eric Charles:
>>
>> On 30/06/11 14:19, Norman Maurer wrote:
>> (snip)
>>>>
>>>> The main issues for me are a slow development cycle caused by
>>>> unnecessary integration and targeting a moving target. IMO eliminating
>>>> this bottleneck will save more time than the cost of cutting the
>>>> additional release. Once we have a smooth pipeline, if this call turns
>>>> out to be wrong, we can just move the assembly back in.
>>>>
>>>> Robert
>>>
>>> I'm still not 100 % conviced.. Why not just freeze trunk for a few days ?
>>>
>>
>> Or at least freeze pom's dependencies/plugins declarations.
>
> Yeah should be good enough..

In general, freezing is just working around a needless bottleneck in
our release pipeline. The best approach is to fix the bottleneck by
lengthening the pipe. Taking the application assembly out creates two
project that will be easy to release from one that's proving
difficult.

In particular, manual auditing takes me quite a while. If I were fully
recovered, this might not be such an issue but ATM my productivity is
very sensitive. Freezing would allow me to audit and create
appropriate manual legal stuff but this would need to be repeated
before each release and again when checking the release. At my current
capacity level, I think this is likely to be a significant drain on my
resources and slow everyone else down.

Pushing assembly downstream from the server would allow me to check
and release the assembled application. I'm pretty sure that we'll get
better throughput this way but if it's not working for us, it would be
easy enough to move the assembly back as a module of server.

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Norman Maurer <no...@apache.org>]

Am 30.06.2011 14:27, schrieb Eric Charles:
> On 30/06/11 14:19, Norman Maurer wrote:
> (snip)
>>>
>>> The main issues for me are a slow development cycle caused by
>>> unnecessary integration and targeting a moving target. IMO eliminating
>>> this bottleneck will save more time than the cost of cutting the
>>> additional release. Once we have a smooth pipeline, if this call turns
>>> out to be wrong, we can just move the assembly back in.
>>>
>>> Robert
>>
>> I'm still not 100 % conviced.. Why not just freeze trunk for a few 
>> days ?
>>
>
> Or at least freeze pom's dependencies/plugins declarations.

Yeah should be good enough..

>
>> Bye,
>> Norman
>>
>>
>
>

Bye,
Norman




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Eric Charles <er...@apache.org>]

On 30/06/11 14:19, Norman Maurer wrote:
(snip)
>>
>> The main issues for me are a slow development cycle caused by
>> unnecessary integration and targeting a moving target. IMO eliminating
>> this bottleneck will save more time than the cost of cutting the
>> additional release. Once we have a smooth pipeline, if this call turns
>> out to be wrong, we can just move the assembly back in.
>>
>> Robert
>
> I'm still not 100 % conviced.. Why not just freeze trunk for a few days ?
>

Or at least freeze pom's dependencies/plugins declarations.

> Bye,
> Norman
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>


-- 
Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Norman Maurer <no...@apache.org>]

Am 30.06.2011 14:17, schrieb Robert Burrell Donkin:
> On Thu, Jun 30, 2011 at 12:39 PM, Eric Charles<er...@apache.org>  wrote:
>
> <snip>
>
>> I was just wondering, especially when I see how the other Apache projects
>> create there releases, if we could rely for beta2 on a manual check, and
>> report the automated track for next release.
> A fully released automated tool set is unlikely to arrive for a few months.
>
> I would like to see an efficient release pipeline which will allow
> James to be released often.
>
>> Let's see you could split and distributed the remaining jars to be reviewed
>> in 3 or 4 parts, each parts being taken by a committer responsible to add
>> the needed mention in the NOTICE.
> Parallelism is unlikely to save much time at this stage, and might
> slow stuff down. Everyone should really independently audit the
> components in the assembled artifact and ensure that the contents of
> the NOTICE and LICENSE are right.
>
> The main issues for me are a slow development cycle caused by
> unnecessary integration and targeting a moving target. IMO eliminating
> this bottleneck will save more time than the cost of cutting the
> additional release. Once we have a smooth pipeline, if this call turns
> out to be wrong, we can just move the assembly back in.
>
> Robert

I'm still not 100 % conviced.. Why not just freeze trunk for a few days ?

Bye,
Norman




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

On Thu, Jun 30, 2011 at 12:39 PM, Eric Charles <er...@apache.org> wrote:

<snip>

> I was just wondering, especially when I see how the other Apache projects
> create there releases, if we could rely for beta2 on a manual check, and
> report the automated track for next release.

A fully released automated tool set is unlikely to arrive for a few months.

I would like to see an efficient release pipeline which will allow
James to be released often.

> Let's see you could split and distributed the remaining jars to be reviewed
> in 3 or 4 parts, each parts being taken by a committer responsible to add
> the needed mention in the NOTICE.

Parallelism is unlikely to save much time at this stage, and might
slow stuff down. Everyone should really independently audit the
components in the assembled artifact and ensure that the contents of
the NOTICE and LICENSE are right.

The main issues for me are a slow development cycle caused by
unnecessary integration and targeting a moving target. IMO eliminating
this bottleneck will save more time than the cost of cutting the
additional release. Once we have a smooth pipeline, if this call turns
out to be wrong, we can just move the assembly back in.

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Eric Charles <er...@apache.org>]

On 30/06/11 12:46, Robert Burrell Donkin wrote:
> On Thu, Jun 30, 2011 at 9:57 AM, Eric Charles<er...@apache.org>  wrote:
>> Hi,
>>
>> If I understand well, we would ship some jars, but not an archive with the
>> needed startup scripts? I don't expect much users for it...
>
> This change is about separating concerns and reducing conflicts. The
> libraries composing the james server application would be
> independently released to the Maven repository. The actual james
> application would be composed and released independently.
>
>> Also, we are in the process of releasing a beta2, not the final release.
>> Could we be a bit more relax on the NOTICE and plan the maintainable
>> solution based on a tool for the next release?
>
> The James PMC would be knowingly break policy. Legal stuff like this
> is part of oversight. It needs to be right.
>

Sure.
I was just wondering, especially when I see how the other Apache 
projects create there releases, if we could rely for beta2 on a manual 
check, and report the automated track for next release.

Let's see you could split and distributed the remaining jars to be 
reviewed in 3 or 4 parts, each parts being taken by a committer 
responsible to add the needed mention in the NOTICE.

> Robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>


-- 
Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

On Thu, Jun 30, 2011 at 11:50 AM, Norman Maurer <no...@apache.org> wrote:
> Am 30.06.2011 12:46, schrieb Robert Burrell Donkin:

<snip>

>>> Also, we are in the process of releasing a beta2, not the final release.
>>> Could we be a bit more relax on the NOTICE and plan the maintainable
>>> solution based on a tool for the next release?
>>
>> The James PMC would be knowingly break policy. Legal stuff like this
>> is part of oversight. It needs to be right.
>>
>> Robert
>
> Do you think you will have time to "fix" the last things ? I think you know
> the best what todo...

My concern isn't so much finding the time but that the progress
currently is slow especially when trunk is under development. My
preference is to increase the length of the release pipeline to avoid
this bottleneck rather than asking for long freezes on trunk.

If I was able to work on the application assembly in isolation using
jars released to the repository then I think I would be able to
release the assembled end user application with a delay of around a
week (depending on the level of automation achieved) from the upstream
server components release.

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Norman Maurer <no...@apache.org>]

Am 30.06.2011 12:46, schrieb Robert Burrell Donkin:
> On Thu, Jun 30, 2011 at 9:57 AM, Eric Charles<er...@apache.org>  wrote:
>> Hi,
>>
>> If I understand well, we would ship some jars, but not an archive with the
>> needed startup scripts? I don't expect much users for it...
> This change is about separating concerns and reducing conflicts. The
> libraries composing the james server application would be
> independently released to the Maven repository. The actual james
> application would be composed and released independently.

How should this help to seperate this ?


>> Also, we are in the process of releasing a beta2, not the final release.
>> Could we be a bit more relax on the NOTICE and plan the maintainable
>> solution based on a tool for the next release?
> The James PMC would be knowingly break policy. Legal stuff like this
> is part of oversight. It needs to be right.
>
> Robert

Do you think you will have time to "fix" the last things ? I think you 
know the best what todo...

Bye,
Norman



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Robert Burrell Donkin <ro...@gmail.com>]

On Thu, Jun 30, 2011 at 9:57 AM, Eric Charles <er...@apache.org> wrote:
> Hi,
>
> If I understand well, we would ship some jars, but not an archive with the
> needed startup scripts? I don't expect much users for it...

This change is about separating concerns and reducing conflicts. The
libraries composing the james server application would be
independently released to the Maven repository. The actual james
application would be composed and released independently.

> Also, we are in the process of releasing a beta2, not the final release.
> Could we be a bit more relax on the NOTICE and plan the maintainable
> solution based on a tool for the next release?

The James PMC would be knowingly break policy. Legal stuff like this
is part of oversight. It needs to be right.

Robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Eric Charles <er...@apache.org>]

Hi,

If I understand well, we would ship some jars, but not an archive with 
the needed startup scripts? I don't expect much users for it...

Also, we are in the process of releasing a beta2, not the final release. 
Could we be a bit more relax on the NOTICE and plan the maintainable 
solution based on a tool for the next release?

Thx.

On 30/06/11 10:19, Norman Maurer wrote:
> Hi Robert,
>
> I don't get why this should improve things. IMHO its just another thing
> which needs care, we already have to many projects for our current devs ..
> Maybe you can give some more background..
>
>
> Bye,
> Norman
>
> Am 30.06.2011 09:21, schrieb Robert Burrell Donkin:
>> We're really close now to being able to ship a James 3 release. It's
>> frustrating that work's still needed on the final assembly (of the
>> application from the built server components). It looks likely that
>> we're going to face ongoing maintenance issues (in this area) until
>> improved tooling is developed.
>>
>> ATM the assembly is included with the server modules, and released
>> with them. Logically, though, assembly is a downstream process. Why
>> not factor out assembly into a separate product (apache-james, say ;-)
>> which is a downstream consumer of the server components?
>>
>> I think that this would allow us to start cutting 3.0.x releases of
>> the server components immediately. The assembled application would be
>> released when it's ready without the need to freeze server trunk. I
>> think this this change would save me a lot of time in the short term,
>> and (by moving towards continuous delivery) improve quality in the
>> long term.
>>
>> Opinions? Objections?
>>
>> Robert
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>


-- 
Eric

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Factor Out James Application Assembly ...?

[Norman Maurer <no...@apache.org>]

Hi Robert,

I don't get why this should improve things. IMHO its just another thing 
which needs care, we already have to many projects for our current devs ..
Maybe you can give some more background..


Bye,
Norman

Am 30.06.2011 09:21, schrieb Robert Burrell Donkin:
> We're really close now to being able to ship a James 3 release. It's
> frustrating that work's still needed on the final assembly (of the
> application from the built server components). It looks likely that
> we're going to face ongoing maintenance issues (in this area) until
> improved tooling is developed.
>
> ATM the assembly is included with the server modules, and released
> with them. Logically, though, assembly is a downstream process. Why
> not factor out assembly into a separate product (apache-james, say ;-)
> which is a downstream consumer of the server components?
>
> I think that this would allow us to start cutting 3.0.x releases of
> the server components immediately. The assembled application would be
> released when it's ready without the need to freeze server trunk. I
> think this this change would save me a lot of time in the short term,
> and (by moving towards continuous delivery) improve quality in the
> long term.
>
> Opinions? Objections?
>
> Robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org