Best Practices for Version Convergence

[Nascif Abousalh-Neto <Na...@sas.com>]

One of the features that attracted me to Ivy was the possibility to
switch between dynamic and fixed versions. While dynamic versions can
help you to keep your code always integrated with the latest and
greatest from your dependencies (specially if backed up by a Continuous
Integration server), a static version can protect you from a buggy new
version and allow you some stability to develop a new feature while your
dependency is churning.

But I got worried after a conversation with a friend from a large
company, which told me that they gave up on using static versions
because they would never converge. Complex products typically have many
"diamond-shape" subgraphs embedded in their dependency graphs, which
creates lateral dependencies that are hard if not impossible to catch
locally. Even if you can detect them early by using tools that can
pinpoint the affected areas in the dependency graph, they are still hard
to solve if they cross team boundaries.

For example: let's say we have the following dependency graph, where all
components are supported by different teams:
A -> B
A -> C
A -> D

B -> E
C -> E
D -> E

When using a "strict" conflict manager, A will only be able to resolve
succesfully if B, C and D are using the same versions of E. Getting them
all to agree on using the same version might not be trivial - and
meanwhile A is stuck on its previous working configuration.

I don't like the idea of using other, more flexible conflict managers
because they create configurations that were not validated by the
dependency teams. So if B was published using an old version of E, I
don't think it is a good idea to allow A->B->E with the latest version
of E - it might result in unpredictable behavior in A's runtime.

Using dynamic versions ('latest.*') would prevent this problem but it
denies the advantage of using Ivy to create a stable (if temporary)
environment for development.

Has anybody working on a similar scenario faced this issue? Any advise
or suggestion? 
Thanks,
  Nascif

RE: Best Practices for Version Convergence

[Nascif Abousalh-Neto <Na...@sas.com>]

Thanks for all the replies. I agree that in the end, technology can't
solve what is basically a human communication problem... but some of the
tips were very interesting. I don't know if centralizing the version
decisions by reviewing warnings or using variables will scale to the
size of our organization, but it is worth investigating.

Regards,
  /Nascif 

-----Original Message-----
From: Xavier Hanin [mailto:xavier.hanin@gmail.com] 
Sent: Thursday, August 16, 2007 3:09 AM
To: ivy-user@incubator.apache.org
Subject: Re: Best Practices for Version Convergence

On 8/15/07, Stephane Bailliez <sb...@gmail.com> wrote:
>
> Nascif Abousalh-Neto wrote:
> > One of the features that attracted me to Ivy was the possibility to 
> > switch between dynamic and fixed versions. While dynamic versions 
> > can help you to keep your code always integrated with the latest and

> > greatest from your dependencies (specially if backed up by a 
> > Continuous Integration server), a static version can protect you 
> > from a buggy new version and allow you some stability to develop a 
> > new feature while your dependency is churning.
> >
> > But I got worried after a conversation with a friend from a large 
> > company, which told me that they gave up on using static versions 
> > because they would never converge. Complex products typically have 
> > many "diamond-shape" subgraphs embedded in their dependency graphs, 
> > which creates lateral dependencies that are hard if not impossible 
> > to catch locally. Even if you can detect them early by using tools 
> > that can pinpoint the affected areas in the dependency graph, they 
> > are still hard to solve if they cross team boundaries.
> >
> [...]
> > Has anybody working on a similar scenario faced this issue? Any 
> > advise or suggestion?
> >
> There is not one size fit all answer for this I'm afraid. It depends a

> lot on the team process. They type of products you are building 
> internally and if all these products obey the same type of deadline 
> and focus, team experience, etc...
>
> I have always found that the barrier to any dependency management is 
> not the tools in any way or any technical issue but it is mostly a 
> human process. You will spend a full year of frustration of people not

> listening and continuously doing weird things and complexify the 
> builds and processes in amazing way.
>
> I have experienced that several times and it is _always_ the same 
> story as frustrating as this is. (and large software companies doing 
> IT work are not any better, I have seen very very very very silly 
> things in 150+ developer projects)
>
> It all depends as well how much the team in question has had to 
> product development in comparison to short-term IT-type development. 
> The approach and mindset will be very different between all those
people.
> Sometimes you will have no choice than to let people do the wrong 
> thing until they are knee deep in a terrible mess and willing to 
> accept the obvious.
>
> Human is the most painful part to change management.


I agree with you, stephane.

To go a little bit further about your question, Nascif, I'd say that
there are some practices which may be interesting for you. For example,
you can use a flexible conflict manager like the default one, and add
warnings when a conflict occur, requiring someone to review the
decisions of the conflict manager. If you disagree with one of these
decisions, you can always use the conflict management section of your
ivy file to make a static choice instead of relying on the conflict
manager for this case. This requires some development on a custom
conflict manager, but can be interesting.

An easier way to do something pretty similar is to review Ivy
dependencies report regularly, or at least whenever you release a
product.

You can also use only static revisions but use variables for the actual
revisions to use. Then you can keep control over the versions you use
for a product in a single property file for instance.

That's only some ideas, as Stephane said, it really depends a lot on
your teams and processes.

HTH,

Xavier

-- stephane
>
>


--
Xavier Hanin - Independent Java Consultant http://xhab.blogspot.com/
http://incubator.apache.org/ivy/ http://www.xoocode.org/

Re: Best Practices for Version Convergence

[Xavier Hanin <xa...@gmail.com>]

On 8/15/07, Stephane Bailliez <sb...@gmail.com> wrote:
>
> Nascif Abousalh-Neto wrote:
> > One of the features that attracted me to Ivy was the possibility to
> > switch between dynamic and fixed versions. While dynamic versions can
> > help you to keep your code always integrated with the latest and
> > greatest from your dependencies (specially if backed up by a Continuous
> > Integration server), a static version can protect you from a buggy new
> > version and allow you some stability to develop a new feature while your
> > dependency is churning.
> >
> > But I got worried after a conversation with a friend from a large
> > company, which told me that they gave up on using static versions
> > because they would never converge. Complex products typically have many
> > "diamond-shape" subgraphs embedded in their dependency graphs, which
> > creates lateral dependencies that are hard if not impossible to catch
> > locally. Even if you can detect them early by using tools that can
> > pinpoint the affected areas in the dependency graph, they are still hard
> > to solve if they cross team boundaries.
> >
> [...]
> > Has anybody working on a similar scenario faced this issue? Any advise
> > or suggestion?
> >
> There is not one size fit all answer for this I'm afraid. It depends a
> lot on the team process. They type of products you are building
> internally and if all these products obey the same type of deadline and
> focus, team experience, etc...
>
> I have always found that the barrier to any dependency management is not
> the tools in any way or any technical issue but it is mostly a human
> process. You will spend a full year of frustration of people not
> listening and continuously doing weird things and complexify the builds
> and processes in amazing way.
>
> I have experienced that several times and it is _always_ the same story
> as frustrating as this is. (and large software companies doing IT work
> are not any better, I have seen very very very very silly things in 150+
> developer projects)
>
> It all depends as well how much the team in question has had to product
> development in comparison to short-term IT-type development. The
> approach and mindset will be very different between all those people.
> Sometimes you will have no choice than to let people do the wrong thing
> until they are knee deep in a terrible mess and willing to accept the
> obvious.
>
> Human is the most painful part to change management.


I agree with you, stephane.

To go a little bit further about your question, Nascif, I'd say that there
are some practices which may be interesting for you. For example, you can
use a flexible conflict manager like the default one, and add warnings when
a conflict occur, requiring someone to review the decisions of the conflict
manager. If you disagree with one of these decisions, you can always use the
conflict management section of your ivy file to make a static choice instead
of relying on the conflict manager for this case. This requires some
development on a custom conflict manager, but can be interesting.

An easier way to do something pretty similar is to review Ivy dependencies
report regularly, or at least whenever you release a product.

You can also use only static revisions but use variables for the actual
revisions to use. Then you can keep control over the versions you use for a
product in a single property file for instance.

That's only some ideas, as Stephane said, it really depends a lot on your
teams and processes.

HTH,

Xavier

-- stephane
>
>


-- 
Xavier Hanin - Independent Java Consultant
http://xhab.blogspot.com/
http://incubator.apache.org/ivy/
http://www.xoocode.org/

Re: Best Practices for Version Convergence

[Stephane Bailliez <sb...@gmail.com>]

Nascif Abousalh-Neto wrote:
> One of the features that attracted me to Ivy was the possibility to
> switch between dynamic and fixed versions. While dynamic versions can
> help you to keep your code always integrated with the latest and
> greatest from your dependencies (specially if backed up by a Continuous
> Integration server), a static version can protect you from a buggy new
> version and allow you some stability to develop a new feature while your
> dependency is churning.
>
> But I got worried after a conversation with a friend from a large
> company, which told me that they gave up on using static versions
> because they would never converge. Complex products typically have many
> "diamond-shape" subgraphs embedded in their dependency graphs, which
> creates lateral dependencies that are hard if not impossible to catch
> locally. Even if you can detect them early by using tools that can
> pinpoint the affected areas in the dependency graph, they are still hard
> to solve if they cross team boundaries.
>   
[...]
> Has anybody working on a similar scenario faced this issue? Any advise
> or suggestion? 
>   
There is not one size fit all answer for this I'm afraid. It depends a 
lot on the team process. They type of products you are building 
internally and if all these products obey the same type of deadline and 
focus, team experience, etc...

I have always found that the barrier to any dependency management is not 
the tools in any way or any technical issue but it is mostly a human 
process. You will spend a full year of frustration of people not 
listening and continuously doing weird things and complexify the builds 
and processes in amazing way.

I have experienced that several times and it is _always_ the same story 
as frustrating as this is. (and large software companies doing IT work 
are not any better, I have seen very very very very silly things in 150+ 
developer projects)

It all depends as well how much the team in question has had to product 
development in comparison to short-term IT-type development. The 
approach and mindset will be very different between all those people. 
Sometimes you will have no choice than to let people do the wrong thing 
until they are knee deep in a terrible mess and willing to accept the 
obvious.

Human is the most painful part to change management.

-- stephane