You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Mark Struberg <st...@yahoo.de> on 2012/09/01 12:21:56 UTC
the BuildContext
Hi!
While doing further research on the incremental build I came accross the
org.sonatype.plexus.build.incremental.BuildContext.
There are a few things around this class which caught my interrest.
1.) This class has a package org.sonatype, the artifact name groupId org.codehaus.plexus and the project in git is sisu-build-api.
Quite confusing...
2.) This replaced our internal scanning. What wonders me a bit is that the DefaultBuildContext always returns true for hasDelta(). Not sure about the original impl, but thiw might need to get changed if we like to support incremental builds. The problematic code can be found in DefaultMavenResourcesFiltering. Not sure if this got used somewhere else as well.
3.) Having the BuildContext API in a m2eclipse related package is not exactly wise imo. This makes m2eclipse depending on maven and maven depending on m2eclipse.
Wdyt? How to proceed?
LieGrue,
strub
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: the BuildContext
Posted by Anders Hammar <an...@hammar.net>.
1) Yes. :-)
2) I also quite surprisingly noticed this as well while converting a
plugin to use BuildContext and making it m2e compatible. IMO it would
be great if DefailtBuildContext could be improved so that will detect
changes. However, there might be some good reason this hasn't been
implemented yet...
3) Don't follow. How is this m2e related? m2e has a dependency to
BuildContext, but this doesn't make Maven dependent on m2e.
/Anders
On Sat, Sep 1, 2012 at 12:21 PM, Mark Struberg <st...@yahoo.de> wrote:
> Hi!
>
>
> While doing further research on the incremental build I came accross the
> org.sonatype.plexus.build.incremental.BuildContext.
>
> There are a few things around this class which caught my interrest.
>
> 1.) This class has a package org.sonatype, the artifact name groupId org.codehaus.plexus and the project in git is sisu-build-api.
> Quite confusing...
>
> 2.) This replaced our internal scanning. What wonders me a bit is that the DefaultBuildContext always returns true for hasDelta(). Not sure about the original impl, but thiw might need to get changed if we like to support incremental builds. The problematic code can be found in DefaultMavenResourcesFiltering. Not sure if this got used somewhere else as well.
>
> 3.) Having the BuildContext API in a m2eclipse related package is not exactly wise imo. This makes m2eclipse depending on maven and maven depending on m2eclipse.
>
>
> Wdyt? How to proceed?
>
>
>
> LieGrue,
> strub
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: the BuildContext
Posted by Jason van Zyl <ja...@tesla.io>.
None of the code we have, and are using today, is bound to m2e.
On Sep 1, 2012, at 1:40 PM, Robert Scholte wrote:
> Op Sat, 01 Sep 2012 14:40:11 +0200 schreef Jason van Zyl <ja...@tesla.io>:
>
>>
>> On Sep 1, 2012, at 6:21 AM, Mark Struberg wrote:
>>
>>> Hi!
>>>
>>>
>>> While doing further research on the incremental build I came accross the
>>> org.sonatype.plexus.build.incremental.BuildContext.
>>>
>>> There are a few things around this class which caught my interrest.
>>>
>>> 1.) This class has a package org.sonatype, the artifact name groupId org.codehaus.plexus and the project in git is sisu-build-api.
>>> Quite confusing...
>>
>> Hence we haven't really published anything. As you will discover incremental builds, or build avoidance as we've been calling it, when taking into consideration all tools and contexts is hard. The most advanced form of build avoidance exists in m2e has much of the implementation bits required. But obviously we don't want any tools depending on m2e. Like Maven depending on m2e isn't acceptable. The latest of our work is here:
>>
>> https://github.com/etesla/tesla-build-avoidance
>>
>> It works but it's not complete, the API is not where we want it and there are a couple branches there too. We have proven we can make plugins just work in m2e which is a primary goal. The current changes that Igor has experimented with in the core are here:
>>
>> https://github.com/etesla/maven-3/tree/build-context-injection
>>
>> And the m2e work is here:
>>
>> https://github.com/etesla/m2e-core
>>
>> But he doesn't particularly like it. So we don't even like all the ideas or implementations but it's been there for a long time. Initially started by Benjamin, Igor picked up the work and now Igor and I are looking at it again in the context of continuous delivery. But it is confusing because we're still not entirely clear on how we would like it to work but currently we feel using the concept of something like an Eclipse workspace. This idea can work in many tools. I'm not only interested in Maven.
>>
>> I also don't want to go puttering around in the core because I believe that with some very tiny changes experiments can be done in a simple say. One, maybe two injection sites and a custom scope and anything is possible.
>>
>>>
>>> 2.) This replaced our internal scanning. What wonders me a bit is that the DefaultBuildContext always returns true for hasDelta(). Not sure about the original impl, but thiw might need to get changed if we like to support incremental builds. The problematic code can be found in DefaultMavenResourcesFiltering. Not sure if this got used somewhere else as well.
>>>
>>> 3.) Having the BuildContext API in a m2eclipse related package is not exactly wise imo. This makes m2eclipse depending on maven and maven depending on m2eclipse.
>>
>> As Anders pointed there is no dependency on m2e. And if we do take code from m2e, which we will because it's the most advance/proven build avoidance code there is, then we would extract it.
>>
>
> If you consider Maven and m2e as 2 separate products, then I have to agree with Mark that this looks like a cross reference between the two. It would make more sense if Maven would offer such interfaces/API.
>
>>>
>>>
>>> Wdyt? How to proceed?
>>
>> I suggest just looking around the code. Once the JSR-330 branch is merged then it might be easier to play around with. I have no doubt it can be implemented because it has been effectively done in m2e for years. But making it accessible, and making an API that will stand the test of time is what I'm interested in, and I'm also keen on trying to make this available to other systems. By this I don't mean Tesla but other tools like SBT, Gradle, Leiningen, Hudson, and Jenkins because they all have the similar requirements for build avoidance.
>>
>>>
>>>
>>>
>>> LieGrue,
>>> strub
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>> Thanks,
>>
>> Jason
>>
>> ----------------------------------------------------------
>> Jason van Zyl
>> Founder & CTO, Sonatype
>> Founder, Apache Maven
>> http://twitter.com/jvanzyl
>> ---------------------------------------------------------
>>
>> I never make the mistake of arguing with people for whose opinions I have no respect.
>>
>> -- Edward Gibbon
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder & CTO, Sonatype
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------
Re: the BuildContext
Posted by Robert Scholte <rf...@apache.org>.
Op Sat, 01 Sep 2012 14:40:11 +0200 schreef Jason van Zyl <ja...@tesla.io>:
>
> On Sep 1, 2012, at 6:21 AM, Mark Struberg wrote:
>
>> Hi!
>>
>>
>> While doing further research on the incremental build I came accross the
>> org.sonatype.plexus.build.incremental.BuildContext.
>>
>> There are a few things around this class which caught my interrest.
>>
>> 1.) This class has a package org.sonatype, the artifact name groupId
>> org.codehaus.plexus and the project in git is sisu-build-api.
>> Quite confusing...
>
> Hence we haven't really published anything. As you will discover
> incremental builds, or build avoidance as we've been calling it, when
> taking into consideration all tools and contexts is hard. The most
> advanced form of build avoidance exists in m2e has much of the
> implementation bits required. But obviously we don't want any tools
> depending on m2e. Like Maven depending on m2e isn't acceptable. The
> latest of our work is here:
>
> https://github.com/etesla/tesla-build-avoidance
>
> It works but it's not complete, the API is not where we want it and
> there are a couple branches there too. We have proven we can make
> plugins just work in m2e which is a primary goal. The current changes
> that Igor has experimented with in the core are here:
>
> https://github.com/etesla/maven-3/tree/build-context-injection
>
> And the m2e work is here:
>
> https://github.com/etesla/m2e-core
>
> But he doesn't particularly like it. So we don't even like all the ideas
> or implementations but it's been there for a long time. Initially
> started by Benjamin, Igor picked up the work and now Igor and I are
> looking at it again in the context of continuous delivery. But it is
> confusing because we're still not entirely clear on how we would like it
> to work but currently we feel using the concept of something like an
> Eclipse workspace. This idea can work in many tools. I'm not only
> interested in Maven.
>
> I also don't want to go puttering around in the core because I believe
> that with some very tiny changes experiments can be done in a simple
> say. One, maybe two injection sites and a custom scope and anything is
> possible.
>
>>
>> 2.) This replaced our internal scanning. What wonders me a bit is that
>> the DefaultBuildContext always returns true for hasDelta(). Not sure
>> about the original impl, but thiw might need to get changed if we like
>> to support incremental builds. The problematic code can be found in
>> DefaultMavenResourcesFiltering. Not sure if this got used somewhere
>> else as well.
>>
>> 3.) Having the BuildContext API in a m2eclipse related package is not
>> exactly wise imo. This makes m2eclipse depending on maven and maven
>> depending on m2eclipse.
>
> As Anders pointed there is no dependency on m2e. And if we do take code
> from m2e, which we will because it's the most advance/proven build
> avoidance code there is, then we would extract it.
>
If you consider Maven and m2e as 2 separate products, then I have to agree
with Mark that this looks like a cross reference between the two. It would
make more sense if Maven would offer such interfaces/API.
>>
>>
>> Wdyt? How to proceed?
>
> I suggest just looking around the code. Once the JSR-330 branch is
> merged then it might be easier to play around with. I have no doubt it
> can be implemented because it has been effectively done in m2e for
> years. But making it accessible, and making an API that will stand the
> test of time is what I'm interested in, and I'm also keen on trying to
> make this available to other systems. By this I don't mean Tesla but
> other tools like SBT, Gradle, Leiningen, Hudson, and Jenkins because
> they all have the similar requirements for build avoidance.
>
>>
>>
>>
>> LieGrue,
>> strub
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder & CTO, Sonatype
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> ---------------------------------------------------------
>
> I never make the mistake of arguing with people for whose opinions I
> have no respect.
>
> -- Edward Gibbon
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: the BuildContext
Posted by Mark Struberg <st...@yahoo.de>.
> Hence we haven't really published anything.
Well, not quite true. The examples I gave were taken from the released project which is used in maven already.
But you are right, incremental builds is a hard nut and hard to crack for a single person. The API you defined is a starting point, but I fear it only covers 60% of all cases. There is one big fish in it. The set of candidates and algorithms is perfect for the compile step but that's not the whole deal. And even the compile step is broken and just works in m2eclipse because
Eclipse itself does the build upfront as I demonstrated to you
yesterday.
In fact each plugin has it's own set of input files and creates another set of output files. These input files might be on the classpath but also might be somewhere else! Each plugin must determine itself if it should do some work or not. We can just make this easy by supplying a set of utilities for the detection.
I also thought about introducing a marker @incremental (similar to what we have with @threadsafe) to warn people that incremental builds will not work. Fact is that we need to fix or at least verify each single plugin if it behaves well. There are 2 simple rules:
1.) If an input file/data changes, then the plugin must re-create a reliable result
2.) If no input file/data changes, then the plugin must not touch/change any result
> As Anders pointed there is no dependency on m2e.
Technically not I agree, but I do see no reason to have an com.sonatype API in maven core which exists just to notify the m2eclipse plugin. And that is the ONLY reason why it exists. It would be better to have that API in maven (with a simple default impl) and you can exchange that component via simple plexus/sisu/jsr-330 rules.
> And if we do take code from
> m2e, which we will because it's the most advance/proven build avoidance code
> there is, then we would extract it.
As I demonstrated yesterday on IRC even m2e fails to detect this properly. This just works because the Eclipse JDT triggers a recompile upfront. If you turn out this feature in Eclipse (disable 'Project-> Build Automatically') then you will see the bugs noted in the Wiki.
Of course m2e has very valuable knowledge around this area, but it doesn't atm have a solution which works 100%.
LieGrue,
strub
----- Original Message -----
> From: Jason van Zyl <ja...@tesla.io>
> To: Maven Developers List <de...@maven.apache.org>
> Cc:
> Sent: Saturday, September 1, 2012 2:40 PM
> Subject: Re: the BuildContext
>
>
> On Sep 1, 2012, at 6:21 AM, Mark Struberg wrote:
>
>> Hi!
>>
>>
>> While doing further research on the incremental build I came accross the
>> org.sonatype.plexus.build.incremental.BuildContext.
>>
>> There are a few things around this class which caught my interrest.
>>
>> 1.) This class has a package org.sonatype, the artifact name groupId
> org.codehaus.plexus and the project in git is sisu-build-api.
>> Quite confusing...
>
> Hence we haven't really published anything. As you will discover incremental
> builds, or build avoidance as we've been calling it, when taking into
> consideration all tools and contexts is hard. The most advanced form of build
> avoidance exists in m2e has much of the implementation bits required. But
> obviously we don't want any tools depending on m2e. Like Maven depending on
> m2e isn't acceptable. The latest of our work is here:
>
> https://github.com/etesla/tesla-build-avoidance
>
> It works but it's not complete, the API is not where we want it and there
> are a couple branches there too. We have proven we can make plugins just work in
> m2e which is a primary goal. The current changes that Igor has experimented with
> in the core are here:
>
> https://github.com/etesla/maven-3/tree/build-context-injection
>
> And the m2e work is here:
>
> https://github.com/etesla/m2e-core
>
> But he doesn't particularly like it. So we don't even like all the ideas
> or implementations but it's been there for a long time. Initially started by
> Benjamin, Igor picked up the work and now Igor and I are looking at it again in
> the context of continuous delivery. But it is confusing because we're still
> not entirely clear on how we would like it to work but currently we feel using
> the concept of something like an Eclipse workspace. This idea can work in many
> tools. I'm not only interested in Maven.
>
> I also don't want to go puttering around in the core because I believe that
> with some very tiny changes experiments can be done in a simple say. One, maybe
> two injection sites and a custom scope and anything is possible.
>
>>
>> 2.) This replaced our internal scanning. What wonders me a bit is that the
> DefaultBuildContext always returns true for hasDelta(). Not sure about the
> original impl, but thiw might need to get changed if we like to support
> incremental builds. The problematic code can be found in
> DefaultMavenResourcesFiltering. Not sure if this got used somewhere else as
> well.
>>
>> 3.) Having the BuildContext API in a m2eclipse related package is not
> exactly wise imo. This makes m2eclipse depending on maven and maven depending on
> m2eclipse.
>
> As Anders pointed there is no dependency on m2e. And if we do take code from
> m2e, which we will because it's the most advance/proven build avoidance code
> there is, then we would extract it.
>
>>
>>
>> Wdyt? How to proceed?
>
> I suggest just looking around the code. Once the JSR-330 branch is merged then
> it might be easier to play around with. I have no doubt it can be implemented
> because it has been effectively done in m2e for years. But making it accessible,
> and making an API that will stand the test of time is what I'm interested
> in, and I'm also keen on trying to make this available to other systems. By
> this I don't mean Tesla but other tools like SBT, Gradle, Leiningen, Hudson,
> and Jenkins because they all have the similar requirements for build avoidance.
>
>>
>>
>>
>> LieGrue,
>> strub
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> Thanks,
>
> Jason
>
> ----------------------------------------------------------
> Jason van Zyl
> Founder & CTO, Sonatype
> Founder, Apache Maven
> http://twitter.com/jvanzyl
> ---------------------------------------------------------
>
> I never make the mistake of arguing with people for whose opinions I have no
> respect.
>
> -- Edward Gibbon
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: the BuildContext
Posted by Jason van Zyl <ja...@tesla.io>.
On Sep 1, 2012, at 6:21 AM, Mark Struberg wrote:
> Hi!
>
>
> While doing further research on the incremental build I came accross the
> org.sonatype.plexus.build.incremental.BuildContext.
>
> There are a few things around this class which caught my interrest.
>
> 1.) This class has a package org.sonatype, the artifact name groupId org.codehaus.plexus and the project in git is sisu-build-api.
> Quite confusing...
Hence we haven't really published anything. As you will discover incremental builds, or build avoidance as we've been calling it, when taking into consideration all tools and contexts is hard. The most advanced form of build avoidance exists in m2e has much of the implementation bits required. But obviously we don't want any tools depending on m2e. Like Maven depending on m2e isn't acceptable. The latest of our work is here:
https://github.com/etesla/tesla-build-avoidance
It works but it's not complete, the API is not where we want it and there are a couple branches there too. We have proven we can make plugins just work in m2e which is a primary goal. The current changes that Igor has experimented with in the core are here:
https://github.com/etesla/maven-3/tree/build-context-injection
And the m2e work is here:
https://github.com/etesla/m2e-core
But he doesn't particularly like it. So we don't even like all the ideas or implementations but it's been there for a long time. Initially started by Benjamin, Igor picked up the work and now Igor and I are looking at it again in the context of continuous delivery. But it is confusing because we're still not entirely clear on how we would like it to work but currently we feel using the concept of something like an Eclipse workspace. This idea can work in many tools. I'm not only interested in Maven.
I also don't want to go puttering around in the core because I believe that with some very tiny changes experiments can be done in a simple say. One, maybe two injection sites and a custom scope and anything is possible.
>
> 2.) This replaced our internal scanning. What wonders me a bit is that the DefaultBuildContext always returns true for hasDelta(). Not sure about the original impl, but thiw might need to get changed if we like to support incremental builds. The problematic code can be found in DefaultMavenResourcesFiltering. Not sure if this got used somewhere else as well.
>
> 3.) Having the BuildContext API in a m2eclipse related package is not exactly wise imo. This makes m2eclipse depending on maven and maven depending on m2eclipse.
As Anders pointed there is no dependency on m2e. And if we do take code from m2e, which we will because it's the most advance/proven build avoidance code there is, then we would extract it.
>
>
> Wdyt? How to proceed?
I suggest just looking around the code. Once the JSR-330 branch is merged then it might be easier to play around with. I have no doubt it can be implemented because it has been effectively done in m2e for years. But making it accessible, and making an API that will stand the test of time is what I'm interested in, and I'm also keen on trying to make this available to other systems. By this I don't mean Tesla but other tools like SBT, Gradle, Leiningen, Hudson, and Jenkins because they all have the similar requirements for build avoidance.
>
>
>
> LieGrue,
> strub
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder & CTO, Sonatype
Founder, Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------
I never make the mistake of arguing with people for whose opinions I have no respect.
-- Edward Gibbon