Inflexibility of --also-make w.r.t. unusual goals

[Jesse Glick <je...@oracle.com>]

While working on IDE support for running Maven on multi-module projects [1] I have run up against what seems to be a fundamental limitation of --also-make. This "reactor 
mode" works fine when the specified goal is standard across all lifecycles in the tree, such as 'install' or 'package'.

But frequently you would want to run some special goal which only makes sense on a particular packaging, or otherwise only for certain kinds of modules. Typically this 
would be some sort of "run" goal. As a simple example, in Hudson main sources the 'core' project (using a custom packaging akin to jar) supports just the usual goals such 
as install, while the 'war' project which depends on 'core' has war packaging and supports a hudson-dev:run goal to start Jetty for interactive testing. To test code 
changes, normally you build core, then run war.

Similarly, when developing NetBeans Platform applications, you typically have an aggregator/parent, which contains one project with nbm-application packaging supporting 
the nbm:run-platform goal to start the application in development mode, which depends on a bunch of module projects with nbm packaging supporting the usual 
package/install phase. If you make a change in a module and want to test it interactively, you must first build the module, then build & run the app; the module project 
cannot in general "know" how to start the app together with all its siblings.

These scenarios seem to call out for make mode. The problem is that 'mvn --also-make --projects <main> <goal>' only permits specification of a single goal, which will be 
run in all the projects encountered; 'mvn -am -pl main something:run' dies with an error on the first dependency since it does not use the plugin with 'something' prefix. 
In these examples, what is wanted is to run 'install' (or perhaps 'package') on all the discovered dependencies, followed by something like 'run' on the "main" project.

The only workaround I can find is to run separate chained Maven commands: 'mvn -am -pl main install && mvn -pl main run'. While this works, it is slower and clumsier, and 
produces two sets of output.

I can imagine several ways Maven could support this use case:

1. Some distinguished Maven CLI argument which would run sequential commands, but within the same JVM, sharing cached plugins and project metadata, and producing a single 
summary. E.g.:

   mvn -am -pl main install -then -pl main run

A bit crude, and not very friendly to parallel builds, but at least simple. This of course could be used for other purposes, such as passing -DskipTests only to certain 
subprojects.

2. Some way of telling --also-make to use a different goal for dependencies than for the specified project, e.g.

   mvn --also-make-goals -pl main install run

where for each project in the reactor, only as many goals in the given sequence would be run as are actually available.

3. Some way of declaring in general (say, plugin.xml#//mojo/dependencyGoal) that a certain goal presupposes a different goal be used on reactor dependencies, so that plain

   mvn -am -pl main run

would automatically invoke just 'install' (or 'package') on the calculated dependencies. Note that this could be useful even with common plugins; 'install' on a jar 
project really only needs to 'package' its jar dependencies. (For compatibility reasons, a different switch than --also-make could be introduced to enable the new behavior.)

Perhaps --also-make-dependents could benefit from per-project goal selection as well; offhand I cannot think of a use case.

Are there plans for a more clever make mode? The 3.0 compatibility guide implies that there are, but does not give any specifics.


[1] https://netbeans.org/bugzilla/show_bug.cgi?id=189417


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Inflexibility of --also-make w.r.t. unusual goals

[Jesse Glick <je...@oracle.com>]

On 10/02/2010 07:27 AM, Brett Porter wrote:
>> Are there plans for a more clever make mode? The 3.0 compatibility guide implies that there are, but does not give any specifics.
>
> I'm not sure what you're referring to that gave that indication?

https://cwiki.apache.org/MAVEN/maven-3x-compatibility-notes.html#Maven3.xCompatibilityNotes-LegacystyleReactorModeusingIncludes/Excludes [BTW anchor link in TOC is broken]

"Future development will focus on improvements to the make-like reactor mode that provides similar functionality but is aware of the project hierarchy."


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Inflexibility of --also-make w.r.t. unusual goals

[Jesse Glick <je...@oracle.com>]

On 04/05/2011 12:31 AM, Brett Porter wrote:
>> 2. mvn --also-make-goals -pl submodule test-compile test
>>
>> Does the issue make sense now, and is it worth filing a JIRA ticket for this?
>
> It makes sense.

Filed then: http://jira.codehaus.org/browse/MNG-5059

Not sure how it would be implemented, however. The current list of options is not just a superficial piece of logic in org.apache.maven.cli but seems baked into how 
MavenExecutionRequest is defined.

> I'm not sure how many people would use this - it's quite some entry complexity to gain a little more runtime efficiency.

I was thinking of using a feature like this from an IDE or other scripted environment, in which case a slightly more complex command is no obstacle.

> I've always thought the solution to these type of problems was to invest more effort in incremental builds (don't run tests if no code + dependencies changed).

That would be useful too, but requires work on the part of plugins (here, Surefire).


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Inflexibility of --also-make w.r.t. unusual goals

[Brett Porter <br...@apache.org>]

On 05/04/2011, at 1:16 AM, Jesse Glick wrote:

> There seems to be no decent solution using current --also-make. The solutions using the styles I proposed previously would look like:
> 
> 1. mvn -am -pl submodule test-compile -then -pl submodule test
> 
> 2. mvn --also-make-goals -pl submodule test-compile test
> 
> 3. mvn --also-make-smart -pl submodule test [where test known to presuppose test-compile in dependents; compile would suffice if there were no deps on secondary test artifacts]
> 
> (An IDE user might ask the IDE to just run JUnit on submodule/src/test/java/**/*.java bypassing Maven and Surefire and using some IDE-specific compilation of dependencies from sources, if the IDE has been configured to run tests exactly the same way as the POM specifies Surefire should run them, but it would be best if command-line Maven could do this to begin with.)
> 
> Does the issue make sense now, and is it worth filing a JIRA ticket for this?

It makes sense. Of the solutions, 2 seems clearest, though the test-compile should sit with the --also-make-goals parameter. However, I'm not sure how many people would use this - it's quite some entry complexity to gain a little more runtime efficiency.

I've always thought the solution to these type of problems was to invest more effort in incremental builds (don't run tests if no code + dependencies changed).

- Brett

--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/
http://au.linkedin.com/in/brettporter





---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Inflexibility of --also-make w.r.t. unusual goals

[Jesse Glick <je...@oracle.com>]

On 10/02/2010 07:27 AM, Brett Porter wrote:
>> mvn -am -pl main run
>>
>> would automatically invoke just 'install' (or 'package') on the calculated dependencies.
>
> I don't agree that "install" on one and package on the dependencies would make sense - this would put something in the repository that potentially doesn't
> have its dependencies there, making it unusable by other projects.

You would rather install the dependencies, and run the selected project without its install phase, so the local repository would not be inconsistent. Or if the run goal 
could work with reactor artifacts, just compile or package on the dependencies would suffice, i.e. without anything getting installed in local.

Let me provide a simpler example using only standard jar packaging and its default plugins. If I have a tree of modules using snapshot dependencies internally, and wish 
to run unit tests on one of them while ensuring that the test classpath includes up-to-date class files from its snapshot dependencies, what should I run?

a. 'cd root/submodule; mvn test' will run tests for just submodule, but relies on JARs in the local repository for all the snapshot dependencies, which is no good; these 
JARs might be missing, or old.

b. 'cd root; mvn test' will run the tests for submodule correctly, but also tests in every other module.

c. 'cd root; mvn test -Dtest=sub/module/pkg/**' might run the right set of tests if you happen to have things in a particular package structure, but this is not general. 
Anyway Surefire will fail the build on upstream modules because it found no tests to execute.

d. 'cd root; mvn install -DskipTests -am -pl submodule; cd submodule; mvn test' will run just tests for submodule and with correct snapshot dependencies, but will be much 
slower as it builds JARs (and use of -DskipTests may be objectionable).

e. 'cd root; mvn test-compile -am -pl submodule; cd submodule; mvn test' will not work since the second Maven command is given no reactor, so it will not look up 
dependencies in target/classes but in the local repository.

There seems to be no decent solution using current --also-make. The solutions using the styles I proposed previously would look like:

1. mvn -am -pl submodule test-compile -then -pl submodule test

2. mvn --also-make-goals -pl submodule test-compile test

3. mvn --also-make-smart -pl submodule test [where test known to presuppose test-compile in dependents; compile would suffice if there were no deps on secondary test 
artifacts]

(An IDE user might ask the IDE to just run JUnit on submodule/src/test/java/**/*.java bypassing Maven and Surefire and using some IDE-specific compilation of dependencies 
from sources, if the IDE has been configured to run tests exactly the same way as the POM specifies Surefire should run them, but it would be best if command-line Maven 
could do this to begin with.)

Does the issue make sense now, and is it worth filing a JIRA ticket for this?


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Inflexibility of --also-make w.r.t. unusual goals

[Brett Porter <br...@apache.org>]

On 02/10/2010, at 5:59 AM, Jesse Glick wrote:

> 3. Some way of declaring in general (say, plugin.xml#//mojo/dependencyGoal) that a certain goal presupposes a different goal be used on reactor dependencies, so that plain
> 
>  mvn -am -pl main run
> 
> would automatically invoke just 'install' (or 'package') on the calculated dependencies. Note that this could be useful even with common plugins; 'install' on a jar project really only needs to 'package' its jar dependencies. (For compatibility reasons, a different switch than --also-make could be introduced to enable the new behavior.)

This sounds most similar to the the open issues about "aliasing" certain goal names.

However, I don't agree that "install" on one and package on the dependencies would make sense - this would put something in the repository that potentially doesn't have it's dependencies there, making it unusable by other projects. This probably holds for other similar instances.

The problem you're seeing rather seems to be that jetty:run, and other standalone goals, which ensure "package" is run first, are not able to pass on the --auto-make to the forked lifecycle. That's probably the point it needs to be aligned. I do like the idea of chaining the goals together as a clearer way to run it though (install all first, then run the single goal locally).

> 
> Perhaps --also-make-dependents could benefit from per-project goal selection as well; offhand I cannot think of a use case.
> 
> Are there plans for a more clever make mode? The 3.0 compatibility guide implies that there are, but does not give any specifics.

I'm not sure what you're referring to that gave that indication?

Cheers,
Brett

--
Brett Porter
brett@apache.org
http://brettporter.wordpress.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org