You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Mark Struberg <st...@yahoo.de> on 2012/09/06 20:52:57 UTC
[incremental build] Detect leftovers from a previous build
Hi!
I had some idea for detecting stale changes in maven which is pretty generic
The problem hits us if you compile BeanA and BeanA2 in a project where BeanA2 is using BeanA.
On a
$> mvn clean compile
you will get both BeanA.class and BeanA2.class in target/classes
Now delete BeanA.java
Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all sources without doing any cleanup and thus BeanA.class will still remain in target/classes.
That is clearly a bug as BeanA2 will be left broken, packaged into a broken jar, ...
How can we avoid that?
Simple answer: A plugin which doesnt support those cases by the underlying took (javac) must always first clean up the stuff it generated during the last invocation.
How can we do that?
step 1: Start a DirectoryScanner and get all files in target/classes. Remember this list!
step 2: Run the compiler
step 3: List all files in target/classes again and remove all files you found in step 1. You will end up with a list of all files generated by the compilation as result.
step 4: Store this list into target/maven-status/maven-compiler-plugin/default/createdfiles.lst ('default' is the plugin execution. We need this in case we have multiple <executions>).
On the next compile you just need to check if you have such a createdfiles.lst file and then first remove all the files listed in it as step 0.
Then you can continue with step 1 from scratch.
We could easily create a utility class for it which keeps the state with methods
public class ChangeDetector /* TODO find better name */
{
File[] readPreviouslyDetectedFileList(File targetFile);
void recordFiles(File baseDir)
File[] detectNewFiles();
storeDetectedFileList(File targetFile)
}
This can e.g. also be used by the maven-resources-plugin to get rid of copied resources which got deleted from src/main/resources.
Do you have a better idea? Any ideas for improving this?
And a big question: how can I get hold of the current <execution> id in a plugin?
LieGrue,
strub
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Ok, not the best but sure
Le 7 sept. 2012 08:52, "Mark Struberg" <st...@yahoo.de> a écrit :
> I guess you missed me. I will not hand-pick which tests to run but run ALL
> tests of a module if a SINGLE file got changed on the input
> classpath/dependencies.
>
> It's all or nothing to be on the safe side.
>
> LieGrue,
> strub
>
>
>
>
> ----- Original Message -----
> > From: Romain Manni-Bucau <rm...@gmail.com>
> > To: Mark Struberg <st...@yahoo.de>; Maven Developers List <
> dev@maven.apache.org>
> > Cc:
> > Sent: Friday, September 7, 2012 8:48 AM
> > Subject: Re: [incremental build] Detect leftovers from a previous build
> >
> > How? If i change a class in main/java but the test is done by reflection
> > youll miss it
> > Le 7 sept. 2012 08:46, "Mark Struberg" <st...@yahoo.de> a
> > écrit :
> >
> >>
> >>
> >> Romain, we can perfectly do that!
> >>
> >> We just need to check whether anything on the input classpath got
> changed
> >> (jars, classes, etc).
> >>
> >> Of course there are cases where you like to manually re-run tests over
> >> again. For those cases we would need to introduce a 'force' flag
> > somehow.
> >>
> >> LieGrue,
> >> strub
> >>
> >>
> >> >________________________________
> >> > From: Romain Manni-Bucau <rm...@gmail.com>
> >> >To: Mark Struberg <st...@yahoo.de>; Maven Developers List <
> >> dev@maven.apache.org>
> >> >Sent: Friday, September 7, 2012 8:41 AM
> >> >Subject: Re: [incremental build] Detect leftovers from a previous
> build
> >> >
> >> >
> >> >You cant do it dor surefire. Tests are sometimes done by reflection
> and
> >> you cant ask a dep tree by test.in real world.
> >> >Le 7 sept. 2012 08:37, "Mark Struberg"
> > <st...@yahoo.de> a écrit :
> >> >
> >> >> I may be missing something, but if all plugins implement this
> > logic,
> >> >>> how it will be different from implicitly doing
> > "clean" during each
> >> >>> build?
> >> >>
> >> >>There is actually a huge difference
> >> >>
> >> >>a.) yes, there are quite a few of such plugins which do not need
> > this.
> >> Think about sql-maven-plugin, openjpa-maven-plugin, etc
> >> >>
> >> >>b.) A plugin will only delete all it's output from a previous
> > invocation
> >> AFTER it detects that it really needs to do something! If there is
> e.g. no
> >> .java file which is newer than it's corresponding .class and no .java
> > file
> >> got added or removed, then the whole compilation doesn't get triggered.
> > Of
> >> course it's currently (without such a helper) hard to detect if a file
> > got
> >> removed from src/...
> >> >>
> >> >>A very simple hack would be to create a DirectoryScanner and store
> > /
> >> later compare the md5 of all ./src files and trigger the clean
> lifecycle if
> >> that got changed. That seems to be the stuff the
> incremental-build-plugin
> >> does.
> >> >>That would be better than nothing, but not sure if it would be
> >> sufficient. I'd at least try to get rid of unnecessary surefire
> > executions.
> >> >>
> >> >>LieGrue,
> >> >>strub
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>----- Original Message -----
> >> >>> From: Igor Fedorenko <ig...@ifedorenko.com>
> >> >>> To: dev@maven.apache.org
> >> >>> Cc:
> >> >>> Sent: Friday, September 7, 2012 4:14 AM
> >> >>> Subject: Re: [incremental build] Detect leftovers from a
> > previous build
> >> >>>
> >> >>> I may be missing something, but if all plugins implement this
> > logic,
> >> >>> how it will be different from implicitly doing
> > "clean" during each
> >> >>> build? Or, put differently, are there plugins that do not need
> > to clean
> >> >>> their previous output to be absolutely sure they properly
> > handle
> >> >>> incremental rebuilds?
> >> >>>
> >> >>> --
> >> >>> Regards,
> >> >>> Igor
> >> >>>
> >> >>> On 12-09-06 2:52 PM, Mark Struberg wrote:
> >> >>>>
> >> >>>>
> >> >>>> Hi!
> >> >>>>
> >> >>>> I had some idea for detecting stale changes in maven
> > which is pretty
> >> >>> generic
> >> >>>>
> >> >>>>
> >> >>>> The problem hits us if you compile BeanA and BeanA2 in a
> > project
> >> where
> >> >>> BeanA2 is using BeanA.
> >> >>>> On a
> >> >>>>
> >> >>>> $> mvn clean compile
> >> >>>>
> >> >>>> you will get both BeanA.class and BeanA2.class in
> > target/classes
> >> >>>> Now delete BeanA.java
> >> >>>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only
> > compiles all
> >> >>> sources without doing any cleanup and thus BeanA.class will
> > still
> >> remain in
> >> >>> target/classes.
> >> >>>>
> >> >>>>
> >> >>>> That is clearly a bug as BeanA2 will be left broken,
> > packaged into a
> >> broken
> >> >>> jar, ...
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> How can we avoid that?
> >> >>>>
> >> >>>> Simple answer: A plugin which doesnt support those cases
> > by the
> >> underlying
> >> >>> took (javac) must always first clean up the stuff it generated
> > during
> >> the last
> >> >>> invocation.
> >> >>>>
> >> >>>> How can we do that?
> >> >>>>
> >> >>>> step 1: Start a DirectoryScanner and get all files in
> > target/classes.
> >> >>> Remember this list!
> >> >>>>
> >> >>>>
> >> >>>> step 2: Run the compiler
> >> >>>>
> >> >>>>
> >> >>>> step 3: List all files in target/classes again and remove
> > all files
> >> you
> >> >>> found in step 1. You will end up with a list of all files
> > generated by
> >> the
> >> >>> compilation as result.
> >> >>>>
> >> >>>> step 4: Store this list into
> >> >>>
> > target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> >> >>> ('default' is the plugin execution. We need this in
> > case we have
> >> >>> multiple <executions>).
> >> >>>>
> >> >>>>
> >> >>>> On the next compile you just need to check if you have
> > such a
> >> >>> createdfiles.lst file and then first remove all the files
> > listed in it
> >> as step
> >> >>> 0.
> >> >>>> Then you can continue with step 1 from scratch.
> >> >>>>
> >> >>>> We could easily create a utility class for it which keeps
> > the state
> >> with
> >> >>> methods
> >> >>>>
> >> >>>> public class ChangeDetector /* TODO find better name */
> >> >>>> {
> >> >>>> File[] readPreviouslyDetectedFileList(File targetFile);
> >> >>>> void recordFiles(File baseDir)
> >> >>>> File[] detectNewFiles();
> >> >>>> storeDetectedFileList(File targetFile)
> >> >>>> }
> >> >>>>
> >> >>>> This can e.g. also be used by the maven-resources-plugin
> > to get rid
> >> of
> >> >>> copied resources which got deleted from src/main/resources.
> >> >>>>
> >> >>>> Do you have a better idea? Any ideas for improving this?
> >> >>>> And a big question: how can I get hold of the current
> > <execution> id
> >> >>> in a plugin?
> >> >>>>
> >> >>>>
> >> >>>> LieGrue,
> >> >>>> strub
> >> >>>>
> >> >>>>
> > ---------------------------------------------------------------------
> >> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> >>>> For additional commands, e-mail:
> > dev-help@maven.apache.org
> >> >>>>
> >> >>>
> >> >>>
> > ---------------------------------------------------------------------
> >> >>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> >>> For additional commands, e-mail: dev-help@maven.apache.org
> >> >>>
> >> >>
> >>
> >>> ---------------------------------------------------------------------
> >> >>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> >>For additional commands, e-mail: dev-help@maven.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
I guess you missed me. I will not hand-pick which tests to run but run ALL tests of a module if a SINGLE file got changed on the input classpath/dependencies.
It's all or nothing to be on the safe side.
LieGrue,
strub
----- Original Message -----
> From: Romain Manni-Bucau <rm...@gmail.com>
> To: Mark Struberg <st...@yahoo.de>; Maven Developers List <de...@maven.apache.org>
> Cc:
> Sent: Friday, September 7, 2012 8:48 AM
> Subject: Re: [incremental build] Detect leftovers from a previous build
>
> How? If i change a class in main/java but the test is done by reflection
> youll miss it
> Le 7 sept. 2012 08:46, "Mark Struberg" <st...@yahoo.de> a
> écrit :
>
>>
>>
>> Romain, we can perfectly do that!
>>
>> We just need to check whether anything on the input classpath got changed
>> (jars, classes, etc).
>>
>> Of course there are cases where you like to manually re-run tests over
>> again. For those cases we would need to introduce a 'force' flag
> somehow.
>>
>> LieGrue,
>> strub
>>
>>
>> >________________________________
>> > From: Romain Manni-Bucau <rm...@gmail.com>
>> >To: Mark Struberg <st...@yahoo.de>; Maven Developers List <
>> dev@maven.apache.org>
>> >Sent: Friday, September 7, 2012 8:41 AM
>> >Subject: Re: [incremental build] Detect leftovers from a previous build
>> >
>> >
>> >You cant do it dor surefire. Tests are sometimes done by reflection and
>> you cant ask a dep tree by test.in real world.
>> >Le 7 sept. 2012 08:37, "Mark Struberg"
> <st...@yahoo.de> a écrit :
>> >
>> >> I may be missing something, but if all plugins implement this
> logic,
>> >>> how it will be different from implicitly doing
> "clean" during each
>> >>> build?
>> >>
>> >>There is actually a huge difference
>> >>
>> >>a.) yes, there are quite a few of such plugins which do not need
> this.
>> Think about sql-maven-plugin, openjpa-maven-plugin, etc
>> >>
>> >>b.) A plugin will only delete all it's output from a previous
> invocation
>> AFTER it detects that it really needs to do something! If there is e.g. no
>> .java file which is newer than it's corresponding .class and no .java
> file
>> got added or removed, then the whole compilation doesn't get triggered.
> Of
>> course it's currently (without such a helper) hard to detect if a file
> got
>> removed from src/...
>> >>
>> >>A very simple hack would be to create a DirectoryScanner and store
> /
>> later compare the md5 of all ./src files and trigger the clean lifecycle if
>> that got changed. That seems to be the stuff the incremental-build-plugin
>> does.
>> >>That would be better than nothing, but not sure if it would be
>> sufficient. I'd at least try to get rid of unnecessary surefire
> executions.
>> >>
>> >>LieGrue,
>> >>strub
>> >>
>> >>
>> >>
>> >>
>> >>----- Original Message -----
>> >>> From: Igor Fedorenko <ig...@ifedorenko.com>
>> >>> To: dev@maven.apache.org
>> >>> Cc:
>> >>> Sent: Friday, September 7, 2012 4:14 AM
>> >>> Subject: Re: [incremental build] Detect leftovers from a
> previous build
>> >>>
>> >>> I may be missing something, but if all plugins implement this
> logic,
>> >>> how it will be different from implicitly doing
> "clean" during each
>> >>> build? Or, put differently, are there plugins that do not need
> to clean
>> >>> their previous output to be absolutely sure they properly
> handle
>> >>> incremental rebuilds?
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Igor
>> >>>
>> >>> On 12-09-06 2:52 PM, Mark Struberg wrote:
>> >>>>
>> >>>>
>> >>>> Hi!
>> >>>>
>> >>>> I had some idea for detecting stale changes in maven
> which is pretty
>> >>> generic
>> >>>>
>> >>>>
>> >>>> The problem hits us if you compile BeanA and BeanA2 in a
> project
>> where
>> >>> BeanA2 is using BeanA.
>> >>>> On a
>> >>>>
>> >>>> $> mvn clean compile
>> >>>>
>> >>>> you will get both BeanA.class and BeanA2.class in
> target/classes
>> >>>> Now delete BeanA.java
>> >>>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only
> compiles all
>> >>> sources without doing any cleanup and thus BeanA.class will
> still
>> remain in
>> >>> target/classes.
>> >>>>
>> >>>>
>> >>>> That is clearly a bug as BeanA2 will be left broken,
> packaged into a
>> broken
>> >>> jar, ...
>> >>>>
>> >>>>
>> >>>>
>> >>>> How can we avoid that?
>> >>>>
>> >>>> Simple answer: A plugin which doesnt support those cases
> by the
>> underlying
>> >>> took (javac) must always first clean up the stuff it generated
> during
>> the last
>> >>> invocation.
>> >>>>
>> >>>> How can we do that?
>> >>>>
>> >>>> step 1: Start a DirectoryScanner and get all files in
> target/classes.
>> >>> Remember this list!
>> >>>>
>> >>>>
>> >>>> step 2: Run the compiler
>> >>>>
>> >>>>
>> >>>> step 3: List all files in target/classes again and remove
> all files
>> you
>> >>> found in step 1. You will end up with a list of all files
> generated by
>> the
>> >>> compilation as result.
>> >>>>
>> >>>> step 4: Store this list into
>> >>>
> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
>> >>> ('default' is the plugin execution. We need this in
> case we have
>> >>> multiple <executions>).
>> >>>>
>> >>>>
>> >>>> On the next compile you just need to check if you have
> such a
>> >>> createdfiles.lst file and then first remove all the files
> listed in it
>> as step
>> >>> 0.
>> >>>> Then you can continue with step 1 from scratch.
>> >>>>
>> >>>> We could easily create a utility class for it which keeps
> the state
>> with
>> >>> methods
>> >>>>
>> >>>> public class ChangeDetector /* TODO find better name */
>> >>>> {
>> >>>> File[] readPreviouslyDetectedFileList(File targetFile);
>> >>>> void recordFiles(File baseDir)
>> >>>> File[] detectNewFiles();
>> >>>> storeDetectedFileList(File targetFile)
>> >>>> }
>> >>>>
>> >>>> This can e.g. also be used by the maven-resources-plugin
> to get rid
>> of
>> >>> copied resources which got deleted from src/main/resources.
>> >>>>
>> >>>> Do you have a better idea? Any ideas for improving this?
>> >>>> And a big question: how can I get hold of the current
> <execution> id
>> >>> in a plugin?
>> >>>>
>> >>>>
>> >>>> LieGrue,
>> >>>> strub
>> >>>>
>> >>>>
> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> >>>> For additional commands, e-mail:
> dev-help@maven.apache.org
>> >>>>
>> >>>
>> >>>
> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> >>> For additional commands, e-mail: dev-help@maven.apache.org
>> >>>
>> >>
>>
>>> ---------------------------------------------------------------------
>> >>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> >>For additional commands, e-mail: dev-help@maven.apache.org
>> >>
>> >>
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Romain Manni-Bucau <rm...@gmail.com>.
How? If i change a class in main/java but the test is done by reflection
youll miss it
Le 7 sept. 2012 08:46, "Mark Struberg" <st...@yahoo.de> a écrit :
>
>
> Romain, we can perfectly do that!
>
> We just need to check whether anything on the input classpath got changed
> (jars, classes, etc).
>
> Of course there are cases where you like to manually re-run tests over
> again. For those cases we would need to introduce a 'force' flag somehow.
>
> LieGrue,
> strub
>
>
> >________________________________
> > From: Romain Manni-Bucau <rm...@gmail.com>
> >To: Mark Struberg <st...@yahoo.de>; Maven Developers List <
> dev@maven.apache.org>
> >Sent: Friday, September 7, 2012 8:41 AM
> >Subject: Re: [incremental build] Detect leftovers from a previous build
> >
> >
> >You cant do it dor surefire. Tests are sometimes done by reflection and
> you cant ask a dep tree by test.in real world.
> >Le 7 sept. 2012 08:37, "Mark Struberg" <st...@yahoo.de> a écrit :
> >
> >> I may be missing something, but if all plugins implement this logic,
> >>> how it will be different from implicitly doing "clean" during each
> >>> build?
> >>
> >>There is actually a huge difference
> >>
> >>a.) yes, there are quite a few of such plugins which do not need this.
> Think about sql-maven-plugin, openjpa-maven-plugin, etc
> >>
> >>b.) A plugin will only delete all it's output from a previous invocation
> AFTER it detects that it really needs to do something! If there is e.g. no
> .java file which is newer than it's corresponding .class and no .java file
> got added or removed, then the whole compilation doesn't get triggered. Of
> course it's currently (without such a helper) hard to detect if a file got
> removed from src/...
> >>
> >>A very simple hack would be to create a DirectoryScanner and store /
> later compare the md5 of all ./src files and trigger the clean lifecycle if
> that got changed. That seems to be the stuff the incremental-build-plugin
> does.
> >>That would be better than nothing, but not sure if it would be
> sufficient. I'd at least try to get rid of unnecessary surefire executions.
> >>
> >>LieGrue,
> >>strub
> >>
> >>
> >>
> >>
> >>----- Original Message -----
> >>> From: Igor Fedorenko <ig...@ifedorenko.com>
> >>> To: dev@maven.apache.org
> >>> Cc:
> >>> Sent: Friday, September 7, 2012 4:14 AM
> >>> Subject: Re: [incremental build] Detect leftovers from a previous build
> >>>
> >>> I may be missing something, but if all plugins implement this logic,
> >>> how it will be different from implicitly doing "clean" during each
> >>> build? Or, put differently, are there plugins that do not need to clean
> >>> their previous output to be absolutely sure they properly handle
> >>> incremental rebuilds?
> >>>
> >>> --
> >>> Regards,
> >>> Igor
> >>>
> >>> On 12-09-06 2:52 PM, Mark Struberg wrote:
> >>>>
> >>>>
> >>>> Hi!
> >>>>
> >>>> I had some idea for detecting stale changes in maven which is pretty
> >>> generic
> >>>>
> >>>>
> >>>> The problem hits us if you compile BeanA and BeanA2 in a project
> where
> >>> BeanA2 is using BeanA.
> >>>> On a
> >>>>
> >>>> $> mvn clean compile
> >>>>
> >>>> you will get both BeanA.class and BeanA2.class in target/classes
> >>>> Now delete BeanA.java
> >>>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> >>> sources without doing any cleanup and thus BeanA.class will still
> remain in
> >>> target/classes.
> >>>>
> >>>>
> >>>> That is clearly a bug as BeanA2 will be left broken, packaged into a
> broken
> >>> jar, ...
> >>>>
> >>>>
> >>>>
> >>>> How can we avoid that?
> >>>>
> >>>> Simple answer: A plugin which doesnt support those cases by the
> underlying
> >>> took (javac) must always first clean up the stuff it generated during
> the last
> >>> invocation.
> >>>>
> >>>> How can we do that?
> >>>>
> >>>> step 1: Start a DirectoryScanner and get all files in target/classes.
> >>> Remember this list!
> >>>>
> >>>>
> >>>> step 2: Run the compiler
> >>>>
> >>>>
> >>>> step 3: List all files in target/classes again and remove all files
> you
> >>> found in step 1. You will end up with a list of all files generated by
> the
> >>> compilation as result.
> >>>>
> >>>> step 4: Store this list into
> >>> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> >>> ('default' is the plugin execution. We need this in case we have
> >>> multiple <executions>).
> >>>>
> >>>>
> >>>> On the next compile you just need to check if you have such a
> >>> createdfiles.lst file and then first remove all the files listed in it
> as step
> >>> 0.
> >>>> Then you can continue with step 1 from scratch.
> >>>>
> >>>> We could easily create a utility class for it which keeps the state
> with
> >>> methods
> >>>>
> >>>> public class ChangeDetector /* TODO find better name */
> >>>> {
> >>>> File[] readPreviouslyDetectedFileList(File targetFile);
> >>>> void recordFiles(File baseDir)
> >>>> File[] detectNewFiles();
> >>>> storeDetectedFileList(File targetFile)
> >>>> }
> >>>>
> >>>> This can e.g. also be used by the maven-resources-plugin to get rid
> of
> >>> copied resources which got deleted from src/main/resources.
> >>>>
> >>>> Do you have a better idea? Any ideas for improving this?
> >>>> And a big question: how can I get hold of the current <execution> id
> >>> in a plugin?
> >>>>
> >>>>
> >>>> LieGrue,
> >>>> strub
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
Romain, we can perfectly do that!
We just need to check whether anything on the input classpath got changed (jars, classes, etc).
Of course there are cases where you like to manually re-run tests over again. For those cases we would need to introduce a 'force' flag somehow.
LieGrue,
strub
>________________________________
> From: Romain Manni-Bucau <rm...@gmail.com>
>To: Mark Struberg <st...@yahoo.de>; Maven Developers List <de...@maven.apache.org>
>Sent: Friday, September 7, 2012 8:41 AM
>Subject: Re: [incremental build] Detect leftovers from a previous build
>
>
>You cant do it dor surefire. Tests are sometimes done by reflection and you cant ask a dep tree by test.in real world.
>Le 7 sept. 2012 08:37, "Mark Struberg" <st...@yahoo.de> a écrit :
>
>> I may be missing something, but if all plugins implement this logic,
>>> how it will be different from implicitly doing "clean" during each
>>> build?
>>
>>There is actually a huge difference
>>
>>a.) yes, there are quite a few of such plugins which do not need this. Think about sql-maven-plugin, openjpa-maven-plugin, etc
>>
>>b.) A plugin will only delete all it's output from a previous invocation AFTER it detects that it really needs to do something! If there is e.g. no .java file which is newer than it's corresponding .class and no .java file got added or removed, then the whole compilation doesn't get triggered. Of course it's currently (without such a helper) hard to detect if a file got removed from src/...
>>
>>A very simple hack would be to create a DirectoryScanner and store / later compare the md5 of all ./src files and trigger the clean lifecycle if that got changed. That seems to be the stuff the incremental-build-plugin does.
>>That would be better than nothing, but not sure if it would be sufficient. I'd at least try to get rid of unnecessary surefire executions.
>>
>>LieGrue,
>>strub
>>
>>
>>
>>
>>----- Original Message -----
>>> From: Igor Fedorenko <ig...@ifedorenko.com>
>>> To: dev@maven.apache.org
>>> Cc:
>>> Sent: Friday, September 7, 2012 4:14 AM
>>> Subject: Re: [incremental build] Detect leftovers from a previous build
>>>
>>> I may be missing something, but if all plugins implement this logic,
>>> how it will be different from implicitly doing "clean" during each
>>> build? Or, put differently, are there plugins that do not need to clean
>>> their previous output to be absolutely sure they properly handle
>>> incremental rebuilds?
>>>
>>> --
>>> Regards,
>>> Igor
>>>
>>> On 12-09-06 2:52 PM, Mark Struberg wrote:
>>>>
>>>>
>>>> Hi!
>>>>
>>>> I had some idea for detecting stale changes in maven which is pretty
>>> generic
>>>>
>>>>
>>>> The problem hits us if you compile BeanA and BeanA2 in a project where
>>> BeanA2 is using BeanA.
>>>> On a
>>>>
>>>> $> mvn clean compile
>>>>
>>>> you will get both BeanA.class and BeanA2.class in target/classes
>>>> Now delete BeanA.java
>>>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
>>> sources without doing any cleanup and thus BeanA.class will still remain in
>>> target/classes.
>>>>
>>>>
>>>> That is clearly a bug as BeanA2 will be left broken, packaged into a broken
>>> jar, ...
>>>>
>>>>
>>>>
>>>> How can we avoid that?
>>>>
>>>> Simple answer: A plugin which doesnt support those cases by the underlying
>>> took (javac) must always first clean up the stuff it generated during the last
>>> invocation.
>>>>
>>>> How can we do that?
>>>>
>>>> step 1: Start a DirectoryScanner and get all files in target/classes.
>>> Remember this list!
>>>>
>>>>
>>>> step 2: Run the compiler
>>>>
>>>>
>>>> step 3: List all files in target/classes again and remove all files you
>>> found in step 1. You will end up with a list of all files generated by the
>>> compilation as result.
>>>>
>>>> step 4: Store this list into
>>> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
>>> ('default' is the plugin execution. We need this in case we have
>>> multiple <executions>).
>>>>
>>>>
>>>> On the next compile you just need to check if you have such a
>>> createdfiles.lst file and then first remove all the files listed in it as step
>>> 0.
>>>> Then you can continue with step 1 from scratch.
>>>>
>>>> We could easily create a utility class for it which keeps the state with
>>> methods
>>>>
>>>> public class ChangeDetector /* TODO find better name */
>>>> {
>>>> File[] readPreviouslyDetectedFileList(File targetFile);
>>>> void recordFiles(File baseDir)
>>>> File[] detectNewFiles();
>>>> storeDetectedFileList(File targetFile)
>>>> }
>>>>
>>>> This can e.g. also be used by the maven-resources-plugin to get rid of
>>> copied resources which got deleted from src/main/resources.
>>>>
>>>> Do you have a better idea? Any ideas for improving this?
>>>> And a big question: how can I get hold of the current <execution> id
>>> in a plugin?
>>>>
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Romain Manni-Bucau <rm...@gmail.com>.
You cant do it dor surefire. Tests are sometimes done by reflection and you
cant ask a dep tree by test.in real world.
Le 7 sept. 2012 08:37, "Mark Struberg" <st...@yahoo.de> a écrit :
> > I may be missing something, but if all plugins implement this logic,
> > how it will be different from implicitly doing "clean" during each
> > build?
>
> There is actually a huge difference
>
> a.) yes, there are quite a few of such plugins which do not need this.
> Think about sql-maven-plugin, openjpa-maven-plugin, etc
>
> b.) A plugin will only delete all it's output from a previous invocation
> AFTER it detects that it really needs to do something! If there is e.g. no
> .java file which is newer than it's corresponding .class and no .java file
> got added or removed, then the whole compilation doesn't get triggered. Of
> course it's currently (without such a helper) hard to detect if a file got
> removed from src/...
>
> A very simple hack would be to create a DirectoryScanner and store / later
> compare the md5 of all ./src files and trigger the clean lifecycle if that
> got changed. That seems to be the stuff the incremental-build-plugin does.
> That would be better than nothing, but not sure if it would be sufficient.
> I'd at least try to get rid of unnecessary surefire executions.
>
> LieGrue,
> strub
>
>
>
>
> ----- Original Message -----
> > From: Igor Fedorenko <ig...@ifedorenko.com>
> > To: dev@maven.apache.org
> > Cc:
> > Sent: Friday, September 7, 2012 4:14 AM
> > Subject: Re: [incremental build] Detect leftovers from a previous build
> >
> > I may be missing something, but if all plugins implement this logic,
> > how it will be different from implicitly doing "clean" during each
> > build? Or, put differently, are there plugins that do not need to clean
> > their previous output to be absolutely sure they properly handle
> > incremental rebuilds?
> >
> > --
> > Regards,
> > Igor
> >
> > On 12-09-06 2:52 PM, Mark Struberg wrote:
> >>
> >>
> >> Hi!
> >>
> >> I had some idea for detecting stale changes in maven which is pretty
> > generic
> >>
> >>
> >> The problem hits us if you compile BeanA and BeanA2 in a project where
> > BeanA2 is using BeanA.
> >> On a
> >>
> >> $> mvn clean compile
> >>
> >> you will get both BeanA.class and BeanA2.class in target/classes
> >> Now delete BeanA.java
> >> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> > sources without doing any cleanup and thus BeanA.class will still remain
> in
> > target/classes.
> >>
> >>
> >> That is clearly a bug as BeanA2 will be left broken, packaged into a
> broken
> > jar, ...
> >>
> >>
> >>
> >> How can we avoid that?
> >>
> >> Simple answer: A plugin which doesnt support those cases by the
> underlying
> > took (javac) must always first clean up the stuff it generated during
> the last
> > invocation.
> >>
> >> How can we do that?
> >>
> >> step 1: Start a DirectoryScanner and get all files in target/classes.
> > Remember this list!
> >>
> >>
> >> step 2: Run the compiler
> >>
> >>
> >> step 3: List all files in target/classes again and remove all files you
> > found in step 1. You will end up with a list of all files generated by
> the
> > compilation as result.
> >>
> >> step 4: Store this list into
> > target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> > ('default' is the plugin execution. We need this in case we have
> > multiple <executions>).
> >>
> >>
> >> On the next compile you just need to check if you have such a
> > createdfiles.lst file and then first remove all the files listed in it
> as step
> > 0.
> >> Then you can continue with step 1 from scratch.
> >>
> >> We could easily create a utility class for it which keeps the state
> with
> > methods
> >>
> >> public class ChangeDetector /* TODO find better name */
> >> {
> >> File[] readPreviouslyDetectedFileList(File targetFile);
> >> void recordFiles(File baseDir)
> >> File[] detectNewFiles();
> >> storeDetectedFileList(File targetFile)
> >> }
> >>
> >> This can e.g. also be used by the maven-resources-plugin to get rid of
> > copied resources which got deleted from src/main/resources.
> >>
> >> Do you have a better idea? Any ideas for improving this?
> >> And a big question: how can I get hold of the current <execution> id
> > in a plugin?
> >>
> >>
> >> LieGrue,
> >> strub
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
> openjpa-maven-plugin is actually an interesting case. openjpa:enhance,
> for example, will make it very hard or impossible for
> maven-compiler-plugin to decide if it needs to rebuild anything or not.
No, not a problem at all. It detects if the classes are already implementing PersistenceCapable and don't touch them in that case.
> This breaks if BeanA and BeanA2 are in in different modules of the same
> reactor build. It really is necessary to track what inputs, i.e. sources
> and external dependencies, contributed to each generated .class to
> guarantee correct output.
Nope, is perfectly working already. Check the latest source from trunk.
If you have them in different modules then you can only have them linked in a single direction anyway. And this direction gets properly detected now in 2.6.
LieGrue,
strub
----- Original Message -----
> From: Igor Fedorenko <ig...@ifedorenko.com>
> To: dev@maven.apache.org
> Cc:
> Sent: Friday, September 7, 2012 11:44 PM
> Subject: Re: [incremental build] Detect leftovers from a previous build
>
>
>
> On 12-09-07 2:37 AM, Mark Struberg wrote:
>>> I may be missing something, but if all plugins implement this
>>> logic, how it will be different from implicitly doing "clean"
>>> during each build?
>>
>> There is actually a huge difference
>>
>> a.) yes, there are quite a few of such plugins which do not need
>> this. Think about sql-maven-plugin, openjpa-maven-plugin, etc
>>
>
> openjpa-maven-plugin is actually an interesting case. openjpa:enhance,
> for example, will make it very hard or impossible for
> maven-compiler-plugin to decide if it needs to rebuild anything or not.
>
>> b.) A plugin will only delete all it's output from a previous
>> invocation AFTER it detects that it really needs to do something! If
>> there is e.g. no .java file which is newer than it's corresponding
>> .class and no .java file got added or removed, then the whole
>> compilation doesn't get triggered. Of course it's currently
> (without
>> such a helper) hard to detect if a file got removed from src/...
>>
>
>
>> A very simple hack would be to create a DirectoryScanner and store /
>> later compare the md5 of all ./src files and trigger the clean
>> lifecycle if that got changed. That seems to be the stuff the
>> incremental-build-plugin does. That would be better than nothing, but
>> not sure if it would be sufficient. I'd at least try to get rid of
>> unnecessary surefire executions.
>>
>
> This breaks if BeanA and BeanA2 are in in different modules of the same
> reactor build. It really is necessary to track what inputs, i.e. sources
> and external dependencies, contributed to each generated .class to
> guarantee correct output.
>
> --
> Regards,
> Igor
>
>> LieGrue, strub >
>>
>>
>>
>> ----- Original Message -----
>>> From: Igor Fedorenko <ig...@ifedorenko.com>
>>> To: dev@maven.apache.org
>>> Cc:
>>> Sent: Friday, September 7, 2012 4:14 AM
>>> Subject: Re: [incremental build] Detect leftovers from a previous build
>>>
>>> I may be missing something, but if all plugins implement this logic,
>>> how it will be different from implicitly doing "clean" during
> each
>>> build? Or, put differently, are there plugins that do not need to clean
>>> their previous output to be absolutely sure they properly handle
>>> incremental rebuilds?
>>>
>>> --
>>> Regards,
>>> Igor
>>>
>>> On 12-09-06 2:52 PM, Mark Struberg wrote:
>>>>
>>>>
>>>> Hi!
>>>>
>>>> I had some idea for detecting stale changes in maven which is
> pretty
>>> generic
>>>>
>>>>
>>>> The problem hits us if you compile BeanA and BeanA2 in a project
> where
>>> BeanA2 is using BeanA.
>>>> On a
>>>>
>>>> $> mvn clean compile
>>>>
>>>> you will get both BeanA.class and BeanA2.class in target/classes
>>>> Now delete BeanA.java
>>>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles
> all
>>> sources without doing any cleanup and thus BeanA.class will still
> remain in
>>> target/classes.
>>>>
>>>>
>>>> That is clearly a bug as BeanA2 will be left broken, packaged
> into a broken
>>> jar, ...
>>>>
>>>>
>>>>
>>>> How can we avoid that?
>>>>
>>>> Simple answer: A plugin which doesnt support those cases by the
> underlying
>>> took (javac) must always first clean up the stuff it generated during
> the last
>>> invocation.
>>>>
>>>> How can we do that?
>>>>
>>>> step 1: Start a DirectoryScanner and get all files in
> target/classes.
>>> Remember this list!
>>>>
>>>>
>>>> step 2: Run the compiler
>>>>
>>>>
>>>> step 3: List all files in target/classes again and remove all
> files you
>>> found in step 1. You will end up with a list of all files generated by
> the
>>> compilation as result.
>>>>
>>>> step 4: Store this list into
>>> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
>>> ('default' is the plugin execution. We need this in case we
> have
>>> multiple <executions>).
>>>>
>>>>
>>>> On the next compile you just need to check if you have such a
>>> createdfiles.lst file and then first remove all the files listed in it
> as step
>>> 0.
>>>> Then you can continue with step 1 from scratch.
>>>>
>>>> We could easily create a utility class for it which keeps the
> state with
>>> methods
>>>>
>>>> public class ChangeDetector /* TODO find better name */
>>>> {
>>>> File[] readPreviouslyDetectedFileList(File targetFile);
>>>> void recordFiles(File baseDir)
>>>> File[] detectNewFiles();
>>>> storeDetectedFileList(File targetFile)
>>>> }
>>>>
>>>> This can e.g. also be used by the maven-resources-plugin to get
> rid of
>>> copied resources which got deleted from src/main/resources.
>>>>
>>>> Do you have a better idea? Any ideas for improving this?
>>>> And a big question: how can I get hold of the current
> <execution> id
>>> in a plugin?
>>>>
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>>
> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
On 12-09-07 2:37 AM, Mark Struberg wrote:
>> I may be missing something, but if all plugins implement this
>> logic, how it will be different from implicitly doing "clean"
>> during each build?
>
> There is actually a huge difference
>
> a.) yes, there are quite a few of such plugins which do not need
> this. Think about sql-maven-plugin, openjpa-maven-plugin, etc
>
openjpa-maven-plugin is actually an interesting case. openjpa:enhance,
for example, will make it very hard or impossible for
maven-compiler-plugin to decide if it needs to rebuild anything or not.
> b.) A plugin will only delete all it's output from a previous
> invocation AFTER it detects that it really needs to do something! If
> there is e.g. no .java file which is newer than it's corresponding
> .class and no .java file got added or removed, then the whole
> compilation doesn't get triggered. Of course it's currently (without
> such a helper) hard to detect if a file got removed from src/...
>
> A very simple hack would be to create a DirectoryScanner and store /
> later compare the md5 of all ./src files and trigger the clean
> lifecycle if that got changed. That seems to be the stuff the
> incremental-build-plugin does. That would be better than nothing, but
> not sure if it would be sufficient. I'd at least try to get rid of
> unnecessary surefire executions.
>
This breaks if BeanA and BeanA2 are in in different modules of the same
reactor build. It really is necessary to track what inputs, i.e. sources
and external dependencies, contributed to each generated .class to
guarantee correct output.
--
Regards,
Igor
> LieGrue, strub >
>
>
>
> ----- Original Message -----
>> From: Igor Fedorenko <ig...@ifedorenko.com>
>> To: dev@maven.apache.org
>> Cc:
>> Sent: Friday, September 7, 2012 4:14 AM
>> Subject: Re: [incremental build] Detect leftovers from a previous build
>>
>> I may be missing something, but if all plugins implement this logic,
>> how it will be different from implicitly doing "clean" during each
>> build? Or, put differently, are there plugins that do not need to clean
>> their previous output to be absolutely sure they properly handle
>> incremental rebuilds?
>>
>> --
>> Regards,
>> Igor
>>
>> On 12-09-06 2:52 PM, Mark Struberg wrote:
>>>
>>>
>>> Hi!
>>>
>>> I had some idea for detecting stale changes in maven which is pretty
>> generic
>>>
>>>
>>> The problem hits us if you compile BeanA and BeanA2 in a project where
>> BeanA2 is using BeanA.
>>> On a
>>>
>>> $> mvn clean compile
>>>
>>> you will get both BeanA.class and BeanA2.class in target/classes
>>> Now delete BeanA.java
>>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
>> sources without doing any cleanup and thus BeanA.class will still remain in
>> target/classes.
>>>
>>>
>>> That is clearly a bug as BeanA2 will be left broken, packaged into a broken
>> jar, ...
>>>
>>>
>>>
>>> How can we avoid that?
>>>
>>> Simple answer: A plugin which doesnt support those cases by the underlying
>> took (javac) must always first clean up the stuff it generated during the last
>> invocation.
>>>
>>> How can we do that?
>>>
>>> step 1: Start a DirectoryScanner and get all files in target/classes.
>> Remember this list!
>>>
>>>
>>> step 2: Run the compiler
>>>
>>>
>>> step 3: List all files in target/classes again and remove all files you
>> found in step 1. You will end up with a list of all files generated by the
>> compilation as result.
>>>
>>> step 4: Store this list into
>> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
>> ('default' is the plugin execution. We need this in case we have
>> multiple <executions>).
>>>
>>>
>>> On the next compile you just need to check if you have such a
>> createdfiles.lst file and then first remove all the files listed in it as step
>> 0.
>>> Then you can continue with step 1 from scratch.
>>>
>>> We could easily create a utility class for it which keeps the state with
>> methods
>>>
>>> public class ChangeDetector /* TODO find better name */
>>> {
>>> File[] readPreviouslyDetectedFileList(File targetFile);
>>> void recordFiles(File baseDir)
>>> File[] detectNewFiles();
>>> storeDetectedFileList(File targetFile)
>>> }
>>>
>>> This can e.g. also be used by the maven-resources-plugin to get rid of
>> copied resources which got deleted from src/main/resources.
>>>
>>> Do you have a better idea? Any ideas for improving this?
>>> And a big question: how can I get hold of the current <execution> id
>> in a plugin?
>>>
>>>
>>> LieGrue,
>>> strub
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
> I may be missing something, but if all plugins implement this logic,
> how it will be different from implicitly doing "clean" during each
> build?
There is actually a huge difference
a.) yes, there are quite a few of such plugins which do not need this. Think about sql-maven-plugin, openjpa-maven-plugin, etc
b.) A plugin will only delete all it's output from a previous invocation AFTER it detects that it really needs to do something! If there is e.g. no .java file which is newer than it's corresponding .class and no .java file got added or removed, then the whole compilation doesn't get triggered. Of course it's currently (without such a helper) hard to detect if a file got removed from src/...
A very simple hack would be to create a DirectoryScanner and store / later compare the md5 of all ./src files and trigger the clean lifecycle if that got changed. That seems to be the stuff the incremental-build-plugin does.
That would be better than nothing, but not sure if it would be sufficient. I'd at least try to get rid of unnecessary surefire executions.
LieGrue,
strub
----- Original Message -----
> From: Igor Fedorenko <ig...@ifedorenko.com>
> To: dev@maven.apache.org
> Cc:
> Sent: Friday, September 7, 2012 4:14 AM
> Subject: Re: [incremental build] Detect leftovers from a previous build
>
> I may be missing something, but if all plugins implement this logic,
> how it will be different from implicitly doing "clean" during each
> build? Or, put differently, are there plugins that do not need to clean
> their previous output to be absolutely sure they properly handle
> incremental rebuilds?
>
> --
> Regards,
> Igor
>
> On 12-09-06 2:52 PM, Mark Struberg wrote:
>>
>>
>> Hi!
>>
>> I had some idea for detecting stale changes in maven which is pretty
> generic
>>
>>
>> The problem hits us if you compile BeanA and BeanA2 in a project where
> BeanA2 is using BeanA.
>> On a
>>
>> $> mvn clean compile
>>
>> you will get both BeanA.class and BeanA2.class in target/classes
>> Now delete BeanA.java
>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> sources without doing any cleanup and thus BeanA.class will still remain in
> target/classes.
>>
>>
>> That is clearly a bug as BeanA2 will be left broken, packaged into a broken
> jar, ...
>>
>>
>>
>> How can we avoid that?
>>
>> Simple answer: A plugin which doesnt support those cases by the underlying
> took (javac) must always first clean up the stuff it generated during the last
> invocation.
>>
>> How can we do that?
>>
>> step 1: Start a DirectoryScanner and get all files in target/classes.
> Remember this list!
>>
>>
>> step 2: Run the compiler
>>
>>
>> step 3: List all files in target/classes again and remove all files you
> found in step 1. You will end up with a list of all files generated by the
> compilation as result.
>>
>> step 4: Store this list into
> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> ('default' is the plugin execution. We need this in case we have
> multiple <executions>).
>>
>>
>> On the next compile you just need to check if you have such a
> createdfiles.lst file and then first remove all the files listed in it as step
> 0.
>> Then you can continue with step 1 from scratch.
>>
>> We could easily create a utility class for it which keeps the state with
> methods
>>
>> public class ChangeDetector /* TODO find better name */
>> {
>> File[] readPreviouslyDetectedFileList(File targetFile);
>> void recordFiles(File baseDir)
>> File[] detectNewFiles();
>> storeDetectedFileList(File targetFile)
>> }
>>
>> This can e.g. also be used by the maven-resources-plugin to get rid of
> copied resources which got deleted from src/main/resources.
>>
>> Do you have a better idea? Any ideas for improving this?
>> And a big question: how can I get hold of the current <execution> id
> in a plugin?
>>
>>
>> LieGrue,
>> strub
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
I may be missing something, but if all plugins implement this logic,
how it will be different from implicitly doing "clean" during each
build? Or, put differently, are there plugins that do not need to clean
their previous output to be absolutely sure they properly handle
incremental rebuilds?
--
Regards,
Igor
On 12-09-06 2:52 PM, Mark Struberg wrote:
>
>
> Hi!
>
> I had some idea for detecting stale changes in maven which is pretty generic
>
>
> The problem hits us if you compile BeanA and BeanA2 in a project where BeanA2 is using BeanA.
> On a
>
> $> mvn clean compile
>
> you will get both BeanA.class and BeanA2.class in target/classes
> Now delete BeanA.java
> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all sources without doing any cleanup and thus BeanA.class will still remain in target/classes.
>
>
> That is clearly a bug as BeanA2 will be left broken, packaged into a broken jar, ...
>
>
>
> How can we avoid that?
>
> Simple answer: A plugin which doesnt support those cases by the underlying took (javac) must always first clean up the stuff it generated during the last invocation.
>
> How can we do that?
>
> step 1: Start a DirectoryScanner and get all files in target/classes. Remember this list!
>
>
> step 2: Run the compiler
>
>
> step 3: List all files in target/classes again and remove all files you found in step 1. You will end up with a list of all files generated by the compilation as result.
>
> step 4: Store this list into target/maven-status/maven-compiler-plugin/default/createdfiles.lst ('default' is the plugin execution. We need this in case we have multiple <executions>).
>
>
> On the next compile you just need to check if you have such a createdfiles.lst file and then first remove all the files listed in it as step 0.
> Then you can continue with step 1 from scratch.
>
> We could easily create a utility class for it which keeps the state with methods
>
> public class ChangeDetector /* TODO find better name */
> {
> File[] readPreviouslyDetectedFileList(File targetFile);
> void recordFiles(File baseDir)
> File[] detectNewFiles();
> storeDetectedFileList(File targetFile)
> }
>
> This can e.g. also be used by the maven-resources-plugin to get rid of copied resources which got deleted from src/main/resources.
>
> Do you have a better idea? Any ideas for improving this?
> And a big question: how can I get hold of the current <execution> id in a plugin?
>
>
> LieGrue,
> strub
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
Hi Kristian!
> How will this help if I delete an old file, one that simply hasnt been
> changed?
This is applicable for all plugins which process all input files, like the compiler plugin does.
In the case you mentioned we will first delete all *.class files created by the previous run (including the class file of your now deleted file) and then do a full recompile which (as this file no longer exists) will not crete this class file again.
> Also, won't the list just grow infinitely ?
No, it will only contain the list of files created by the last plugin run for the very execution-id. Of course this tool is only useful (and shall be only used) for plugins which re-create the output over again if a change got detected.
Again: The downside with all that approach is that all plugins must support incremental builds.
The easy alternative would be to force a clean if any file in src/ got changed.
> re executions; search for "a tribute to Linus Torvalds"
will do.
LieGrue,
strub
----- Original Message -----
> From: Kristian Rosenvold <kr...@zenior.no>
> To: Maven Developers List <de...@maven.apache.org>
> Cc:
> Sent: Friday, September 7, 2012 7:09 AM
> Subject: Re: [incremental build] Detect leftovers from a previous build
>
> How will this help if I delete an old file, one that simply hasnt been
> changed? Also, won't the list just grow infinitely ?
>
> re executions; search for "a tribute to Linus Torvalds" in the
> surefire source to find a really neat workaround that solves several
> of your other problems..
>
> Kristian
>
> Den 6. sep. 2012 kl. 20:53 skrev Mark Struberg <st...@yahoo.de>:
>
>>
>>
>> Hi!
>>
>> I had some idea for detecting stale changes in maven which is pretty
> generic
>>
>>
>> The problem hits us if you compile BeanA and BeanA2 in a project where
> BeanA2 is using BeanA.
>> On a
>>
>> $> mvn clean compile
>>
>> you will get both BeanA.class and BeanA2.class in target/classes
>> Now delete BeanA.java
>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> sources without doing any cleanup and thus BeanA.class will still remain in
> target/classes.
>>
>>
>> That is clearly a bug as BeanA2 will be left broken, packaged into a broken
> jar, ...
>>
>>
>>
>> How can we avoid that?
>>
>> Simple answer: A plugin which doesnt support those cases by the underlying
> took (javac) must always first clean up the stuff it generated during the last
> invocation.
>>
>> How can we do that?
>>
>> step 1: Start a DirectoryScanner and get all files in target/classes.
> Remember this list!
>>
>>
>> step 2: Run the compiler
>>
>>
>> step 3: List all files in target/classes again and remove all files you
> found in step 1. You will end up with a list of all files generated by the
> compilation as result.
>>
>> step 4: Store this list into
> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> ('default' is the plugin execution. We need this in case we have
> multiple <executions>).
>>
>>
>> On the next compile you just need to check if you have such a
> createdfiles.lst file and then first remove all the files listed in it as step
> 0.
>> Then you can continue with step 1 from scratch.
>>
>> We could easily create a utility class for it which keeps the state with
> methods
>>
>> public class ChangeDetector /* TODO find better name */
>> {
>> File[] readPreviouslyDetectedFileList(File targetFile);
>> void recordFiles(File baseDir)
>> File[] detectNewFiles();
>> storeDetectedFileList(File targetFile)
>> }
>>
>> This can e.g. also be used by the maven-resources-plugin to get rid of
> copied resources which got deleted from src/main/resources.
>>
>> Do you have a better idea? Any ideas for improving this?
>> And a big question: how can I get hold of the current <execution> id
> in a plugin?
>>
>>
>> LieGrue,
>> strub
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Kristian Rosenvold <kr...@zenior.no>.
How will this help if I delete an old file, one that simply hasnt been
changed? Also, won't the list just grow infinitely ?
re executions; search for "a tribute to Linus Torvalds" in the
surefire source to find a really neat workaround that solves several
of your other problems..
Kristian
Den 6. sep. 2012 kl. 20:53 skrev Mark Struberg <st...@yahoo.de>:
>
>
> Hi!
>
> I had some idea for detecting stale changes in maven which is pretty generic
>
>
> The problem hits us if you compile BeanA and BeanA2 in a project where BeanA2 is using BeanA.
> On a
>
> $> mvn clean compile
>
> you will get both BeanA.class and BeanA2.class in target/classes
> Now delete BeanA.java
> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all sources without doing any cleanup and thus BeanA.class will still remain in target/classes.
>
>
> That is clearly a bug as BeanA2 will be left broken, packaged into a broken jar, ...
>
>
>
> How can we avoid that?
>
> Simple answer: A plugin which doesnt support those cases by the underlying took (javac) must always first clean up the stuff it generated during the last invocation.
>
> How can we do that?
>
> step 1: Start a DirectoryScanner and get all files in target/classes. Remember this list!
>
>
> step 2: Run the compiler
>
>
> step 3: List all files in target/classes again and remove all files you found in step 1. You will end up with a list of all files generated by the compilation as result.
>
> step 4: Store this list into target/maven-status/maven-compiler-plugin/default/createdfiles.lst ('default' is the plugin execution. We need this in case we have multiple <executions>).
>
>
> On the next compile you just need to check if you have such a createdfiles.lst file and then first remove all the files listed in it as step 0.
> Then you can continue with step 1 from scratch.
>
> We could easily create a utility class for it which keeps the state with methods
>
> public class ChangeDetector /* TODO find better name */
> {
> File[] readPreviouslyDetectedFileList(File targetFile);
> void recordFiles(File baseDir)
> File[] detectNewFiles();
> storeDetectedFileList(File targetFile)
> }
>
> This can e.g. also be used by the maven-resources-plugin to get rid of copied resources which got deleted from src/main/resources.
>
> Do you have a better idea? Any ideas for improving this?
> And a big question: how can I get hold of the current <execution> id in a plugin?
>
>
> LieGrue,
> strub
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
Hi Vincent!
I've now looked at the code and it appears that this is a relatively easy approach. It is still better than the current state.
It's kind of my bullet A. (invoke a 'clean') but extends this idea to the own input sources.
This will work fine as long as you only strictly have default input paths and don't need additional input data.
It will not work if you e.g. have multiple input paths to compile (multiple <execution> entries for the maven-compiler-plugin).
It will also not help you suppressing a test run for projects which didn't get changed (no change in sources + dependencies for that module) - and that is actually the most expensive part of a build.
LieGrue,
strub
----- Original Message -----
> From: Vincent Latombe <vi...@gmail.com>
> To: Maven Developers List <de...@maven.apache.org>
> Cc:
> Sent: Thursday, September 6, 2012 9:37 PM
> Subject: Re: [incremental build] Detect leftovers from a previous build
>
> Hi,
>
> the topic has been raised a long time ago, someone even wrote a plugin to
> try to address this problem.
> see http://maven-incremental-build.java.net/site/index.html
>
> Vincent
>
>
> 2012/9/6 Romain Manni-Bucau <rm...@gmail.com>
>
>> Ok so bad thread but the not stays valid. Triggering a clean is not a
>> solution for me
>> Le 6 sept. 2012 21:05, "Mark Struberg" <st...@yahoo.de>
> a écrit :
>>
>> > Hi Romain,
>> >
>> > Should have prefaced the baseline ;)
>> >
>> > I am now only focusing on plugin internal work inside a single maven
> pom
>> > module. See bullet B. in [1]
>> >
>> > The Maven Reactor code will trigger a 'clean' on the whole
> module if it
>> > detects an external dependency change / pom change / profile change /
>> etc.
>> > See bullet A. [1]
>> >
>> >
>> > We are now really only focusing on the plugins itself as first step
> (aka
>> > B.).
>> >
>> > LieGrue,
>> > strub
>> >
>> >
>> > [1]
> https://cwiki.apache.org/confluence/display/MAVEN/Incremental+Builds
>> >
>> >
>> > >________________________________
>> > > From: Romain Manni-Bucau <rm...@gmail.com>
>> > >To: Mark Struberg <st...@yahoo.de>; Maven Developers List
> <
>> > dev@maven.apache.org>
>> > >Sent: Thursday, September 6, 2012 8:59 PM
>> > >Subject: Re: [incremental build] Detect leftovers from a previous
> build
>> > >
>> > >
>> > >What about browsing the build tree to detect the dep modules which
> needs
>> > to be built (avoid a real clean which can cost really too much to make
>> incr
>> > feature useful)? Can be done in parallel and can be pretty fast
>> > >Le 6 sept. 2012 20:53, "Mark Struberg"
> <st...@yahoo.de> a écrit :
>> > >
>> > >
>> > >>
>> > >>Hi!
>> > >>
>> > >>I had some idea for detecting stale changes in maven which is
> pretty
>> > generic
>> > >>
>> > >>
>> > >>The problem hits us if you compile BeanA and BeanA2 in a
> project where
>> > BeanA2 is using BeanA.
>> > >>On a
>> > >>
>> > >>$> mvn clean compile
>> > >>
>> > >>you will get both BeanA.class and BeanA2.class in
> target/classes
>> > >>Now delete BeanA.java
>> > >>Currently (2.6-SNAPSHOT) the maven-compiler-plugin only
> compiles all
>> > sources without doing any cleanup and thus BeanA.class will still
> remain
>> in
>> > target/classes.
>> > >>
>> > >>
>> > >>That is clearly a bug as BeanA2 will be left broken, packaged
> into a
>> > broken jar, ...
>> > >>
>> > >>
>> > >>
>> > >>How can we avoid that?
>> > >>
>> > >>Simple answer: A plugin which doesnt support those cases by
> the
>> > underlying took (javac) must always first clean up the stuff it
> generated
>> > during the last invocation.
>> > >>
>> > >>How can we do that?
>> > >>
>> > >>step 1: Start a DirectoryScanner and get all files in
> target/classes.
>> > Remember this list!
>> > >>
>> > >>
>> > >>step 2: Run the compiler
>> > >>
>> > >>
>> > >>step 3: List all files in target/classes again and remove all
> files you
>> > found in step 1. You will end up with a list of all files generated by
>> the
>> > compilation as result.
>> > >>
>> > >>step 4: Store this list into
>> > target/maven-status/maven-compiler-plugin/default/createdfiles.lst
>> > ('default' is the plugin execution. We need this in case we
> have multiple
>> > <executions>).
>> > >>
>> > >>
>> > >>On the next compile you just need to check if you have such a
>> > createdfiles.lst file and then first remove all the files listed in it
> as
>> > step 0.
>> > >>Then you can continue with step 1 from scratch.
>> > >>
>> > >>We could easily create a utility class for it which keeps the
> state
>> with
>> > methods
>> > >>
>> > >>public class ChangeDetector /* TODO find better name */
>> > >>{
>> > >>File[] readPreviouslyDetectedFileList(File targetFile);
>> > >>void recordFiles(File baseDir)
>> > >>File[] detectNewFiles();
>> > >>storeDetectedFileList(File targetFile)
>> > >>}
>> > >>
>> > >>This can e.g. also be used by the maven-resources-plugin to
> get rid of
>> > copied resources which got deleted from src/main/resources.
>> > >>
>> > >>Do you have a better idea? Any ideas for improving this?
>> > >>And a big question: how can I get hold of the current
> <execution> id in
>> > a plugin?
>> > >>
>> > >>
>> > >>LieGrue,
>> > >>strub
>> > >>
>> >
>>> ---------------------------------------------------------------------
>> > >>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > >>For additional commands, e-mail: dev-help@maven.apache.org
>> > >>
>> > >>
>> > >
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > For additional commands, e-mail: dev-help@maven.apache.org
>> >
>> >
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Vincent Latombe <vi...@gmail.com>.
Hi,
the topic has been raised a long time ago, someone even wrote a plugin to
try to address this problem.
see http://maven-incremental-build.java.net/site/index.html
Vincent
2012/9/6 Romain Manni-Bucau <rm...@gmail.com>
> Ok so bad thread but the not stays valid. Triggering a clean is not a
> solution for me
> Le 6 sept. 2012 21:05, "Mark Struberg" <st...@yahoo.de> a écrit :
>
> > Hi Romain,
> >
> > Should have prefaced the baseline ;)
> >
> > I am now only focusing on plugin internal work inside a single maven pom
> > module. See bullet B. in [1]
> >
> > The Maven Reactor code will trigger a 'clean' on the whole module if it
> > detects an external dependency change / pom change / profile change /
> etc.
> > See bullet A. [1]
> >
> >
> > We are now really only focusing on the plugins itself as first step (aka
> > B.).
> >
> > LieGrue,
> > strub
> >
> >
> > [1] https://cwiki.apache.org/confluence/display/MAVEN/Incremental+Builds
> >
> >
> > >________________________________
> > > From: Romain Manni-Bucau <rm...@gmail.com>
> > >To: Mark Struberg <st...@yahoo.de>; Maven Developers List <
> > dev@maven.apache.org>
> > >Sent: Thursday, September 6, 2012 8:59 PM
> > >Subject: Re: [incremental build] Detect leftovers from a previous build
> > >
> > >
> > >What about browsing the build tree to detect the dep modules which needs
> > to be built (avoid a real clean which can cost really too much to make
> incr
> > feature useful)? Can be done in parallel and can be pretty fast
> > >Le 6 sept. 2012 20:53, "Mark Struberg" <st...@yahoo.de> a écrit :
> > >
> > >
> > >>
> > >>Hi!
> > >>
> > >>I had some idea for detecting stale changes in maven which is pretty
> > generic
> > >>
> > >>
> > >>The problem hits us if you compile BeanA and BeanA2 in a project where
> > BeanA2 is using BeanA.
> > >>On a
> > >>
> > >>$> mvn clean compile
> > >>
> > >>you will get both BeanA.class and BeanA2.class in target/classes
> > >>Now delete BeanA.java
> > >>Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> > sources without doing any cleanup and thus BeanA.class will still remain
> in
> > target/classes.
> > >>
> > >>
> > >>That is clearly a bug as BeanA2 will be left broken, packaged into a
> > broken jar, ...
> > >>
> > >>
> > >>
> > >>How can we avoid that?
> > >>
> > >>Simple answer: A plugin which doesnt support those cases by the
> > underlying took (javac) must always first clean up the stuff it generated
> > during the last invocation.
> > >>
> > >>How can we do that?
> > >>
> > >>step 1: Start a DirectoryScanner and get all files in target/classes.
> > Remember this list!
> > >>
> > >>
> > >>step 2: Run the compiler
> > >>
> > >>
> > >>step 3: List all files in target/classes again and remove all files you
> > found in step 1. You will end up with a list of all files generated by
> the
> > compilation as result.
> > >>
> > >>step 4: Store this list into
> > target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> > ('default' is the plugin execution. We need this in case we have multiple
> > <executions>).
> > >>
> > >>
> > >>On the next compile you just need to check if you have such a
> > createdfiles.lst file and then first remove all the files listed in it as
> > step 0.
> > >>Then you can continue with step 1 from scratch.
> > >>
> > >>We could easily create a utility class for it which keeps the state
> with
> > methods
> > >>
> > >>public class ChangeDetector /* TODO find better name */
> > >>{
> > >>File[] readPreviouslyDetectedFileList(File targetFile);
> > >>void recordFiles(File baseDir)
> > >>File[] detectNewFiles();
> > >>storeDetectedFileList(File targetFile)
> > >>}
> > >>
> > >>This can e.g. also be used by the maven-resources-plugin to get rid of
> > copied resources which got deleted from src/main/resources.
> > >>
> > >>Do you have a better idea? Any ideas for improving this?
> > >>And a big question: how can I get hold of the current <execution> id in
> > a plugin?
> > >>
> > >>
> > >>LieGrue,
> > >>strub
> > >>
> > >>---------------------------------------------------------------------
> > >>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>For additional commands, e-mail: dev-help@maven.apache.org
> > >>
> > >>
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
Re: [incremental build] Detect leftovers from a previous build
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Ok so bad thread but the not stays valid. Triggering a clean is not a
solution for me
Le 6 sept. 2012 21:05, "Mark Struberg" <st...@yahoo.de> a écrit :
> Hi Romain,
>
> Should have prefaced the baseline ;)
>
> I am now only focusing on plugin internal work inside a single maven pom
> module. See bullet B. in [1]
>
> The Maven Reactor code will trigger a 'clean' on the whole module if it
> detects an external dependency change / pom change / profile change / etc.
> See bullet A. [1]
>
>
> We are now really only focusing on the plugins itself as first step (aka
> B.).
>
> LieGrue,
> strub
>
>
> [1] https://cwiki.apache.org/confluence/display/MAVEN/Incremental+Builds
>
>
> >________________________________
> > From: Romain Manni-Bucau <rm...@gmail.com>
> >To: Mark Struberg <st...@yahoo.de>; Maven Developers List <
> dev@maven.apache.org>
> >Sent: Thursday, September 6, 2012 8:59 PM
> >Subject: Re: [incremental build] Detect leftovers from a previous build
> >
> >
> >What about browsing the build tree to detect the dep modules which needs
> to be built (avoid a real clean which can cost really too much to make incr
> feature useful)? Can be done in parallel and can be pretty fast
> >Le 6 sept. 2012 20:53, "Mark Struberg" <st...@yahoo.de> a écrit :
> >
> >
> >>
> >>Hi!
> >>
> >>I had some idea for detecting stale changes in maven which is pretty
> generic
> >>
> >>
> >>The problem hits us if you compile BeanA and BeanA2 in a project where
> BeanA2 is using BeanA.
> >>On a
> >>
> >>$> mvn clean compile
> >>
> >>you will get both BeanA.class and BeanA2.class in target/classes
> >>Now delete BeanA.java
> >>Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> sources without doing any cleanup and thus BeanA.class will still remain in
> target/classes.
> >>
> >>
> >>That is clearly a bug as BeanA2 will be left broken, packaged into a
> broken jar, ...
> >>
> >>
> >>
> >>How can we avoid that?
> >>
> >>Simple answer: A plugin which doesnt support those cases by the
> underlying took (javac) must always first clean up the stuff it generated
> during the last invocation.
> >>
> >>How can we do that?
> >>
> >>step 1: Start a DirectoryScanner and get all files in target/classes.
> Remember this list!
> >>
> >>
> >>step 2: Run the compiler
> >>
> >>
> >>step 3: List all files in target/classes again and remove all files you
> found in step 1. You will end up with a list of all files generated by the
> compilation as result.
> >>
> >>step 4: Store this list into
> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> ('default' is the plugin execution. We need this in case we have multiple
> <executions>).
> >>
> >>
> >>On the next compile you just need to check if you have such a
> createdfiles.lst file and then first remove all the files listed in it as
> step 0.
> >>Then you can continue with step 1 from scratch.
> >>
> >>We could easily create a utility class for it which keeps the state with
> methods
> >>
> >>public class ChangeDetector /* TODO find better name */
> >>{
> >>File[] readPreviouslyDetectedFileList(File targetFile);
> >>void recordFiles(File baseDir)
> >>File[] detectNewFiles();
> >>storeDetectedFileList(File targetFile)
> >>}
> >>
> >>This can e.g. also be used by the maven-resources-plugin to get rid of
> copied resources which got deleted from src/main/resources.
> >>
> >>Do you have a better idea? Any ideas for improving this?
> >>And a big question: how can I get hold of the current <execution> id in
> a plugin?
> >>
> >>
> >>LieGrue,
> >>strub
> >>
> >>---------------------------------------------------------------------
> >>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
Hi Romain,
Should have prefaced the baseline ;)
I am now only focusing on plugin internal work inside a single maven pom module. See bullet B. in [1]
The Maven Reactor code will trigger a 'clean' on the whole module if it detects an external dependency change / pom change / profile change / etc. See bullet A. [1]
We are now really only focusing on the plugins itself as first step (aka B.).
LieGrue,
strub
[1] https://cwiki.apache.org/confluence/display/MAVEN/Incremental+Builds
>________________________________
> From: Romain Manni-Bucau <rm...@gmail.com>
>To: Mark Struberg <st...@yahoo.de>; Maven Developers List <de...@maven.apache.org>
>Sent: Thursday, September 6, 2012 8:59 PM
>Subject: Re: [incremental build] Detect leftovers from a previous build
>
>
>What about browsing the build tree to detect the dep modules which needs to be built (avoid a real clean which can cost really too much to make incr feature useful)? Can be done in parallel and can be pretty fast
>Le 6 sept. 2012 20:53, "Mark Struberg" <st...@yahoo.de> a écrit :
>
>
>>
>>Hi!
>>
>>I had some idea for detecting stale changes in maven which is pretty generic
>>
>>
>>The problem hits us if you compile BeanA and BeanA2 in a project where BeanA2 is using BeanA.
>>On a
>>
>>$> mvn clean compile
>>
>>you will get both BeanA.class and BeanA2.class in target/classes
>>Now delete BeanA.java
>>Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all sources without doing any cleanup and thus BeanA.class will still remain in target/classes.
>>
>>
>>That is clearly a bug as BeanA2 will be left broken, packaged into a broken jar, ...
>>
>>
>>
>>How can we avoid that?
>>
>>Simple answer: A plugin which doesnt support those cases by the underlying took (javac) must always first clean up the stuff it generated during the last invocation.
>>
>>How can we do that?
>>
>>step 1: Start a DirectoryScanner and get all files in target/classes. Remember this list!
>>
>>
>>step 2: Run the compiler
>>
>>
>>step 3: List all files in target/classes again and remove all files you found in step 1. You will end up with a list of all files generated by the compilation as result.
>>
>>step 4: Store this list into target/maven-status/maven-compiler-plugin/default/createdfiles.lst ('default' is the plugin execution. We need this in case we have multiple <executions>).
>>
>>
>>On the next compile you just need to check if you have such a createdfiles.lst file and then first remove all the files listed in it as step 0.
>>Then you can continue with step 1 from scratch.
>>
>>We could easily create a utility class for it which keeps the state with methods
>>
>>public class ChangeDetector /* TODO find better name */
>>{
>>File[] readPreviouslyDetectedFileList(File targetFile);
>>void recordFiles(File baseDir)
>>File[] detectNewFiles();
>>storeDetectedFileList(File targetFile)
>>}
>>
>>This can e.g. also be used by the maven-resources-plugin to get rid of copied resources which got deleted from src/main/resources.
>>
>>Do you have a better idea? Any ideas for improving this?
>>And a big question: how can I get hold of the current <execution> id in a plugin?
>>
>>
>>LieGrue,
>>strub
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Romain Manni-Bucau <rm...@gmail.com>.
What about browsing the build tree to detect the dep modules which needs to
be built (avoid a real clean which can cost really too much to make incr
feature useful)? Can be done in parallel and can be pretty fast
Le 6 sept. 2012 20:53, "Mark Struberg" <st...@yahoo.de> a écrit :
>
>
> Hi!
>
> I had some idea for detecting stale changes in maven which is pretty
> generic
>
>
> The problem hits us if you compile BeanA and BeanA2 in a project where
> BeanA2 is using BeanA.
> On a
>
> $> mvn clean compile
>
> you will get both BeanA.class and BeanA2.class in target/classes
> Now delete BeanA.java
> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> sources without doing any cleanup and thus BeanA.class will still remain in
> target/classes.
>
>
> That is clearly a bug as BeanA2 will be left broken, packaged into a
> broken jar, ...
>
>
>
> How can we avoid that?
>
> Simple answer: A plugin which doesnt support those cases by the underlying
> took (javac) must always first clean up the stuff it generated during the
> last invocation.
>
> How can we do that?
>
> step 1: Start a DirectoryScanner and get all files in target/classes.
> Remember this list!
>
>
> step 2: Run the compiler
>
>
> step 3: List all files in target/classes again and remove all files you
> found in step 1. You will end up with a list of all files generated by the
> compilation as result.
>
> step 4: Store this list into
> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> ('default' is the plugin execution. We need this in case we have multiple
> <executions>).
>
>
> On the next compile you just need to check if you have such a
> createdfiles.lst file and then first remove all the files listed in it as
> step 0.
> Then you can continue with step 1 from scratch.
>
> We could easily create a utility class for it which keeps the state with
> methods
>
> public class ChangeDetector /* TODO find better name */
> {
> File[] readPreviouslyDetectedFileList(File targetFile);
> void recordFiles(File baseDir)
> File[] detectNewFiles();
> storeDetectedFileList(File targetFile)
> }
>
> This can e.g. also be used by the maven-resources-plugin to get rid of
> copied resources which got deleted from src/main/resources.
>
> Do you have a better idea? Any ideas for improving this?
> And a big question: how can I get hold of the current <execution> id in a
> plugin?
>
>
> LieGrue,
> strub
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: [incremental build] Detect leftovers from a previous build
Posted by Mark Struberg <st...@yahoo.de>.
Answers inside.
LieGrue,
strub
----- Original Message -----
> From: Anders Hammar <an...@hammar.net>
> To: Maven Developers List <de...@maven.apache.org>; Mark Struberg <st...@yahoo.de>
> Cc:
> Sent: Thursday, September 6, 2012 9:51 PM
> Subject: Re: [incremental build] Detect leftovers from a previous build
>
> What you talk about here is something that BuildContext should
> provide, if we stay with that. There's a scanner for changes and a
> deleteScanner.
The BuildContext looks indeed like a good candidate!
> The really tricky thing is when one source file has more than one
> output/target file (like inner classes). The plugin needs some way of
> knowing what target file(s) to remove.
I know, that + the cross dependencies which are almost unpredictable (would require to parse and evaluate all sources). I opted for recompiling all sources because of that.
> I fit can't do that, then I
> guess the only option is to remove all the files it produced in the
> last execution.
Yes, that was this algorithm should make possible.
The problem is that we cannot simply delete all files in target/classes, because parts of it might have been created by other plugins (e.g. the resource processing).
>
> /Anders
>
> On Thu, Sep 6, 2012 at 8:52 PM, Mark Struberg <st...@yahoo.de> wrote:
>>
>>
>> Hi!
>>
>> I had some idea for detecting stale changes in maven which is pretty
> generic
>>
>>
>> The problem hits us if you compile BeanA and BeanA2 in a project where
> BeanA2 is using BeanA.
>> On a
>>
>> $> mvn clean compile
>>
>> you will get both BeanA.class and BeanA2.class in target/classes
>> Now delete BeanA.java
>> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all
> sources without doing any cleanup and thus BeanA.class will still remain in
> target/classes.
>>
>>
>> That is clearly a bug as BeanA2 will be left broken, packaged into a broken
> jar, ...
>>
>>
>>
>> How can we avoid that?
>>
>> Simple answer: A plugin which doesnt support those cases by the underlying
> took (javac) must always first clean up the stuff it generated during the last
> invocation.
>>
>> How can we do that?
>>
>> step 1: Start a DirectoryScanner and get all files in target/classes.
> Remember this list!
>>
>>
>> step 2: Run the compiler
>>
>>
>> step 3: List all files in target/classes again and remove all files you
> found in step 1. You will end up with a list of all files generated by the
> compilation as result.
>>
>> step 4: Store this list into
> target/maven-status/maven-compiler-plugin/default/createdfiles.lst
> ('default' is the plugin execution. We need this in case we have
> multiple <executions>).
>>
>>
>> On the next compile you just need to check if you have such a
> createdfiles.lst file and then first remove all the files listed in it as step
> 0.
>> Then you can continue with step 1 from scratch.
>>
>> We could easily create a utility class for it which keeps the state with
> methods
>>
>> public class ChangeDetector /* TODO find better name */
>> {
>> File[] readPreviouslyDetectedFileList(File targetFile);
>> void recordFiles(File baseDir)
>> File[] detectNewFiles();
>> storeDetectedFileList(File targetFile)
>> }
>>
>> This can e.g. also be used by the maven-resources-plugin to get rid of
> copied resources which got deleted from src/main/resources.
>>
>> Do you have a better idea? Any ideas for improving this?
>> And a big question: how can I get hold of the current <execution> id
> in a plugin?
>>
>>
>> LieGrue,
>> strub
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: [incremental build] Detect leftovers from a previous build
Posted by Anders Hammar <an...@hammar.net>.
What you talk about here is something that BuildContext should
provide, if we stay with that. There's a scanner for changes and a
deleteScanner.
The really tricky thing is when one source file has more than one
output/target file (like inner classes). The plugin needs some way of
knowing what target file(s) to remove. I fit can't do that, then I
guess the only option is to remove all the files it produced in the
last execution.
/Anders
On Thu, Sep 6, 2012 at 8:52 PM, Mark Struberg <st...@yahoo.de> wrote:
>
>
> Hi!
>
> I had some idea for detecting stale changes in maven which is pretty generic
>
>
> The problem hits us if you compile BeanA and BeanA2 in a project where BeanA2 is using BeanA.
> On a
>
> $> mvn clean compile
>
> you will get both BeanA.class and BeanA2.class in target/classes
> Now delete BeanA.java
> Currently (2.6-SNAPSHOT) the maven-compiler-plugin only compiles all sources without doing any cleanup and thus BeanA.class will still remain in target/classes.
>
>
> That is clearly a bug as BeanA2 will be left broken, packaged into a broken jar, ...
>
>
>
> How can we avoid that?
>
> Simple answer: A plugin which doesnt support those cases by the underlying took (javac) must always first clean up the stuff it generated during the last invocation.
>
> How can we do that?
>
> step 1: Start a DirectoryScanner and get all files in target/classes. Remember this list!
>
>
> step 2: Run the compiler
>
>
> step 3: List all files in target/classes again and remove all files you found in step 1. You will end up with a list of all files generated by the compilation as result.
>
> step 4: Store this list into target/maven-status/maven-compiler-plugin/default/createdfiles.lst ('default' is the plugin execution. We need this in case we have multiple <executions>).
>
>
> On the next compile you just need to check if you have such a createdfiles.lst file and then first remove all the files listed in it as step 0.
> Then you can continue with step 1 from scratch.
>
> We could easily create a utility class for it which keeps the state with methods
>
> public class ChangeDetector /* TODO find better name */
> {
> File[] readPreviouslyDetectedFileList(File targetFile);
> void recordFiles(File baseDir)
> File[] detectNewFiles();
> storeDetectedFileList(File targetFile)
> }
>
> This can e.g. also be used by the maven-resources-plugin to get rid of copied resources which got deleted from src/main/resources.
>
> Do you have a better idea? Any ideas for improving this?
> And a big question: how can I get hold of the current <execution> id in a plugin?
>
>
> LieGrue,
> strub
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org