Reimplementing the SCR Generator using annotation processing (FELIX-3550)

[Robert Munteanu <rm...@adobe.com>]

Hi,

I've noticed that the SCR Generator is being reimplemented ( https://issues.apache.org/jira/browse/FELIX-3550 ) so this might a good time to discuss annotation processing [1] as an implementation option.

I think that the javax.annotation.processing API and tools support the SCR use case, which is - to my knowledge:

1. analyse annotated classes
2. write XML descriptors based on the annotated classes
3. fail the build under certain conditions

I have written a couple of annotation processors some time ago - not open source - but I can at least validate that the above are possible.

The benefits of implementing the SCR generator using an annotation process are IMO:

1. Simpler integration with build tools by simply dropping the jar in the compilation classpath . This works for both Maven/Ant, which we now support, but would make it easier to integrate with other build tools , like Gradle .

2. Better IDE integration 
- at least  Eclipse can report errors / warnings in the editor [2,3] 
- it will no longer be required to provide a specific m2eclipse plugin 

Some points which are still not clear for me implementation-wise are:

1. Configuration

There is a standard way of passing compiler arguments when invoking javac (-Akey=value ) but it would require configuring the compiler rather than the scr plugin. At least for maven builds this is less convenient.

2. Incremental builds

When performing a partial build ( for instance in Eclipse, but I think it also applies to Maven ) not all annotated classes are passed to the processor so the SCR generator should either only work on full builds by deign or be able to perform incremental builds.

Thoughts?

Robert


[1]: http://docs.oracle.com/javase/6/docs/technotes/guides/language/annotations.html
[2]: http://rombertw.wordpress.com/2010/07/03/cutting-down-gwts-verbosity-with-gwt-mpv-apt/
[3]: http://rombertw.files.wordpress.com/2010/07/genevent-error.png

RE: Reimplementing the SCR Generator using annotation processing (FELIX-3550)

[Robert Munteanu <rm...@adobe.com>]

> -----Original Message-----
> From: Carsten Ziegeler [mailto:cziegeler@apache.org]
> Sent: Monday, June 25, 2012 11:03 PM
> To: dev@felix.apache.org
> Subject: Re: Reimplementing the SCR Generator using annotation
> processing (FELIX-3550)
> 
> Hi,
> 
> 2012/6/15 Robert Munteanu <rm...@adobe.com>:
> >
> > If I understand correctly one particular annotated class can also
> depend on another annotated class and perhaps on files on the compiler
> classpath . An annotation processing engine should be able to
> >
> > - supply complete type information when processing an annotated class
> - this includes all superclasses and their annotations
> > - supply all the classes which require compilation in the same
> annotation processor invocation . One notable exception is Eclipse -
> when running an incremental compilation only the changed classes are
> set up for processing.
> >
> > Also, an annotation processor can be declared to handle multiple
> annotations, so we don't have to implement one processor for each
> annotation. For flexibility we can implement multiple annotation
> handlers, but we will hook in a single class in the compilation
> process.
> 
> Yes, this was my understanding as well.
> 
> I think its really worth exploring this, so if you (or someone else)
> could come up with a prototype demonstrating that this way is
> possible, it would be a great starting point for further discussions.

I should be able to prototype this approach in mid-July.

Robert

> 
> Regards
> Carsten
> >
> > Robert
> >
> >>
> >> Regards
> >> Carsten
> >> --
> >> Carsten Ziegeler
> >> cziegeler@apache.org
> 
> 
> 
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: Reimplementing the SCR Generator using annotation processing (FELIX-3550)

[Carsten Ziegeler <cz...@apache.org>]

Hi,

2012/6/15 Robert Munteanu <rm...@adobe.com>:
>
> If I understand correctly one particular annotated class can also depend on another annotated class and perhaps on files on the compiler classpath . An annotation processing engine should be able to
>
> - supply complete type information when processing an annotated class - this includes all superclasses and their annotations
> - supply all the classes which require compilation in the same annotation processor invocation . One notable exception is Eclipse - when running an incremental compilation only the changed classes are set up for processing.
>
> Also, an annotation processor can be declared to handle multiple annotations, so we don't have to implement one processor for each annotation. For flexibility we can implement multiple annotation handlers, but we will hook in a single class in the compilation process.

Yes, this was my understanding as well.

I think its really worth exploring this, so if you (or someone else)
could come up with a prototype demonstrating that this way is
possible, it would be a great starting point for further discussions.

Regards
Carsten
>
> Robert
>
>>
>> Regards
>> Carsten
>> --
>> Carsten Ziegeler
>> cziegeler@apache.org



-- 
Carsten Ziegeler
cziegeler@apache.org

RE: Reimplementing the SCR Generator using annotation processing (FELIX-3550)

[Robert Munteanu <rm...@adobe.com>]

Hi Carsten,

> Hi Robert,
> 
> thanks for starting this discussion. It really would be nice to base
> the implementation on the javax.annotation.proessing stuff.
> 
> 2012/6/14 Robert Munteanu <rm...@adobe.com>:
> > 1. Configuration
> >
> > There is a standard way of passing compiler arguments when invoking
> javac (-Akey=value ) but it would require configuring the compiler
> rather than the scr plugin. At least for maven builds this is less
> convenient.
> 
> Yes, I guess this is one of the biggest challenges. Now, we might
> simply pass the information through a static field in some common
> class. However this would be problematic with parallel builds like
> Maven. So we maybe could use a thread local which would only work if
> the same thread is used. But I guess we could find a solution here. If
> we are able to pass objects into the processor, we can use these
> objects to pass back the result. For example, the Maven plugin is
> creating a manifest header afterwards.

I _think_ that the arguments are passed to each processor instance individually, so we should be safe here. As far as I remember not even Eclipse reuses instances between processing runs.

> 
> >
> > 2. Incremental builds
> >
> > When performing a partial build ( for instance in Eclipse, but I
> think it also applies to Maven ) not all annotated classes are passed
> to the processor so the SCR generator should either only work on full
> builds by deign or be able to perform incremental builds.
> 
> Incremental builds should not pose a problem. Right now, all
> descriptors are written into a single xml file, however we can change
> this easily with generating a file for each component. This can easily
> be updated incrementally (as long as we can handle deletes etc.)

OK, I was under the impression that there has to be a single xml file. Having one file per annotated component will greatly simplify matters.

> 
> But I see some problems in the general processing: right now the tool
> is parsing two different kinds of classes: classes belonging to the
> project and classes from dependencies as a project class might inherit
> from a dependency and it might want to inherit the descriptors as
> well.  I guess we would still need ASM for handling the dependencies.
> The result of a component descriptor is the assembly of several
> annotations and these annotations are pluggable and therefore are
> processed by different processors. So we need a way to first assemble
> all annotations and once all annotations of a particular class are
> assembled, process them in one go. I'm not sure if this is possible.

If I understand correctly one particular annotated class can also depend on another annotated class and perhaps on files on the compiler classpath . An annotation processing engine should be able to 

- supply complete type information when processing an annotated class - this includes all superclasses and their annotations
- supply all the classes which require compilation in the same annotation processor invocation . One notable exception is Eclipse - when running an incremental compilation only the changed classes are set up for processing.

Also, an annotation processor can be declared to handle multiple annotations, so we don't have to implement one processor for each annotation. For flexibility we can implement multiple annotation handlers, but we will hook in a single class in the compilation process.

Robert

> 
> Regards
> Carsten
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: Reimplementing the SCR Generator using annotation processing (FELIX-3550)

[Carsten Ziegeler <cz...@apache.org>]

Hi Robert,

thanks for starting this discussion. It really would be nice to base
the implementation on the javax.annotation.proessing stuff.

2012/6/14 Robert Munteanu <rm...@adobe.com>:
> 1. Configuration
>
> There is a standard way of passing compiler arguments when invoking javac (-Akey=value ) but it would require configuring the compiler rather than the scr plugin. At least for maven builds this is less convenient.

Yes, I guess this is one of the biggest challenges. Now, we might
simply pass the information through a static field in some common
class. However this would be problematic with parallel builds like
Maven. So we maybe could use a thread local which would only work if
the same thread is used. But I guess we could find a solution here. If
we are able to pass objects into the processor, we can use these
objects to pass back the result. For example, the Maven plugin is
creating a manifest header afterwards.

>
> 2. Incremental builds
>
> When performing a partial build ( for instance in Eclipse, but I think it also applies to Maven ) not all annotated classes are passed to the processor so the SCR generator should either only work on full builds by deign or be able to perform incremental builds.

Incremental builds should not pose a problem. Right now, all
descriptors are written into a single xml file, however we can change
this easily with generating a file for each component. This can easily
be updated incrementally (as long as we can handle deletes etc.)

But I see some problems in the general processing: right now the tool
is parsing two different kinds of classes: classes belonging to the
project and classes from dependencies as a project class might inherit
from a dependency and it might want to inherit the descriptors as
well.  I guess we would still need ASM for handling the dependencies.
The result of a component descriptor is the assembly of several
annotations and these annotations are pluggable and therefore are
processed by different processors. So we need a way to first assemble
all annotations and once all annotations of a particular class are
assembled, process them in one go. I'm not sure if this is possible.

Regards
Carsten
-- 
Carsten Ziegeler
cziegeler@apache.org