Re: Apache Yetus annotations : clarifying public vs private api

[Joe Witt <jo...@gmail.com>]

Team,

The big ticket back-end changes for NiFi 1.x line seem to be closing
down and Apache Yetus has pushed their artifacts to maven central so I
am proceeding with NIFI-1373.  I plan to start with a limited rollout
but specifically will annotate the nifi-api completely.  I also
updated our wiki which talks about our version scheme [1].  We can see
how this goes and then if we want to expand it to other parts of the
codebase we can do so later.

The specific text i added to that was:
"
Also, starting with the Apache NiFi 1.x codebase we have adopted
Apache Yetus 'Audience Annotations' to explicitly mark code for things
like interfaces, classes, and methods to indicate the intended
audience and stability of those APIs.  If not otherwise marked please
consider the interface, class, or method to be private/internal and
unstable.  The vast majority of 'nifi-api' should be both public use
and stable meaning we will treat any compatibility changes for these
items as things which would likely require a major release.
Otherwise, as a community we should be able to more easily navigate
the necessary evolution and improvement of the codebase for these
interfaces, classes, and methods for which we do not have to be so
strictly adherent to backward compatibility.  A further consideration
here though is as mentioned previously which is that even if we want
to change classes, interfaces, and methods which are not public and
stable we must also be mindful of things like configuration, user
experience, REST API, etc... as these are also an important part of
our 'interface' and compatibility which we must honor and account for.
"

The idea is that unless we have marked something it should be assumed
to be private and unstable as far as implementation goes though this
doesn't eliminate our need to honor the holistic interpretation of our
"interface".

[1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme

Thanks
Joe

On Tue, Jul 26, 2016 at 7:59 AM, Joe Skora <js...@gmail.com> wrote:
> +1
>
> This should help clarify things when learning NiFi and when diving in new
> areas of the codebase.
>
> Thanks, I'm looking forward to it.
>
> On Mon, Jul 25, 2016 at 7:45 PM, Matt Burgess <ma...@gmail.com> wrote:
>
>> agreed, big +1 here, will make things much more clear re: stability
>> and extensibility.
>>
>> On Mon, Jul 25, 2016 at 7:43 PM, Tony Kurc <tr...@gmail.com> wrote:
>> > awesome!
>> >
>> > On Mon, Jul 25, 2016 at 7:32 PM, Joe Witt <jo...@gmail.com> wrote:
>> >
>> >> Team
>> >>
>> >> For those interested I am going to take a stab at working through the
>> >> codebase [1] to apply Apache Yetus [2] annotations to make it more
>> >> clear and well communicated what are the public vs private aspects of
>> >> the API.
>> >>
>> >> We've had the 'nifi-api' module for some time now but unless you know
>> >> that this was meant to be the public API for supported extension types
>> >> you'd not really know that other parts of NiFi were not meant to be
>> >> altered and could be changed.  Therefore, we've not been able to
>> >> evolve the framework as directly as we'd like at times.  We had to
>> >> wait until 1.0 to make changes which otherwise should have been ok to
>> >> make.  Also, previously 'nifi-api' had a lot of extraneous bits in it
>> >> that have now been refactored out to where they belong such as core
>> >> framework itself or in a 'nifi-framework-api' module.  This means
>> >> things in 'nifi-api' should be quite stable and public.
>> >>
>> >> Yetus gives us a chance to clearly articulate which parts of the API
>> >> are meant for public use and which are meant for private use only and
>> >> therefore are subject to change.  Furthermore, even for those public
>> >> elements we can more effectively articulate stability in ways that map
>> >> really nicely to our versioning scheme.
>> >>
>> >> My intent is to be very cautious in labeling anything beyond the
>> >> nifi-api as public.  We can open things up as they prove to be stable
>> >> or clearly need to be made available as supported points of extension
>> >> but we can't really go back the other way.
>> >>
>> >> We of course also have to realize our 'interface' is more than these
>> >> APIs we're talking about.  It is the REST interface, it is
>> >> configuration files, it is templates, and so on.
>> >>
>> >> Anyway, I hope to have the PR up for this very soon but wanted to send
>> >> a special note to the dev mailing list in case there are folks
>> >> particularly interested in this and who want to share their views.  If
>> >> you are please keep an eye out for NIFI-1373.
>> >>
>> >> [1] https://issues.apache.org/jira/browse/NIFI-1373
>> >> [2]
>> >>
>> https://yetus.apache.org/documentation/0.3.0/audience-annotations-apidocs/
>> >>
>> >> Thanks
>> >> Joe
>> >>
>>

Re: Apache Yetus annotations : clarifying public vs private api

[Joe Witt <jo...@gmail.com>]

Team,

I've gone ahead and focused more on ensuring that the
versioning/scheme and compatibility model we've been following is more
clearly articulated on [1] and also more specifically defined what is
considered part of the nifi-api.

I still think the yetus annotations or something inspired by them is
the right way to go but at this point wanted to avoid adding a
dependency to the nifi-api.  I'm actually inclined to think that if we
go further with this it should be done by bringing the yetus
annotations in as a source dependency rather than a binary one.  In
any event, the near term pain was to more clearly articulate the API
compatibility model and i believe that is resolved.  Also added some
comments about it on the JIRA i raised for it [2].

[1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility
[2] https://issues.apache.org/jira/browse/NIFI-1373

On Sun, Jul 31, 2016 at 7:15 PM, Joe Witt <jo...@gmail.com> wrote:
> Team,
>
> The big ticket back-end changes for NiFi 1.x line seem to be closing
> down and Apache Yetus has pushed their artifacts to maven central so I
> am proceeding with NIFI-1373.  I plan to start with a limited rollout
> but specifically will annotate the nifi-api completely.  I also
> updated our wiki which talks about our version scheme [1].  We can see
> how this goes and then if we want to expand it to other parts of the
> codebase we can do so later.
>
> The specific text i added to that was:
> "
> Also, starting with the Apache NiFi 1.x codebase we have adopted
> Apache Yetus 'Audience Annotations' to explicitly mark code for things
> like interfaces, classes, and methods to indicate the intended
> audience and stability of those APIs.  If not otherwise marked please
> consider the interface, class, or method to be private/internal and
> unstable.  The vast majority of 'nifi-api' should be both public use
> and stable meaning we will treat any compatibility changes for these
> items as things which would likely require a major release.
> Otherwise, as a community we should be able to more easily navigate
> the necessary evolution and improvement of the codebase for these
> interfaces, classes, and methods for which we do not have to be so
> strictly adherent to backward compatibility.  A further consideration
> here though is as mentioned previously which is that even if we want
> to change classes, interfaces, and methods which are not public and
> stable we must also be mindful of things like configuration, user
> experience, REST API, etc... as these are also an important part of
> our 'interface' and compatibility which we must honor and account for.
> "
>
> The idea is that unless we have marked something it should be assumed
> to be private and unstable as far as implementation goes though this
> doesn't eliminate our need to honor the holistic interpretation of our
> "interface".
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme
>
> Thanks
> Joe
>
> On Tue, Jul 26, 2016 at 7:59 AM, Joe Skora <js...@gmail.com> wrote:
>> +1
>>
>> This should help clarify things when learning NiFi and when diving in new
>> areas of the codebase.
>>
>> Thanks, I'm looking forward to it.
>>
>> On Mon, Jul 25, 2016 at 7:45 PM, Matt Burgess <ma...@gmail.com> wrote:
>>
>>> agreed, big +1 here, will make things much more clear re: stability
>>> and extensibility.
>>>
>>> On Mon, Jul 25, 2016 at 7:43 PM, Tony Kurc <tr...@gmail.com> wrote:
>>> > awesome!
>>> >
>>> > On Mon, Jul 25, 2016 at 7:32 PM, Joe Witt <jo...@gmail.com> wrote:
>>> >
>>> >> Team
>>> >>
>>> >> For those interested I am going to take a stab at working through the
>>> >> codebase [1] to apply Apache Yetus [2] annotations to make it more
>>> >> clear and well communicated what are the public vs private aspects of
>>> >> the API.
>>> >>
>>> >> We've had the 'nifi-api' module for some time now but unless you know
>>> >> that this was meant to be the public API for supported extension types
>>> >> you'd not really know that other parts of NiFi were not meant to be
>>> >> altered and could be changed.  Therefore, we've not been able to
>>> >> evolve the framework as directly as we'd like at times.  We had to
>>> >> wait until 1.0 to make changes which otherwise should have been ok to
>>> >> make.  Also, previously 'nifi-api' had a lot of extraneous bits in it
>>> >> that have now been refactored out to where they belong such as core
>>> >> framework itself or in a 'nifi-framework-api' module.  This means
>>> >> things in 'nifi-api' should be quite stable and public.
>>> >>
>>> >> Yetus gives us a chance to clearly articulate which parts of the API
>>> >> are meant for public use and which are meant for private use only and
>>> >> therefore are subject to change.  Furthermore, even for those public
>>> >> elements we can more effectively articulate stability in ways that map
>>> >> really nicely to our versioning scheme.
>>> >>
>>> >> My intent is to be very cautious in labeling anything beyond the
>>> >> nifi-api as public.  We can open things up as they prove to be stable
>>> >> or clearly need to be made available as supported points of extension
>>> >> but we can't really go back the other way.
>>> >>
>>> >> We of course also have to realize our 'interface' is more than these
>>> >> APIs we're talking about.  It is the REST interface, it is
>>> >> configuration files, it is templates, and so on.
>>> >>
>>> >> Anyway, I hope to have the PR up for this very soon but wanted to send
>>> >> a special note to the dev mailing list in case there are folks
>>> >> particularly interested in this and who want to share their views.  If
>>> >> you are please keep an eye out for NIFI-1373.
>>> >>
>>> >> [1] https://issues.apache.org/jira/browse/NIFI-1373
>>> >> [2]
>>> >>
>>> https://yetus.apache.org/documentation/0.3.0/audience-annotations-apidocs/
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>>