You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Daniel Bress <db...@onyxconsults.com> on 2015/01/11 18:31:52 UTC
Processor documentation plugin
Hi All,
Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
What do you think?
Dan Bress
Software Engineer
ONYX Consulting Services
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
All,
I tweaked the layout of the generated documentation and put a new rendered copy here: http://danbress.github.io/generated-documentation/components-02072015/
Please provide any feedback you may have. I changed the css for the table a little bit to make things more readable. I'm not married to it, so if anyone wants to build upon it, please go forward.
I am cancelling my old pull request and issuing a new one as I did the changes for this in a new branch. Let me know how this looks.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Daniel Bress <db...@onyxconsults.com>
Sent: Sunday, January 25, 2015 9:43 AM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Jenn,
Good point, your text makes a lot more sense than what I had. Thanks for that! I just included it and regenerated the docs, let me know what you think.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Jennifer Barnabee <je...@gmail.com>
Sent: Sunday, January 25, 2015 9:15 AM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Dan,
It looks good. Assuming we decide to move forward with it, we'll want to
change the boilerplate paragraph under Properties to suit what you have a
little better. I propose something like the following, but it may need to
change as things evolve. Please see what you think...
In the list below, the names of required properties appear in *bold*. Any
other properties (not in bold) are considered optional. The table also
indicates any default values, whether a property supports the NiFi
Expression Language (or simply EL), and whether a property is considered
"sensitive", meaning that its value will be encrypted. Before entering a
value in a sensitive property, ensure that the *nifi.properties* file has
an entry for the property
*nifi.sensitive.props.key.*
-Jenn
On Sun, Jan 25, 2015 at 9:01 AM, Daniel Bress <db...@onyxconsults.com>
wrote:
> Tony,
> Thanks for pointing that out. I had a "\" when I needed a "/". I
> fixed this and re uploaded it. Let me know how that looks.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Tony Kurc <tr...@gmail.com>
> Sent: Saturday, January 24, 2015 8:07 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
> For whatever reason, when I follow the hyperlinks, the last "slash" is a \
> instead of a / and I get a 404 (
>
> http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html
> ).
>
>
>
>
> On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > All,
> > I've generated the first round of documentation I'd like everyone to
> > look at to see if they like this new format. Its really only slightly
> > changed from the current format(using tables instead of lists for
> > Properties and Relationships), but its all machine generated, except for
> > the stuff linked by "Additional Details".
> >
> > Again, this would be generated by the app at startup. I'd also like
> to
> > have this generated sometime during the build and linked from the
> website.
> > For now I just pulled that generated html and uploaded it to my github
> page
> > for your review.
> >
> > http://danbress.github.io/generated-documentation/components/
> >
> > Let me know what you think! I'm open to any ideas/suggestions.
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Thursday, January 22, 2015 8:16 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > OK, now I understand where you're coming from. Sorry, I misunderstood the
> > idea before. If you think it would be too verbose then I think we should
> > hold off for now. I think it's best to keep it simple. If we want to add
> it
> > in later, we can. If we add it now and decide it's too verbose, it would
> be
> > awkward to go back and remove documentation later but it's easy to later
> > add in something that's missing.
> >
> > So in short I agree with your assessment :)
> > -Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 22 Jan 2015 03:13:16 +0000
> > >
> > > Mark,
> > > No worries. On the dynamic property tip, I see what you are saying.
> > I was hoping to provide the same level of detail in describing dynamic
> > properties as non-dynamic ones, but right now I see no way to do that.
> > Thats fine, I'll put that part(providing detailed documentation of the
> > dynamic properties) on the back burner.
> > >
> > > On the Validator tip, for each PropertyDescriptor I am getting the
> > list of AllowableValues so that I can say "This property has X, Y and Z
> for
> > valid values". If there are no AllowableValues present I would like to
> > look at the validators and say "This property requires a positive
> > integer". or "This requires a <data size> followed by a <data unit>".
> > Although typing this all out now makes it seem like it might be too
> verbose
> > and cluttered. I think I'll forget this idea and just work with
> > AllowableValues.
> > >
> > > Thanks again,
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Tuesday, January 20, 2015 12:16 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Sorry, I never did respond to this e-mail.
> > > You cannot get the dynamic PropertyDescriptor because the
> > AbstractConfigurableComponent (the base abstract class from which every
> > processor, controller service, reporting task that I know of extends)
> > provides the following method signature:
> > > protected PropertyDescriptor
> getSupportedDynamicPropertyDescriptor(final
> > String propertyDescriptorName)
> > > I.e., there is not a PropertyDescriptor for dynamic properties but
> > rather the descriptor can be different for every dynamic property. For
> > example, consider a Processor that allows dynamic properties but for
> every
> > property X there must be 2 other properties: X.format (any string) and
> > X.size (a non-zero integer). If there were a single PropertyDescriptor,
> > this wouldn't be possible because all dynamic properties would have the
> > same validators, etc. But by making a property descriptor based on the
> > property name, we can easily achieve this.
> > > Regarding the second note: I don't think Validator can provide
> > information about what it is validating because it doesn't know. For
> > instance, PositiveInteger validator just validates that some value is a
> > positive integer. It does not (and I think should not) know the context
> of
> > the value.
> > > If anything here doesn't make sense or if you think it could be done
> > better, let me know and we can discuss further.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Fri, 16 Jan 2015 21:14:41 +0000
> > > >
> > > > Mark & Matt,
> > > > Cool. While I like the idea of @SupportsDynamicProperties,
> > especially if it has a description, I'd also like a guaranteed way to get
> > the dynamic PropertyDescriptor so I can interrogate it further. I'd be
> > interested in knowing whether it supports expression language, and/or how
> > the value of the property is validated.
> > > >
> > > > On that note, I was thinking it may be helpful to add either an
> > annotation or a method to the Validator interface that could provide a
> text
> > description about what it is validating. Is this useful or overkill?
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Friday, January 16, 2015 11:14 AM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > It might make sense to have an annotation for this:
> > > > @SupportsDynamicProperties
> > > > Perhaps even allow developer to provide a string for how they are
> used:
> > > > @SupportsDynamicProperties("Name of dynamic property becomes
> attribute
> > name; value becomes attribute value.")
> > > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6
> > but I have not pushed anything to a branch or anything yet. Basically, I
> > deprecated the existing annotations and created 3 new packages for them.
> > Instead of org.apache.nifi.processor.annotation, we will have:
> > > >
> >
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > > > So the @SupportsDynamicProperties could go under 'behavior' if that
> > makes sense.
> > > >
> > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > > > Subject: Re: Processor documentation plugin
> > > > > From: matt.c.gilman@gmail.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > >
> > > > > Dan,
> > > > >
> > > > > We discussed item 1 awhile back and even created a ticket for it.
> > However,
> > > > > there was always higher priority work that took precedence. I am
> not
> > sure
> > > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> > should.
> > > > > I believe the current mechanism allows the processor to decide if a
> > given
> > > > > dynamic property should be a supported. I think we would need a
> more
> > > > > general indicator that dynamic properties are supported at all to
> > trigger
> > > > > that button state. This would likely help with item 2 as well.
> > > > >
> > > > > Matt
> > > > >
> > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > wrote:
> > > > >
> > > > > > Mark,
> > > > > > Sounds good. I'm on board. I was thinking of having the
> > > > > > ControllerServices/ReportingTask's documentation include an XML
> > template,
> > > > > > but if those will be configurable via the UI then that will not
> be
> > needed.
> > > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll
> get
> > to work
> > > > > > accordingly.
> > > > > >
> > > > > >
> > > > > > Matt/Mark,
> > > > > > Two UI/API questions:
> > > > > > 1) If a processor does not support dynamic properties, why does
> > the UI let
> > > > > > them hit [+] New Property?
> > > > > > 2) If I have an instance of Processor class, I can call
> > > > > > getSupportedProperties() on it to get a list of the
> > required/optional
> > > > > > properties. But I don't have a way to know if that processor
> > supports
> > > > > > dynamic properties or not. Supporting dynamic properties or not
> > was
> > > > > > something I was planning on reflecting in the documentation. Is
> > there any
> > > > > > way to detect this currently?
> > > > > >
> > > > > > Thanks,
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: RE: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > > I think this is great!
> > > > > > I would recommend we slate this for the 0.1.0 version, which
> > includes
> > > > > > putting Controller Services and Reporting Tasks in the UI. But by
> > the same
> > > > > > token I think that makes your first bulletin for "rev 2" more of
> a
> > priority
> > > > > > for "rev 1" if we can get it there. I could pitch in and help if
> > needed.
> > > > > > I'd also recommend you put the Capability Description in for Rev
> 1
> > as
> > > > > > that's arguably the most important part of the docs (which may
> > make the
> > > > > > Tags very easy at that point??)
> > > > > > The rest of what you have listed for "rev 2" i would agree should
> > be there.
> > > > > >
> > > > > > Very excited to see this happening!
> > > > > > Thanks-Mark
> > > > > >
> > > > > > > From: dbress@onyxconsults.com
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > > > >
> > > > > > > Mark et al,
> > > > > > > Ok, I think I hear everyone's concerns and ideas. I will
> > challenge
> > > > > > myself to come up with a solution that makes everyone happy all
> > the time.
> > > > > > >
> > > > > > > The goals of this effort will be:
> > > > > > > - Machine generate as much of the basic(processor
> > description,
> > > > > > property description, relationship description) info as possible.
> > This
> > > > > > will make the developers lives better by storing this information
> > in one
> > > > > > place, and make users lives better by having the "inline"
> > documentation
> > > > > > match the "usage" documentation. This will also ensure that all
> > the
> > > > > > documentation is formatted consistently.
> > > > > > > - Allow the developer to optionally provide additional
> > html and
> > > > > > images that would be linked from the machine generated
> > documentation
> > > > > > > - Provide a way for the documentation to be generated
> > statically
> > > > > > and linked from the Apache NiFi webpage
> > > > > > >
> > > > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > > > - Support documenting ControllerServices and
> > ReportingTasks is a
> > > > > > similar way
> > > > > > > -Maybe leverage existing annotations on these types?
> > E.g
> > > > > > @Tags, @CapabilityDescription...
> > > > > > > - Provide new 'documentation' annotations that would
> allow
> > a
> > > > > > developer to describe the FlowFile attributes a processor uses on
> > input,
> > > > > > and which attributes a processor modifies on output
> > > > > > > - Provide new 'documentation' annotations that describe
> the
> > > > > > expected input format of the flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > whether
> > > > > > the processor modifies the flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > the format
> > > > > > of the outbound flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > other NiFi
> > > > > > artifacts(processors, controller services, reporting tasks) that
> > are
> > > > > > related to this one, e.g. "See Also ..."
> > > > > > > - Consider if/how current Processor annotations should be
> > > > > > displayed (@SideEffectFree, @TriggerSerially,
> @TriggerWhenEmpty...,
> > > > > > @EventDriven)
> > > > > > >
> > > > > > >
> > > > > > > High level implementation plan:
> > > > > > > - Develop a utility class to read an
> AbstractProcessor(and
> > > > > > existing HTML documention) and generated the basic documentation
> > with links
> > > > > > to existing HTML
> > > > > > > - Integrate this class into the framework such that all
> > > > > > documentation displayed within a NiFi instance is generated by
> the
> > utility
> > > > > > > - Integrate this utility into something that can take a
> > NiFi
> > > > > > build, dig through the nars, find the
> > > > > > Processors/ControllerServices/ReportingTasks and generate their
> > appropriate
> > > > > > HTML documentation so that it can be stored somewhere and linked
> > from the
> > > > > > Apache NiFi website.
> > > > > > >
> > > > > > > If this feels like the wrong path, or if you have any other
> > ideas
> > > > > > please speak up. Thanks!
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: RE: Processor documentation plugin
> > > > > > >
> > > > > > > Dan,
> > > > > > > Yes, that's what I'm saying. Right now, it's all done manually
> > by hand,
> > > > > > and it's a pain. The down side i think to doing it via a Maven
> > plugin is
> > > > > > that if the user wants to modify it, they have to rebuild, pull
> the
> > > > > > generated html out of the target directory, and then modify it
> and
> > put it
> > > > > > in the src/main/resources/docs. Then, if they change something
> > they have to
> > > > > > do this again. If it's done on-the-fly, rather than building
> > static content
> > > > > > with a plugin, that doesn't have to happen.
> > > > > > > But you may come up with something much better than I'm
> > envisioning :)
> > > > > > Having a util to do it is a good call though. Then it could be
> > used in a
> > > > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > > > Thanks-Mark
> > > > > > >
> > > > > > > > From: dbress@onyxconsults.com
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > > > >
> > > > > > > > Mark,
> > > > > > > > Sorry.. I missed your reply in the thread.
> > > > > > > >
> > > > > > > > Are you saying that going forward, you want to auto
> > generate the
> > > > > > doc and allow the developer to include a "more info" link? I
> > wanted to
> > > > > > confirm that it doesn't work that way now.
> > > > > > > >
> > > > > > > > I'll think about this a little more. Might write some
> code
> > to
> > > > > > generate the documentation from an AbstractProcessor class, then
> > decide
> > > > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Dan Bress
> > > > > > > > Software Engineer
> > > > > > > > ONYX Consulting Services
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > >
> > > > > > > > Dan,
> > > > > > > >
> > > > > > > > Matt and I discussed this approach before but ultimately
> > decided that
> > > > > > we like the idea of just auto generating the docs on demand with
> a
> > "More
> > > > > > Info" type of link that lets you add more in html format if you
> > care to do
> > > > > > so.
> > > > > > > >
> > > > > > > > Others' thoughts and ideas are welcome though!
> > > > > > > >
> > > > > > > > Sent from my iPhone
> > > > > > > >
> > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Just pulled down the code out of Git and got everything
> up
> > and
> > > > > > running on Windows. This was pretty simple, and the UI looks
> > great under
> > > > > > chrome.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I was thinking of writing a maven plugin to generate the
> > processor
> > > > > > usage HTML documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Looking through the existing processor documentation, it
> > seems
> > > > > > like most of this information comes directly from the code in the
> > > > > > processors, via @CapabilityDescription, and the
> > PropertyDescriptors and
> > > > > > Relationships. Then if your processor only requires this level
> of
> > > > > > documentation in the HTML format, you could tie this plugin to
> the
> > build,
> > > > > > and keep your inline processor documentation in sync with the
> HTML
> > > > > > documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Dan Bress
> > > > > > > > > Software Engineer
> > > > > > > > > ONYX Consulting Services
> > > > > >
> >
>
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Jenn,
Good point, your text makes a lot more sense than what I had. Thanks for that! I just included it and regenerated the docs, let me know what you think.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Jennifer Barnabee <je...@gmail.com>
Sent: Sunday, January 25, 2015 9:15 AM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Dan,
It looks good. Assuming we decide to move forward with it, we'll want to
change the boilerplate paragraph under Properties to suit what you have a
little better. I propose something like the following, but it may need to
change as things evolve. Please see what you think...
In the list below, the names of required properties appear in *bold*. Any
other properties (not in bold) are considered optional. The table also
indicates any default values, whether a property supports the NiFi
Expression Language (or simply EL), and whether a property is considered
"sensitive", meaning that its value will be encrypted. Before entering a
value in a sensitive property, ensure that the *nifi.properties* file has
an entry for the property
*nifi.sensitive.props.key.*
-Jenn
On Sun, Jan 25, 2015 at 9:01 AM, Daniel Bress <db...@onyxconsults.com>
wrote:
> Tony,
> Thanks for pointing that out. I had a "\" when I needed a "/". I
> fixed this and re uploaded it. Let me know how that looks.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Tony Kurc <tr...@gmail.com>
> Sent: Saturday, January 24, 2015 8:07 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
> For whatever reason, when I follow the hyperlinks, the last "slash" is a \
> instead of a / and I get a 404 (
>
> http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html
> ).
>
>
>
>
> On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > All,
> > I've generated the first round of documentation I'd like everyone to
> > look at to see if they like this new format. Its really only slightly
> > changed from the current format(using tables instead of lists for
> > Properties and Relationships), but its all machine generated, except for
> > the stuff linked by "Additional Details".
> >
> > Again, this would be generated by the app at startup. I'd also like
> to
> > have this generated sometime during the build and linked from the
> website.
> > For now I just pulled that generated html and uploaded it to my github
> page
> > for your review.
> >
> > http://danbress.github.io/generated-documentation/components/
> >
> > Let me know what you think! I'm open to any ideas/suggestions.
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Thursday, January 22, 2015 8:16 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > OK, now I understand where you're coming from. Sorry, I misunderstood the
> > idea before. If you think it would be too verbose then I think we should
> > hold off for now. I think it's best to keep it simple. If we want to add
> it
> > in later, we can. If we add it now and decide it's too verbose, it would
> be
> > awkward to go back and remove documentation later but it's easy to later
> > add in something that's missing.
> >
> > So in short I agree with your assessment :)
> > -Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 22 Jan 2015 03:13:16 +0000
> > >
> > > Mark,
> > > No worries. On the dynamic property tip, I see what you are saying.
> > I was hoping to provide the same level of detail in describing dynamic
> > properties as non-dynamic ones, but right now I see no way to do that.
> > Thats fine, I'll put that part(providing detailed documentation of the
> > dynamic properties) on the back burner.
> > >
> > > On the Validator tip, for each PropertyDescriptor I am getting the
> > list of AllowableValues so that I can say "This property has X, Y and Z
> for
> > valid values". If there are no AllowableValues present I would like to
> > look at the validators and say "This property requires a positive
> > integer". or "This requires a <data size> followed by a <data unit>".
> > Although typing this all out now makes it seem like it might be too
> verbose
> > and cluttered. I think I'll forget this idea and just work with
> > AllowableValues.
> > >
> > > Thanks again,
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Tuesday, January 20, 2015 12:16 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Sorry, I never did respond to this e-mail.
> > > You cannot get the dynamic PropertyDescriptor because the
> > AbstractConfigurableComponent (the base abstract class from which every
> > processor, controller service, reporting task that I know of extends)
> > provides the following method signature:
> > > protected PropertyDescriptor
> getSupportedDynamicPropertyDescriptor(final
> > String propertyDescriptorName)
> > > I.e., there is not a PropertyDescriptor for dynamic properties but
> > rather the descriptor can be different for every dynamic property. For
> > example, consider a Processor that allows dynamic properties but for
> every
> > property X there must be 2 other properties: X.format (any string) and
> > X.size (a non-zero integer). If there were a single PropertyDescriptor,
> > this wouldn't be possible because all dynamic properties would have the
> > same validators, etc. But by making a property descriptor based on the
> > property name, we can easily achieve this.
> > > Regarding the second note: I don't think Validator can provide
> > information about what it is validating because it doesn't know. For
> > instance, PositiveInteger validator just validates that some value is a
> > positive integer. It does not (and I think should not) know the context
> of
> > the value.
> > > If anything here doesn't make sense or if you think it could be done
> > better, let me know and we can discuss further.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Fri, 16 Jan 2015 21:14:41 +0000
> > > >
> > > > Mark & Matt,
> > > > Cool. While I like the idea of @SupportsDynamicProperties,
> > especially if it has a description, I'd also like a guaranteed way to get
> > the dynamic PropertyDescriptor so I can interrogate it further. I'd be
> > interested in knowing whether it supports expression language, and/or how
> > the value of the property is validated.
> > > >
> > > > On that note, I was thinking it may be helpful to add either an
> > annotation or a method to the Validator interface that could provide a
> text
> > description about what it is validating. Is this useful or overkill?
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Friday, January 16, 2015 11:14 AM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > It might make sense to have an annotation for this:
> > > > @SupportsDynamicProperties
> > > > Perhaps even allow developer to provide a string for how they are
> used:
> > > > @SupportsDynamicProperties("Name of dynamic property becomes
> attribute
> > name; value becomes attribute value.")
> > > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6
> > but I have not pushed anything to a branch or anything yet. Basically, I
> > deprecated the existing annotations and created 3 new packages for them.
> > Instead of org.apache.nifi.processor.annotation, we will have:
> > > >
> >
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > > > So the @SupportsDynamicProperties could go under 'behavior' if that
> > makes sense.
> > > >
> > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > > > Subject: Re: Processor documentation plugin
> > > > > From: matt.c.gilman@gmail.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > >
> > > > > Dan,
> > > > >
> > > > > We discussed item 1 awhile back and even created a ticket for it.
> > However,
> > > > > there was always higher priority work that took precedence. I am
> not
> > sure
> > > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> > should.
> > > > > I believe the current mechanism allows the processor to decide if a
> > given
> > > > > dynamic property should be a supported. I think we would need a
> more
> > > > > general indicator that dynamic properties are supported at all to
> > trigger
> > > > > that button state. This would likely help with item 2 as well.
> > > > >
> > > > > Matt
> > > > >
> > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > wrote:
> > > > >
> > > > > > Mark,
> > > > > > Sounds good. I'm on board. I was thinking of having the
> > > > > > ControllerServices/ReportingTask's documentation include an XML
> > template,
> > > > > > but if those will be configurable via the UI then that will not
> be
> > needed.
> > > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll
> get
> > to work
> > > > > > accordingly.
> > > > > >
> > > > > >
> > > > > > Matt/Mark,
> > > > > > Two UI/API questions:
> > > > > > 1) If a processor does not support dynamic properties, why does
> > the UI let
> > > > > > them hit [+] New Property?
> > > > > > 2) If I have an instance of Processor class, I can call
> > > > > > getSupportedProperties() on it to get a list of the
> > required/optional
> > > > > > properties. But I don't have a way to know if that processor
> > supports
> > > > > > dynamic properties or not. Supporting dynamic properties or not
> > was
> > > > > > something I was planning on reflecting in the documentation. Is
> > there any
> > > > > > way to detect this currently?
> > > > > >
> > > > > > Thanks,
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: RE: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > > I think this is great!
> > > > > > I would recommend we slate this for the 0.1.0 version, which
> > includes
> > > > > > putting Controller Services and Reporting Tasks in the UI. But by
> > the same
> > > > > > token I think that makes your first bulletin for "rev 2" more of
> a
> > priority
> > > > > > for "rev 1" if we can get it there. I could pitch in and help if
> > needed.
> > > > > > I'd also recommend you put the Capability Description in for Rev
> 1
> > as
> > > > > > that's arguably the most important part of the docs (which may
> > make the
> > > > > > Tags very easy at that point??)
> > > > > > The rest of what you have listed for "rev 2" i would agree should
> > be there.
> > > > > >
> > > > > > Very excited to see this happening!
> > > > > > Thanks-Mark
> > > > > >
> > > > > > > From: dbress@onyxconsults.com
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > > > >
> > > > > > > Mark et al,
> > > > > > > Ok, I think I hear everyone's concerns and ideas. I will
> > challenge
> > > > > > myself to come up with a solution that makes everyone happy all
> > the time.
> > > > > > >
> > > > > > > The goals of this effort will be:
> > > > > > > - Machine generate as much of the basic(processor
> > description,
> > > > > > property description, relationship description) info as possible.
> > This
> > > > > > will make the developers lives better by storing this information
> > in one
> > > > > > place, and make users lives better by having the "inline"
> > documentation
> > > > > > match the "usage" documentation. This will also ensure that all
> > the
> > > > > > documentation is formatted consistently.
> > > > > > > - Allow the developer to optionally provide additional
> > html and
> > > > > > images that would be linked from the machine generated
> > documentation
> > > > > > > - Provide a way for the documentation to be generated
> > statically
> > > > > > and linked from the Apache NiFi webpage
> > > > > > >
> > > > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > > > - Support documenting ControllerServices and
> > ReportingTasks is a
> > > > > > similar way
> > > > > > > -Maybe leverage existing annotations on these types?
> > E.g
> > > > > > @Tags, @CapabilityDescription...
> > > > > > > - Provide new 'documentation' annotations that would
> allow
> > a
> > > > > > developer to describe the FlowFile attributes a processor uses on
> > input,
> > > > > > and which attributes a processor modifies on output
> > > > > > > - Provide new 'documentation' annotations that describe
> the
> > > > > > expected input format of the flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > whether
> > > > > > the processor modifies the flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > the format
> > > > > > of the outbound flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > other NiFi
> > > > > > artifacts(processors, controller services, reporting tasks) that
> > are
> > > > > > related to this one, e.g. "See Also ..."
> > > > > > > - Consider if/how current Processor annotations should be
> > > > > > displayed (@SideEffectFree, @TriggerSerially,
> @TriggerWhenEmpty...,
> > > > > > @EventDriven)
> > > > > > >
> > > > > > >
> > > > > > > High level implementation plan:
> > > > > > > - Develop a utility class to read an
> AbstractProcessor(and
> > > > > > existing HTML documention) and generated the basic documentation
> > with links
> > > > > > to existing HTML
> > > > > > > - Integrate this class into the framework such that all
> > > > > > documentation displayed within a NiFi instance is generated by
> the
> > utility
> > > > > > > - Integrate this utility into something that can take a
> > NiFi
> > > > > > build, dig through the nars, find the
> > > > > > Processors/ControllerServices/ReportingTasks and generate their
> > appropriate
> > > > > > HTML documentation so that it can be stored somewhere and linked
> > from the
> > > > > > Apache NiFi website.
> > > > > > >
> > > > > > > If this feels like the wrong path, or if you have any other
> > ideas
> > > > > > please speak up. Thanks!
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: RE: Processor documentation plugin
> > > > > > >
> > > > > > > Dan,
> > > > > > > Yes, that's what I'm saying. Right now, it's all done manually
> > by hand,
> > > > > > and it's a pain. The down side i think to doing it via a Maven
> > plugin is
> > > > > > that if the user wants to modify it, they have to rebuild, pull
> the
> > > > > > generated html out of the target directory, and then modify it
> and
> > put it
> > > > > > in the src/main/resources/docs. Then, if they change something
> > they have to
> > > > > > do this again. If it's done on-the-fly, rather than building
> > static content
> > > > > > with a plugin, that doesn't have to happen.
> > > > > > > But you may come up with something much better than I'm
> > envisioning :)
> > > > > > Having a util to do it is a good call though. Then it could be
> > used in a
> > > > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > > > Thanks-Mark
> > > > > > >
> > > > > > > > From: dbress@onyxconsults.com
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > > > >
> > > > > > > > Mark,
> > > > > > > > Sorry.. I missed your reply in the thread.
> > > > > > > >
> > > > > > > > Are you saying that going forward, you want to auto
> > generate the
> > > > > > doc and allow the developer to include a "more info" link? I
> > wanted to
> > > > > > confirm that it doesn't work that way now.
> > > > > > > >
> > > > > > > > I'll think about this a little more. Might write some
> code
> > to
> > > > > > generate the documentation from an AbstractProcessor class, then
> > decide
> > > > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Dan Bress
> > > > > > > > Software Engineer
> > > > > > > > ONYX Consulting Services
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > >
> > > > > > > > Dan,
> > > > > > > >
> > > > > > > > Matt and I discussed this approach before but ultimately
> > decided that
> > > > > > we like the idea of just auto generating the docs on demand with
> a
> > "More
> > > > > > Info" type of link that lets you add more in html format if you
> > care to do
> > > > > > so.
> > > > > > > >
> > > > > > > > Others' thoughts and ideas are welcome though!
> > > > > > > >
> > > > > > > > Sent from my iPhone
> > > > > > > >
> > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Just pulled down the code out of Git and got everything
> up
> > and
> > > > > > running on Windows. This was pretty simple, and the UI looks
> > great under
> > > > > > chrome.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I was thinking of writing a maven plugin to generate the
> > processor
> > > > > > usage HTML documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Looking through the existing processor documentation, it
> > seems
> > > > > > like most of this information comes directly from the code in the
> > > > > > processors, via @CapabilityDescription, and the
> > PropertyDescriptors and
> > > > > > Relationships. Then if your processor only requires this level
> of
> > > > > > documentation in the HTML format, you could tie this plugin to
> the
> > build,
> > > > > > and keep your inline processor documentation in sync with the
> HTML
> > > > > > documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Dan Bress
> > > > > > > > > Software Engineer
> > > > > > > > > ONYX Consulting Services
> > > > > >
> >
>
Re: Processor documentation plugin
Posted by Jennifer Barnabee <je...@gmail.com>.
Dan,
It looks good. Assuming we decide to move forward with it, we'll want to
change the boilerplate paragraph under Properties to suit what you have a
little better. I propose something like the following, but it may need to
change as things evolve. Please see what you think...
In the list below, the names of required properties appear in *bold*. Any
other properties (not in bold) are considered optional. The table also
indicates any default values, whether a property supports the NiFi
Expression Language (or simply EL), and whether a property is considered
"sensitive", meaning that its value will be encrypted. Before entering a
value in a sensitive property, ensure that the *nifi.properties* file has
an entry for the property
*nifi.sensitive.props.key.*
-Jenn
On Sun, Jan 25, 2015 at 9:01 AM, Daniel Bress <db...@onyxconsults.com>
wrote:
> Tony,
> Thanks for pointing that out. I had a "\" when I needed a "/". I
> fixed this and re uploaded it. Let me know how that looks.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Tony Kurc <tr...@gmail.com>
> Sent: Saturday, January 24, 2015 8:07 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
> For whatever reason, when I follow the hyperlinks, the last "slash" is a \
> instead of a / and I get a 404 (
>
> http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html
> ).
>
>
>
>
> On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > All,
> > I've generated the first round of documentation I'd like everyone to
> > look at to see if they like this new format. Its really only slightly
> > changed from the current format(using tables instead of lists for
> > Properties and Relationships), but its all machine generated, except for
> > the stuff linked by "Additional Details".
> >
> > Again, this would be generated by the app at startup. I'd also like
> to
> > have this generated sometime during the build and linked from the
> website.
> > For now I just pulled that generated html and uploaded it to my github
> page
> > for your review.
> >
> > http://danbress.github.io/generated-documentation/components/
> >
> > Let me know what you think! I'm open to any ideas/suggestions.
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Thursday, January 22, 2015 8:16 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > OK, now I understand where you're coming from. Sorry, I misunderstood the
> > idea before. If you think it would be too verbose then I think we should
> > hold off for now. I think it's best to keep it simple. If we want to add
> it
> > in later, we can. If we add it now and decide it's too verbose, it would
> be
> > awkward to go back and remove documentation later but it's easy to later
> > add in something that's missing.
> >
> > So in short I agree with your assessment :)
> > -Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 22 Jan 2015 03:13:16 +0000
> > >
> > > Mark,
> > > No worries. On the dynamic property tip, I see what you are saying.
> > I was hoping to provide the same level of detail in describing dynamic
> > properties as non-dynamic ones, but right now I see no way to do that.
> > Thats fine, I'll put that part(providing detailed documentation of the
> > dynamic properties) on the back burner.
> > >
> > > On the Validator tip, for each PropertyDescriptor I am getting the
> > list of AllowableValues so that I can say "This property has X, Y and Z
> for
> > valid values". If there are no AllowableValues present I would like to
> > look at the validators and say "This property requires a positive
> > integer". or "This requires a <data size> followed by a <data unit>".
> > Although typing this all out now makes it seem like it might be too
> verbose
> > and cluttered. I think I'll forget this idea and just work with
> > AllowableValues.
> > >
> > > Thanks again,
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Tuesday, January 20, 2015 12:16 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Sorry, I never did respond to this e-mail.
> > > You cannot get the dynamic PropertyDescriptor because the
> > AbstractConfigurableComponent (the base abstract class from which every
> > processor, controller service, reporting task that I know of extends)
> > provides the following method signature:
> > > protected PropertyDescriptor
> getSupportedDynamicPropertyDescriptor(final
> > String propertyDescriptorName)
> > > I.e., there is not a PropertyDescriptor for dynamic properties but
> > rather the descriptor can be different for every dynamic property. For
> > example, consider a Processor that allows dynamic properties but for
> every
> > property X there must be 2 other properties: X.format (any string) and
> > X.size (a non-zero integer). If there were a single PropertyDescriptor,
> > this wouldn't be possible because all dynamic properties would have the
> > same validators, etc. But by making a property descriptor based on the
> > property name, we can easily achieve this.
> > > Regarding the second note: I don't think Validator can provide
> > information about what it is validating because it doesn't know. For
> > instance, PositiveInteger validator just validates that some value is a
> > positive integer. It does not (and I think should not) know the context
> of
> > the value.
> > > If anything here doesn't make sense or if you think it could be done
> > better, let me know and we can discuss further.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Fri, 16 Jan 2015 21:14:41 +0000
> > > >
> > > > Mark & Matt,
> > > > Cool. While I like the idea of @SupportsDynamicProperties,
> > especially if it has a description, I'd also like a guaranteed way to get
> > the dynamic PropertyDescriptor so I can interrogate it further. I'd be
> > interested in knowing whether it supports expression language, and/or how
> > the value of the property is validated.
> > > >
> > > > On that note, I was thinking it may be helpful to add either an
> > annotation or a method to the Validator interface that could provide a
> text
> > description about what it is validating. Is this useful or overkill?
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Friday, January 16, 2015 11:14 AM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > It might make sense to have an annotation for this:
> > > > @SupportsDynamicProperties
> > > > Perhaps even allow developer to provide a string for how they are
> used:
> > > > @SupportsDynamicProperties("Name of dynamic property becomes
> attribute
> > name; value becomes attribute value.")
> > > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6
> > but I have not pushed anything to a branch or anything yet. Basically, I
> > deprecated the existing annotations and created 3 new packages for them.
> > Instead of org.apache.nifi.processor.annotation, we will have:
> > > >
> >
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > > > So the @SupportsDynamicProperties could go under 'behavior' if that
> > makes sense.
> > > >
> > > > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > > > Subject: Re: Processor documentation plugin
> > > > > From: matt.c.gilman@gmail.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > >
> > > > > Dan,
> > > > >
> > > > > We discussed item 1 awhile back and even created a ticket for it.
> > However,
> > > > > there was always higher priority work that took precedence. I am
> not
> > sure
> > > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> > should.
> > > > > I believe the current mechanism allows the processor to decide if a
> > given
> > > > > dynamic property should be a supported. I think we would need a
> more
> > > > > general indicator that dynamic properties are supported at all to
> > trigger
> > > > > that button state. This would likely help with item 2 as well.
> > > > >
> > > > > Matt
> > > > >
> > > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > wrote:
> > > > >
> > > > > > Mark,
> > > > > > Sounds good. I'm on board. I was thinking of having the
> > > > > > ControllerServices/ReportingTask's documentation include an XML
> > template,
> > > > > > but if those will be configurable via the UI then that will not
> be
> > needed.
> > > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll
> get
> > to work
> > > > > > accordingly.
> > > > > >
> > > > > >
> > > > > > Matt/Mark,
> > > > > > Two UI/API questions:
> > > > > > 1) If a processor does not support dynamic properties, why does
> > the UI let
> > > > > > them hit [+] New Property?
> > > > > > 2) If I have an instance of Processor class, I can call
> > > > > > getSupportedProperties() on it to get a list of the
> > required/optional
> > > > > > properties. But I don't have a way to know if that processor
> > supports
> > > > > > dynamic properties or not. Supporting dynamic properties or not
> > was
> > > > > > something I was planning on reflecting in the documentation. Is
> > there any
> > > > > > way to detect this currently?
> > > > > >
> > > > > > Thanks,
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: RE: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > > I think this is great!
> > > > > > I would recommend we slate this for the 0.1.0 version, which
> > includes
> > > > > > putting Controller Services and Reporting Tasks in the UI. But by
> > the same
> > > > > > token I think that makes your first bulletin for "rev 2" more of
> a
> > priority
> > > > > > for "rev 1" if we can get it there. I could pitch in and help if
> > needed.
> > > > > > I'd also recommend you put the Capability Description in for Rev
> 1
> > as
> > > > > > that's arguably the most important part of the docs (which may
> > make the
> > > > > > Tags very easy at that point??)
> > > > > > The rest of what you have listed for "rev 2" i would agree should
> > be there.
> > > > > >
> > > > > > Very excited to see this happening!
> > > > > > Thanks-Mark
> > > > > >
> > > > > > > From: dbress@onyxconsults.com
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > > > >
> > > > > > > Mark et al,
> > > > > > > Ok, I think I hear everyone's concerns and ideas. I will
> > challenge
> > > > > > myself to come up with a solution that makes everyone happy all
> > the time.
> > > > > > >
> > > > > > > The goals of this effort will be:
> > > > > > > - Machine generate as much of the basic(processor
> > description,
> > > > > > property description, relationship description) info as possible.
> > This
> > > > > > will make the developers lives better by storing this information
> > in one
> > > > > > place, and make users lives better by having the "inline"
> > documentation
> > > > > > match the "usage" documentation. This will also ensure that all
> > the
> > > > > > documentation is formatted consistently.
> > > > > > > - Allow the developer to optionally provide additional
> > html and
> > > > > > images that would be linked from the machine generated
> > documentation
> > > > > > > - Provide a way for the documentation to be generated
> > statically
> > > > > > and linked from the Apache NiFi webpage
> > > > > > >
> > > > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > > > - Support documenting ControllerServices and
> > ReportingTasks is a
> > > > > > similar way
> > > > > > > -Maybe leverage existing annotations on these types?
> > E.g
> > > > > > @Tags, @CapabilityDescription...
> > > > > > > - Provide new 'documentation' annotations that would
> allow
> > a
> > > > > > developer to describe the FlowFile attributes a processor uses on
> > input,
> > > > > > and which attributes a processor modifies on output
> > > > > > > - Provide new 'documentation' annotations that describe
> the
> > > > > > expected input format of the flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > whether
> > > > > > the processor modifies the flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > the format
> > > > > > of the outbound flow file content
> > > > > > > - Provide new 'documentation' annotations that describe
> > other NiFi
> > > > > > artifacts(processors, controller services, reporting tasks) that
> > are
> > > > > > related to this one, e.g. "See Also ..."
> > > > > > > - Consider if/how current Processor annotations should be
> > > > > > displayed (@SideEffectFree, @TriggerSerially,
> @TriggerWhenEmpty...,
> > > > > > @EventDriven)
> > > > > > >
> > > > > > >
> > > > > > > High level implementation plan:
> > > > > > > - Develop a utility class to read an
> AbstractProcessor(and
> > > > > > existing HTML documention) and generated the basic documentation
> > with links
> > > > > > to existing HTML
> > > > > > > - Integrate this class into the framework such that all
> > > > > > documentation displayed within a NiFi instance is generated by
> the
> > utility
> > > > > > > - Integrate this utility into something that can take a
> > NiFi
> > > > > > build, dig through the nars, find the
> > > > > > Processors/ControllerServices/ReportingTasks and generate their
> > appropriate
> > > > > > HTML documentation so that it can be stored somewhere and linked
> > from the
> > > > > > Apache NiFi website.
> > > > > > >
> > > > > > > If this feels like the wrong path, or if you have any other
> > ideas
> > > > > > please speak up. Thanks!
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: RE: Processor documentation plugin
> > > > > > >
> > > > > > > Dan,
> > > > > > > Yes, that's what I'm saying. Right now, it's all done manually
> > by hand,
> > > > > > and it's a pain. The down side i think to doing it via a Maven
> > plugin is
> > > > > > that if the user wants to modify it, they have to rebuild, pull
> the
> > > > > > generated html out of the target directory, and then modify it
> and
> > put it
> > > > > > in the src/main/resources/docs. Then, if they change something
> > they have to
> > > > > > do this again. If it's done on-the-fly, rather than building
> > static content
> > > > > > with a plugin, that doesn't have to happen.
> > > > > > > But you may come up with something much better than I'm
> > envisioning :)
> > > > > > Having a util to do it is a good call though. Then it could be
> > used in a
> > > > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > > > Thanks-Mark
> > > > > > >
> > > > > > > > From: dbress@onyxconsults.com
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > > > >
> > > > > > > > Mark,
> > > > > > > > Sorry.. I missed your reply in the thread.
> > > > > > > >
> > > > > > > > Are you saying that going forward, you want to auto
> > generate the
> > > > > > doc and allow the developer to include a "more info" link? I
> > wanted to
> > > > > > confirm that it doesn't work that way now.
> > > > > > > >
> > > > > > > > I'll think about this a little more. Might write some
> code
> > to
> > > > > > generate the documentation from an AbstractProcessor class, then
> > decide
> > > > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Dan Bress
> > > > > > > > Software Engineer
> > > > > > > > ONYX Consulting Services
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > >
> > > > > > > > Dan,
> > > > > > > >
> > > > > > > > Matt and I discussed this approach before but ultimately
> > decided that
> > > > > > we like the idea of just auto generating the docs on demand with
> a
> > "More
> > > > > > Info" type of link that lets you add more in html format if you
> > care to do
> > > > > > so.
> > > > > > > >
> > > > > > > > Others' thoughts and ideas are welcome though!
> > > > > > > >
> > > > > > > > Sent from my iPhone
> > > > > > > >
> > > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > Just pulled down the code out of Git and got everything
> up
> > and
> > > > > > running on Windows. This was pretty simple, and the UI looks
> > great under
> > > > > > chrome.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I was thinking of writing a maven plugin to generate the
> > processor
> > > > > > usage HTML documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Looking through the existing processor documentation, it
> > seems
> > > > > > like most of this information comes directly from the code in the
> > > > > > processors, via @CapabilityDescription, and the
> > PropertyDescriptors and
> > > > > > Relationships. Then if your processor only requires this level
> of
> > > > > > documentation in the HTML format, you could tie this plugin to
> the
> > build,
> > > > > > and keep your inline processor documentation in sync with the
> HTML
> > > > > > documentation.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Dan Bress
> > > > > > > > > Software Engineer
> > > > > > > > > ONYX Consulting Services
> > > > > >
> >
>
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Tony,
Thanks for pointing that out. I had a "\" when I needed a "/". I fixed this and re uploaded it. Let me know how that looks.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Tony Kurc <tr...@gmail.com>
Sent: Saturday, January 24, 2015 8:07 PM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Dan,
For whatever reason, when I follow the hyperlinks, the last "slash" is a \
instead of a / and I get a 404 (
http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html).
On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <db...@onyxconsults.com>
wrote:
> All,
> I've generated the first round of documentation I'd like everyone to
> look at to see if they like this new format. Its really only slightly
> changed from the current format(using tables instead of lists for
> Properties and Relationships), but its all machine generated, except for
> the stuff linked by "Additional Details".
>
> Again, this would be generated by the app at startup. I'd also like to
> have this generated sometime during the build and linked from the website.
> For now I just pulled that generated html and uploaded it to my github page
> for your review.
>
> http://danbress.github.io/generated-documentation/components/
>
> Let me know what you think! I'm open to any ideas/suggestions.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Thursday, January 22, 2015 8:16 AM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> OK, now I understand where you're coming from. Sorry, I misunderstood the
> idea before. If you think it would be too verbose then I think we should
> hold off for now. I think it's best to keep it simple. If we want to add it
> in later, we can. If we add it now and decide it's too verbose, it would be
> awkward to go back and remove documentation later but it's easy to later
> add in something that's missing.
>
> So in short I agree with your assessment :)
> -Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Thu, 22 Jan 2015 03:13:16 +0000
> >
> > Mark,
> > No worries. On the dynamic property tip, I see what you are saying.
> I was hoping to provide the same level of detail in describing dynamic
> properties as non-dynamic ones, but right now I see no way to do that.
> Thats fine, I'll put that part(providing detailed documentation of the
> dynamic properties) on the back burner.
> >
> > On the Validator tip, for each PropertyDescriptor I am getting the
> list of AllowableValues so that I can say "This property has X, Y and Z for
> valid values". If there are no AllowableValues present I would like to
> look at the validators and say "This property requires a positive
> integer". or "This requires a <data size> followed by a <data unit>".
> Although typing this all out now makes it seem like it might be too verbose
> and cluttered. I think I'll forget this idea and just work with
> AllowableValues.
> >
> > Thanks again,
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Tuesday, January 20, 2015 12:16 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > Sorry, I never did respond to this e-mail.
> > You cannot get the dynamic PropertyDescriptor because the
> AbstractConfigurableComponent (the base abstract class from which every
> processor, controller service, reporting task that I know of extends)
> provides the following method signature:
> > protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final
> String propertyDescriptorName)
> > I.e., there is not a PropertyDescriptor for dynamic properties but
> rather the descriptor can be different for every dynamic property. For
> example, consider a Processor that allows dynamic properties but for every
> property X there must be 2 other properties: X.format (any string) and
> X.size (a non-zero integer). If there were a single PropertyDescriptor,
> this wouldn't be possible because all dynamic properties would have the
> same validators, etc. But by making a property descriptor based on the
> property name, we can easily achieve this.
> > Regarding the second note: I don't think Validator can provide
> information about what it is validating because it doesn't know. For
> instance, PositiveInteger validator just validates that some value is a
> positive integer. It does not (and I think should not) know the context of
> the value.
> > If anything here doesn't make sense or if you think it could be done
> better, let me know and we can discuss further.
> > Thanks-Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Fri, 16 Jan 2015 21:14:41 +0000
> > >
> > > Mark & Matt,
> > > Cool. While I like the idea of @SupportsDynamicProperties,
> especially if it has a description, I'd also like a guaranteed way to get
> the dynamic PropertyDescriptor so I can interrogate it further. I'd be
> interested in knowing whether it supports expression language, and/or how
> the value of the property is validated.
> > >
> > > On that note, I was thinking it may be helpful to add either an
> annotation or a method to the Validator interface that could provide a text
> description about what it is validating. Is this useful or overkill?
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Friday, January 16, 2015 11:14 AM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > It might make sense to have an annotation for this:
> > > @SupportsDynamicProperties
> > > Perhaps even allow developer to provide a string for how they are used:
> > > @SupportsDynamicProperties("Name of dynamic property becomes attribute
> name; value becomes attribute value.")
> > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6
> but I have not pushed anything to a branch or anything yet. Basically, I
> deprecated the existing annotations and created 3 new packages for them.
> Instead of org.apache.nifi.processor.annotation, we will have:
> > >
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > > So the @SupportsDynamicProperties could go under 'behavior' if that
> makes sense.
> > >
> > > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > > Subject: Re: Processor documentation plugin
> > > > From: matt.c.gilman@gmail.com
> > > > To: dev@nifi.incubator.apache.org
> > > >
> > > > Dan,
> > > >
> > > > We discussed item 1 awhile back and even created a ticket for it.
> However,
> > > > there was always higher priority work that took precedence. I am not
> sure
> > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> should.
> > > > I believe the current mechanism allows the processor to decide if a
> given
> > > > dynamic property should be a supported. I think we would need a more
> > > > general indicator that dynamic properties are supported at all to
> trigger
> > > > that button state. This would likely help with item 2 as well.
> > > >
> > > > Matt
> > > >
> > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <
> dbress@onyxconsults.com>
> > > > wrote:
> > > >
> > > > > Mark,
> > > > > Sounds good. I'm on board. I was thinking of having the
> > > > > ControllerServices/ReportingTask's documentation include an XML
> template,
> > > > > but if those will be configurable via the UI then that will not be
> needed.
> > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get
> to work
> > > > > accordingly.
> > > > >
> > > > >
> > > > > Matt/Mark,
> > > > > Two UI/API questions:
> > > > > 1) If a processor does not support dynamic properties, why does
> the UI let
> > > > > them hit [+] New Property?
> > > > > 2) If I have an instance of Processor class, I can call
> > > > > getSupportedProperties() on it to get a list of the
> required/optional
> > > > > properties. But I don't have a way to know if that processor
> supports
> > > > > dynamic properties or not. Supporting dynamic properties or not
> was
> > > > > something I was planning on reflecting in the documentation. Is
> there any
> > > > > way to detect this currently?
> > > > >
> > > > > Thanks,
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: RE: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > > I think this is great!
> > > > > I would recommend we slate this for the 0.1.0 version, which
> includes
> > > > > putting Controller Services and Reporting Tasks in the UI. But by
> the same
> > > > > token I think that makes your first bulletin for "rev 2" more of a
> priority
> > > > > for "rev 1" if we can get it there. I could pitch in and help if
> needed.
> > > > > I'd also recommend you put the Capability Description in for Rev 1
> as
> > > > > that's arguably the most important part of the docs (which may
> make the
> > > > > Tags very easy at that point??)
> > > > > The rest of what you have listed for "rev 2" i would agree should
> be there.
> > > > >
> > > > > Very excited to see this happening!
> > > > > Thanks-Mark
> > > > >
> > > > > > From: dbress@onyxconsults.com
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > > >
> > > > > > Mark et al,
> > > > > > Ok, I think I hear everyone's concerns and ideas. I will
> challenge
> > > > > myself to come up with a solution that makes everyone happy all
> the time.
> > > > > >
> > > > > > The goals of this effort will be:
> > > > > > - Machine generate as much of the basic(processor
> description,
> > > > > property description, relationship description) info as possible.
> This
> > > > > will make the developers lives better by storing this information
> in one
> > > > > place, and make users lives better by having the "inline"
> documentation
> > > > > match the "usage" documentation. This will also ensure that all
> the
> > > > > documentation is formatted consistently.
> > > > > > - Allow the developer to optionally provide additional
> html and
> > > > > images that would be linked from the machine generated
> documentation
> > > > > > - Provide a way for the documentation to be generated
> statically
> > > > > and linked from the Apache NiFi webpage
> > > > > >
> > > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > > - Support documenting ControllerServices and
> ReportingTasks is a
> > > > > similar way
> > > > > > -Maybe leverage existing annotations on these types?
> E.g
> > > > > @Tags, @CapabilityDescription...
> > > > > > - Provide new 'documentation' annotations that would allow
> a
> > > > > developer to describe the FlowFile attributes a processor uses on
> input,
> > > > > and which attributes a processor modifies on output
> > > > > > - Provide new 'documentation' annotations that describe the
> > > > > expected input format of the flow file content
> > > > > > - Provide new 'documentation' annotations that describe
> whether
> > > > > the processor modifies the flow file content
> > > > > > - Provide new 'documentation' annotations that describe
> the format
> > > > > of the outbound flow file content
> > > > > > - Provide new 'documentation' annotations that describe
> other NiFi
> > > > > artifacts(processors, controller services, reporting tasks) that
> are
> > > > > related to this one, e.g. "See Also ..."
> > > > > > - Consider if/how current Processor annotations should be
> > > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > > > @EventDriven)
> > > > > >
> > > > > >
> > > > > > High level implementation plan:
> > > > > > - Develop a utility class to read an AbstractProcessor(and
> > > > > existing HTML documention) and generated the basic documentation
> with links
> > > > > to existing HTML
> > > > > > - Integrate this class into the framework such that all
> > > > > documentation displayed within a NiFi instance is generated by the
> utility
> > > > > > - Integrate this utility into something that can take a
> NiFi
> > > > > build, dig through the nars, find the
> > > > > Processors/ControllerServices/ReportingTasks and generate their
> appropriate
> > > > > HTML documentation so that it can be stored somewhere and linked
> from the
> > > > > Apache NiFi website.
> > > > > >
> > > > > > If this feels like the wrong path, or if you have any other
> ideas
> > > > > please speak up. Thanks!
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: RE: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > > Yes, that's what I'm saying. Right now, it's all done manually
> by hand,
> > > > > and it's a pain. The down side i think to doing it via a Maven
> plugin is
> > > > > that if the user wants to modify it, they have to rebuild, pull the
> > > > > generated html out of the target directory, and then modify it and
> put it
> > > > > in the src/main/resources/docs. Then, if they change something
> they have to
> > > > > do this again. If it's done on-the-fly, rather than building
> static content
> > > > > with a plugin, that doesn't have to happen.
> > > > > > But you may come up with something much better than I'm
> envisioning :)
> > > > > Having a util to do it is a good call though. Then it could be
> used in a
> > > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > > Thanks-Mark
> > > > > >
> > > > > > > From: dbress@onyxconsults.com
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > > >
> > > > > > > Mark,
> > > > > > > Sorry.. I missed your reply in the thread.
> > > > > > >
> > > > > > > Are you saying that going forward, you want to auto
> generate the
> > > > > doc and allow the developer to include a "more info" link? I
> wanted to
> > > > > confirm that it doesn't work that way now.
> > > > > > >
> > > > > > > I'll think about this a little more. Might write some code
> to
> > > > > generate the documentation from an AbstractProcessor class, then
> decide
> > > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > >
> > > > > > > Dan,
> > > > > > >
> > > > > > > Matt and I discussed this approach before but ultimately
> decided that
> > > > > we like the idea of just auto generating the docs on demand with a
> "More
> > > > > Info" type of link that lets you add more in html format if you
> care to do
> > > > > so.
> > > > > > >
> > > > > > > Others' thoughts and ideas are welcome though!
> > > > > > >
> > > > > > > Sent from my iPhone
> > > > > > >
> > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> dbress@onyxconsults.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Just pulled down the code out of Git and got everything up
> and
> > > > > running on Windows. This was pretty simple, and the UI looks
> great under
> > > > > chrome.
> > > > > > > >
> > > > > > > >
> > > > > > > > I was thinking of writing a maven plugin to generate the
> processor
> > > > > usage HTML documentation.
> > > > > > > >
> > > > > > > >
> > > > > > > > Looking through the existing processor documentation, it
> seems
> > > > > like most of this information comes directly from the code in the
> > > > > processors, via @CapabilityDescription, and the
> PropertyDescriptors and
> > > > > Relationships. Then if your processor only requires this level of
> > > > > documentation in the HTML format, you could tie this plugin to the
> build,
> > > > > and keep your inline processor documentation in sync with the HTML
> > > > > documentation.
> > > > > > > >
> > > > > > > >
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > >
> > > > > > > > Dan Bress
> > > > > > > > Software Engineer
> > > > > > > > ONYX Consulting Services
> > > > >
>
Re: Processor documentation plugin
Posted by Tony Kurc <tr...@gmail.com>.
Dan,
For whatever reason, when I follow the hyperlinks, the last "slash" is a \
instead of a / and I get a 404 (
http://danbress.github.io/generated-documentation/components/org.apache.nifi.processors.standard.InvokeHTTP%5Cindex.html).
On Sat, Jan 24, 2015 at 1:01 PM, Daniel Bress <db...@onyxconsults.com>
wrote:
> All,
> I've generated the first round of documentation I'd like everyone to
> look at to see if they like this new format. Its really only slightly
> changed from the current format(using tables instead of lists for
> Properties and Relationships), but its all machine generated, except for
> the stuff linked by "Additional Details".
>
> Again, this would be generated by the app at startup. I'd also like to
> have this generated sometime during the build and linked from the website.
> For now I just pulled that generated html and uploaded it to my github page
> for your review.
>
> http://danbress.github.io/generated-documentation/components/
>
> Let me know what you think! I'm open to any ideas/suggestions.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Thursday, January 22, 2015 8:16 AM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> OK, now I understand where you're coming from. Sorry, I misunderstood the
> idea before. If you think it would be too verbose then I think we should
> hold off for now. I think it's best to keep it simple. If we want to add it
> in later, we can. If we add it now and decide it's too verbose, it would be
> awkward to go back and remove documentation later but it's easy to later
> add in something that's missing.
>
> So in short I agree with your assessment :)
> -Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Thu, 22 Jan 2015 03:13:16 +0000
> >
> > Mark,
> > No worries. On the dynamic property tip, I see what you are saying.
> I was hoping to provide the same level of detail in describing dynamic
> properties as non-dynamic ones, but right now I see no way to do that.
> Thats fine, I'll put that part(providing detailed documentation of the
> dynamic properties) on the back burner.
> >
> > On the Validator tip, for each PropertyDescriptor I am getting the
> list of AllowableValues so that I can say "This property has X, Y and Z for
> valid values". If there are no AllowableValues present I would like to
> look at the validators and say "This property requires a positive
> integer". or "This requires a <data size> followed by a <data unit>".
> Although typing this all out now makes it seem like it might be too verbose
> and cluttered. I think I'll forget this idea and just work with
> AllowableValues.
> >
> > Thanks again,
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Tuesday, January 20, 2015 12:16 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > Sorry, I never did respond to this e-mail.
> > You cannot get the dynamic PropertyDescriptor because the
> AbstractConfigurableComponent (the base abstract class from which every
> processor, controller service, reporting task that I know of extends)
> provides the following method signature:
> > protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final
> String propertyDescriptorName)
> > I.e., there is not a PropertyDescriptor for dynamic properties but
> rather the descriptor can be different for every dynamic property. For
> example, consider a Processor that allows dynamic properties but for every
> property X there must be 2 other properties: X.format (any string) and
> X.size (a non-zero integer). If there were a single PropertyDescriptor,
> this wouldn't be possible because all dynamic properties would have the
> same validators, etc. But by making a property descriptor based on the
> property name, we can easily achieve this.
> > Regarding the second note: I don't think Validator can provide
> information about what it is validating because it doesn't know. For
> instance, PositiveInteger validator just validates that some value is a
> positive integer. It does not (and I think should not) know the context of
> the value.
> > If anything here doesn't make sense or if you think it could be done
> better, let me know and we can discuss further.
> > Thanks-Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Fri, 16 Jan 2015 21:14:41 +0000
> > >
> > > Mark & Matt,
> > > Cool. While I like the idea of @SupportsDynamicProperties,
> especially if it has a description, I'd also like a guaranteed way to get
> the dynamic PropertyDescriptor so I can interrogate it further. I'd be
> interested in knowing whether it supports expression language, and/or how
> the value of the property is validated.
> > >
> > > On that note, I was thinking it may be helpful to add either an
> annotation or a method to the Validator interface that could provide a text
> description about what it is validating. Is this useful or overkill?
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Friday, January 16, 2015 11:14 AM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > It might make sense to have an annotation for this:
> > > @SupportsDynamicProperties
> > > Perhaps even allow developer to provide a string for how they are used:
> > > @SupportsDynamicProperties("Name of dynamic property becomes attribute
> name; value becomes attribute value.")
> > > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6
> but I have not pushed anything to a branch or anything yet. Basically, I
> deprecated the existing annotations and created 3 new packages for them.
> Instead of org.apache.nifi.processor.annotation, we will have:
> > >
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > > So the @SupportsDynamicProperties could go under 'behavior' if that
> makes sense.
> > >
> > > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > > Subject: Re: Processor documentation plugin
> > > > From: matt.c.gilman@gmail.com
> > > > To: dev@nifi.incubator.apache.org
> > > >
> > > > Dan,
> > > >
> > > > We discussed item 1 awhile back and even created a ticket for it.
> However,
> > > > there was always higher priority work that took precedence. I am not
> sure
> > > > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> should.
> > > > I believe the current mechanism allows the processor to decide if a
> given
> > > > dynamic property should be a supported. I think we would need a more
> > > > general indicator that dynamic properties are supported at all to
> trigger
> > > > that button state. This would likely help with item 2 as well.
> > > >
> > > > Matt
> > > >
> > > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <
> dbress@onyxconsults.com>
> > > > wrote:
> > > >
> > > > > Mark,
> > > > > Sounds good. I'm on board. I was thinking of having the
> > > > > ControllerServices/ReportingTask's documentation include an XML
> template,
> > > > > but if those will be configurable via the UI then that will not be
> needed.
> > > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get
> to work
> > > > > accordingly.
> > > > >
> > > > >
> > > > > Matt/Mark,
> > > > > Two UI/API questions:
> > > > > 1) If a processor does not support dynamic properties, why does
> the UI let
> > > > > them hit [+] New Property?
> > > > > 2) If I have an instance of Processor class, I can call
> > > > > getSupportedProperties() on it to get a list of the
> required/optional
> > > > > properties. But I don't have a way to know if that processor
> supports
> > > > > dynamic properties or not. Supporting dynamic properties or not
> was
> > > > > something I was planning on reflecting in the documentation. Is
> there any
> > > > > way to detect this currently?
> > > > >
> > > > > Thanks,
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: RE: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > > I think this is great!
> > > > > I would recommend we slate this for the 0.1.0 version, which
> includes
> > > > > putting Controller Services and Reporting Tasks in the UI. But by
> the same
> > > > > token I think that makes your first bulletin for "rev 2" more of a
> priority
> > > > > for "rev 1" if we can get it there. I could pitch in and help if
> needed.
> > > > > I'd also recommend you put the Capability Description in for Rev 1
> as
> > > > > that's arguably the most important part of the docs (which may
> make the
> > > > > Tags very easy at that point??)
> > > > > The rest of what you have listed for "rev 2" i would agree should
> be there.
> > > > >
> > > > > Very excited to see this happening!
> > > > > Thanks-Mark
> > > > >
> > > > > > From: dbress@onyxconsults.com
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > > >
> > > > > > Mark et al,
> > > > > > Ok, I think I hear everyone's concerns and ideas. I will
> challenge
> > > > > myself to come up with a solution that makes everyone happy all
> the time.
> > > > > >
> > > > > > The goals of this effort will be:
> > > > > > - Machine generate as much of the basic(processor
> description,
> > > > > property description, relationship description) info as possible.
> This
> > > > > will make the developers lives better by storing this information
> in one
> > > > > place, and make users lives better by having the "inline"
> documentation
> > > > > match the "usage" documentation. This will also ensure that all
> the
> > > > > documentation is formatted consistently.
> > > > > > - Allow the developer to optionally provide additional
> html and
> > > > > images that would be linked from the machine generated
> documentation
> > > > > > - Provide a way for the documentation to be generated
> statically
> > > > > and linked from the Apache NiFi webpage
> > > > > >
> > > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > > - Support documenting ControllerServices and
> ReportingTasks is a
> > > > > similar way
> > > > > > -Maybe leverage existing annotations on these types?
> E.g
> > > > > @Tags, @CapabilityDescription...
> > > > > > - Provide new 'documentation' annotations that would allow
> a
> > > > > developer to describe the FlowFile attributes a processor uses on
> input,
> > > > > and which attributes a processor modifies on output
> > > > > > - Provide new 'documentation' annotations that describe the
> > > > > expected input format of the flow file content
> > > > > > - Provide new 'documentation' annotations that describe
> whether
> > > > > the processor modifies the flow file content
> > > > > > - Provide new 'documentation' annotations that describe
> the format
> > > > > of the outbound flow file content
> > > > > > - Provide new 'documentation' annotations that describe
> other NiFi
> > > > > artifacts(processors, controller services, reporting tasks) that
> are
> > > > > related to this one, e.g. "See Also ..."
> > > > > > - Consider if/how current Processor annotations should be
> > > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > > > @EventDriven)
> > > > > >
> > > > > >
> > > > > > High level implementation plan:
> > > > > > - Develop a utility class to read an AbstractProcessor(and
> > > > > existing HTML documention) and generated the basic documentation
> with links
> > > > > to existing HTML
> > > > > > - Integrate this class into the framework such that all
> > > > > documentation displayed within a NiFi instance is generated by the
> utility
> > > > > > - Integrate this utility into something that can take a
> NiFi
> > > > > build, dig through the nars, find the
> > > > > Processors/ControllerServices/ReportingTasks and generate their
> appropriate
> > > > > HTML documentation so that it can be stored somewhere and linked
> from the
> > > > > Apache NiFi website.
> > > > > >
> > > > > > If this feels like the wrong path, or if you have any other
> ideas
> > > > > please speak up. Thanks!
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: RE: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > > Yes, that's what I'm saying. Right now, it's all done manually
> by hand,
> > > > > and it's a pain. The down side i think to doing it via a Maven
> plugin is
> > > > > that if the user wants to modify it, they have to rebuild, pull the
> > > > > generated html out of the target directory, and then modify it and
> put it
> > > > > in the src/main/resources/docs. Then, if they change something
> they have to
> > > > > do this again. If it's done on-the-fly, rather than building
> static content
> > > > > with a plugin, that doesn't have to happen.
> > > > > > But you may come up with something much better than I'm
> envisioning :)
> > > > > Having a util to do it is a good call though. Then it could be
> used in a
> > > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > > Thanks-Mark
> > > > > >
> > > > > > > From: dbress@onyxconsults.com
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > > >
> > > > > > > Mark,
> > > > > > > Sorry.. I missed your reply in the thread.
> > > > > > >
> > > > > > > Are you saying that going forward, you want to auto
> generate the
> > > > > doc and allow the developer to include a "more info" link? I
> wanted to
> > > > > confirm that it doesn't work that way now.
> > > > > > >
> > > > > > > I'll think about this a little more. Might write some code
> to
> > > > > generate the documentation from an AbstractProcessor class, then
> decide
> > > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > > To: dev@nifi.incubator.apache.org
> > > > > > > Subject: Re: Processor documentation plugin
> > > > > > >
> > > > > > > Dan,
> > > > > > >
> > > > > > > Matt and I discussed this approach before but ultimately
> decided that
> > > > > we like the idea of just auto generating the docs on demand with a
> "More
> > > > > Info" type of link that lets you add more in html format if you
> care to do
> > > > > so.
> > > > > > >
> > > > > > > Others' thoughts and ideas are welcome though!
> > > > > > >
> > > > > > > Sent from my iPhone
> > > > > > >
> > > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> dbress@onyxconsults.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Just pulled down the code out of Git and got everything up
> and
> > > > > running on Windows. This was pretty simple, and the UI looks
> great under
> > > > > chrome.
> > > > > > > >
> > > > > > > >
> > > > > > > > I was thinking of writing a maven plugin to generate the
> processor
> > > > > usage HTML documentation.
> > > > > > > >
> > > > > > > >
> > > > > > > > Looking through the existing processor documentation, it
> seems
> > > > > like most of this information comes directly from the code in the
> > > > > processors, via @CapabilityDescription, and the
> PropertyDescriptors and
> > > > > Relationships. Then if your processor only requires this level of
> > > > > documentation in the HTML format, you could tie this plugin to the
> build,
> > > > > and keep your inline processor documentation in sync with the HTML
> > > > > documentation.
> > > > > > > >
> > > > > > > >
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > >
> > > > > > > > Dan Bress
> > > > > > > > Software Engineer
> > > > > > > > ONYX Consulting Services
> > > > >
>
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
All,
I've generated the first round of documentation I'd like everyone to look at to see if they like this new format. Its really only slightly changed from the current format(using tables instead of lists for Properties and Relationships), but its all machine generated, except for the stuff linked by "Additional Details".
Again, this would be generated by the app at startup. I'd also like to have this generated sometime during the build and linked from the website. For now I just pulled that generated html and uploaded it to my github page for your review.
http://danbress.github.io/generated-documentation/components/
Let me know what you think! I'm open to any ideas/suggestions.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Thursday, January 22, 2015 8:16 AM
To: dev@nifi.incubator.apache.org
Subject: RE: Processor documentation plugin
Dan,
OK, now I understand where you're coming from. Sorry, I misunderstood the idea before. If you think it would be too verbose then I think we should hold off for now. I think it's best to keep it simple. If we want to add it in later, we can. If we add it now and decide it's too verbose, it would be awkward to go back and remove documentation later but it's easy to later add in something that's missing.
So in short I agree with your assessment :)
-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Thu, 22 Jan 2015 03:13:16 +0000
>
> Mark,
> No worries. On the dynamic property tip, I see what you are saying. I was hoping to provide the same level of detail in describing dynamic properties as non-dynamic ones, but right now I see no way to do that. Thats fine, I'll put that part(providing detailed documentation of the dynamic properties) on the back burner.
>
> On the Validator tip, for each PropertyDescriptor I am getting the list of AllowableValues so that I can say "This property has X, Y and Z for valid values". If there are no AllowableValues present I would like to look at the validators and say "This property requires a positive integer". or "This requires a <data size> followed by a <data unit>". Although typing this all out now makes it seem like it might be too verbose and cluttered. I think I'll forget this idea and just work with AllowableValues.
>
> Thanks again,
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Tuesday, January 20, 2015 12:16 PM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> Sorry, I never did respond to this e-mail.
> You cannot get the dynamic PropertyDescriptor because the AbstractConfigurableComponent (the base abstract class from which every processor, controller service, reporting task that I know of extends) provides the following method signature:
> protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName)
> I.e., there is not a PropertyDescriptor for dynamic properties but rather the descriptor can be different for every dynamic property. For example, consider a Processor that allows dynamic properties but for every property X there must be 2 other properties: X.format (any string) and X.size (a non-zero integer). If there were a single PropertyDescriptor, this wouldn't be possible because all dynamic properties would have the same validators, etc. But by making a property descriptor based on the property name, we can easily achieve this.
> Regarding the second note: I don't think Validator can provide information about what it is validating because it doesn't know. For instance, PositiveInteger validator just validates that some value is a positive integer. It does not (and I think should not) know the context of the value.
> If anything here doesn't make sense or if you think it could be done better, let me know and we can discuss further.
> Thanks-Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Fri, 16 Jan 2015 21:14:41 +0000
> >
> > Mark & Matt,
> > Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
> >
> > On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Friday, January 16, 2015 11:14 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > It might make sense to have an annotation for this:
> > @SupportsDynamicProperties
> > Perhaps even allow developer to provide a string for how they are used:
> > @SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
> > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
> > org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
> >
> > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > Subject: Re: Processor documentation plugin
> > > From: matt.c.gilman@gmail.com
> > > To: dev@nifi.incubator.apache.org
> > >
> > > Dan,
> > >
> > > We discussed item 1 awhile back and even created a ticket for it. However,
> > > there was always higher priority work that took precedence. I am not sure
> > > if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> > > I believe the current mechanism allows the processor to decide if a given
> > > dynamic property should be a supported. I think we would need a more
> > > general indicator that dynamic properties are supported at all to trigger
> > > that button state. This would likely help with item 2 as well.
> > >
> > > Matt
> > >
> > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > > wrote:
> > >
> > > > Mark,
> > > > Sounds good. I'm on board. I was thinking of having the
> > > > ControllerServices/ReportingTask's documentation include an XML template,
> > > > but if those will be configurable via the UI then that will not be needed.
> > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > > > accordingly.
> > > >
> > > >
> > > > Matt/Mark,
> > > > Two UI/API questions:
> > > > 1) If a processor does not support dynamic properties, why does the UI let
> > > > them hit [+] New Property?
> > > > 2) If I have an instance of Processor class, I can call
> > > > getSupportedProperties() on it to get a list of the required/optional
> > > > properties. But I don't have a way to know if that processor supports
> > > > dynamic properties or not. Supporting dynamic properties or not was
> > > > something I was planning on reflecting in the documentation. Is there any
> > > > way to detect this currently?
> > > >
> > > > Thanks,
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > I think this is great!
> > > > I would recommend we slate this for the 0.1.0 version, which includes
> > > > putting Controller Services and Reporting Tasks in the UI. But by the same
> > > > token I think that makes your first bulletin for "rev 2" more of a priority
> > > > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > > that's arguably the most important part of the docs (which may make the
> > > > Tags very easy at that point??)
> > > > The rest of what you have listed for "rev 2" i would agree should be there.
> > > >
> > > > Very excited to see this happening!
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > >
> > > > > Mark et al,
> > > > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > > > myself to come up with a solution that makes everyone happy all the time.
> > > > >
> > > > > The goals of this effort will be:
> > > > > - Machine generate as much of the basic(processor description,
> > > > property description, relationship description) info as possible. This
> > > > will make the developers lives better by storing this information in one
> > > > place, and make users lives better by having the "inline" documentation
> > > > match the "usage" documentation. This will also ensure that all the
> > > > documentation is formatted consistently.
> > > > > - Allow the developer to optionally provide additional html and
> > > > images that would be linked from the machine generated documentation
> > > > > - Provide a way for the documentation to be generated statically
> > > > and linked from the Apache NiFi webpage
> > > > >
> > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > - Support documenting ControllerServices and ReportingTasks is a
> > > > similar way
> > > > > -Maybe leverage existing annotations on these types? E.g
> > > > @Tags, @CapabilityDescription...
> > > > > - Provide new 'documentation' annotations that would allow a
> > > > developer to describe the FlowFile attributes a processor uses on input,
> > > > and which attributes a processor modifies on output
> > > > > - Provide new 'documentation' annotations that describe the
> > > > expected input format of the flow file content
> > > > > - Provide new 'documentation' annotations that describe whether
> > > > the processor modifies the flow file content
> > > > > - Provide new 'documentation' annotations that describe the format
> > > > of the outbound flow file content
> > > > > - Provide new 'documentation' annotations that describe other NiFi
> > > > artifacts(processors, controller services, reporting tasks) that are
> > > > related to this one, e.g. "See Also ..."
> > > > > - Consider if/how current Processor annotations should be
> > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > > @EventDriven)
> > > > >
> > > > >
> > > > > High level implementation plan:
> > > > > - Develop a utility class to read an AbstractProcessor(and
> > > > existing HTML documention) and generated the basic documentation with links
> > > > to existing HTML
> > > > > - Integrate this class into the framework such that all
> > > > documentation displayed within a NiFi instance is generated by the utility
> > > > > - Integrate this utility into something that can take a NiFi
> > > > build, dig through the nars, find the
> > > > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > > > HTML documentation so that it can be stored somewhere and linked from the
> > > > Apache NiFi website.
> > > > >
> > > > > If this feels like the wrong path, or if you have any other ideas
> > > > please speak up. Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: RE: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > > > and it's a pain. The down side i think to doing it via a Maven plugin is
> > > > that if the user wants to modify it, they have to rebuild, pull the
> > > > generated html out of the target directory, and then modify it and put it
> > > > in the src/main/resources/docs. Then, if they change something they have to
> > > > do this again. If it's done on-the-fly, rather than building static content
> > > > with a plugin, that doesn't have to happen.
> > > > > But you may come up with something much better than I'm envisioning :)
> > > > Having a util to do it is a good call though. Then it could be used in a
> > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > Thanks-Mark
> > > > >
> > > > > > From: dbress@onyxconsults.com
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > >
> > > > > > Mark,
> > > > > > Sorry.. I missed your reply in the thread.
> > > > > >
> > > > > > Are you saying that going forward, you want to auto generate the
> > > > doc and allow the developer to include a "more info" link? I wanted to
> > > > confirm that it doesn't work that way now.
> > > > > >
> > > > > > I'll think about this a little more. Might write some code to
> > > > generate the documentation from an AbstractProcessor class, then decide
> > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > >
> > > > > > Matt and I discussed this approach before but ultimately decided that
> > > > we like the idea of just auto generating the docs on demand with a "More
> > > > Info" type of link that lets you add more in html format if you care to do
> > > > so.
> > > > > >
> > > > > > Others' thoughts and ideas are welcome though!
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Just pulled down the code out of Git and got everything up and
> > > > running on Windows. This was pretty simple, and the UI looks great under
> > > > chrome.
> > > > > > >
> > > > > > >
> > > > > > > I was thinking of writing a maven plugin to generate the processor
> > > > usage HTML documentation.
> > > > > > >
> > > > > > >
> > > > > > > Looking through the existing processor documentation, it seems
> > > > like most of this information comes directly from the code in the
> > > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > > Relationships. Then if your processor only requires this level of
> > > > documentation in the HTML format, you could tie this plugin to the build,
> > > > and keep your inline processor documentation in sync with the HTML
> > > > documentation.
> > > > > > >
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > >
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
OK, now I understand where you're coming from. Sorry, I misunderstood the idea before. If you think it would be too verbose then I think we should hold off for now. I think it's best to keep it simple. If we want to add it in later, we can. If we add it now and decide it's too verbose, it would be awkward to go back and remove documentation later but it's easy to later add in something that's missing.
So in short I agree with your assessment :)
-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Thu, 22 Jan 2015 03:13:16 +0000
>
> Mark,
> No worries. On the dynamic property tip, I see what you are saying. I was hoping to provide the same level of detail in describing dynamic properties as non-dynamic ones, but right now I see no way to do that. Thats fine, I'll put that part(providing detailed documentation of the dynamic properties) on the back burner.
>
> On the Validator tip, for each PropertyDescriptor I am getting the list of AllowableValues so that I can say "This property has X, Y and Z for valid values". If there are no AllowableValues present I would like to look at the validators and say "This property requires a positive integer". or "This requires a <data size> followed by a <data unit>". Although typing this all out now makes it seem like it might be too verbose and cluttered. I think I'll forget this idea and just work with AllowableValues.
>
> Thanks again,
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Tuesday, January 20, 2015 12:16 PM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> Sorry, I never did respond to this e-mail.
> You cannot get the dynamic PropertyDescriptor because the AbstractConfigurableComponent (the base abstract class from which every processor, controller service, reporting task that I know of extends) provides the following method signature:
> protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName)
> I.e., there is not a PropertyDescriptor for dynamic properties but rather the descriptor can be different for every dynamic property. For example, consider a Processor that allows dynamic properties but for every property X there must be 2 other properties: X.format (any string) and X.size (a non-zero integer). If there were a single PropertyDescriptor, this wouldn't be possible because all dynamic properties would have the same validators, etc. But by making a property descriptor based on the property name, we can easily achieve this.
> Regarding the second note: I don't think Validator can provide information about what it is validating because it doesn't know. For instance, PositiveInteger validator just validates that some value is a positive integer. It does not (and I think should not) know the context of the value.
> If anything here doesn't make sense or if you think it could be done better, let me know and we can discuss further.
> Thanks-Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Fri, 16 Jan 2015 21:14:41 +0000
> >
> > Mark & Matt,
> > Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
> >
> > On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Friday, January 16, 2015 11:14 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > It might make sense to have an annotation for this:
> > @SupportsDynamicProperties
> > Perhaps even allow developer to provide a string for how they are used:
> > @SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
> > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
> > org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
> >
> > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > Subject: Re: Processor documentation plugin
> > > From: matt.c.gilman@gmail.com
> > > To: dev@nifi.incubator.apache.org
> > >
> > > Dan,
> > >
> > > We discussed item 1 awhile back and even created a ticket for it. However,
> > > there was always higher priority work that took precedence. I am not sure
> > > if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> > > I believe the current mechanism allows the processor to decide if a given
> > > dynamic property should be a supported. I think we would need a more
> > > general indicator that dynamic properties are supported at all to trigger
> > > that button state. This would likely help with item 2 as well.
> > >
> > > Matt
> > >
> > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > > wrote:
> > >
> > > > Mark,
> > > > Sounds good. I'm on board. I was thinking of having the
> > > > ControllerServices/ReportingTask's documentation include an XML template,
> > > > but if those will be configurable via the UI then that will not be needed.
> > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > > > accordingly.
> > > >
> > > >
> > > > Matt/Mark,
> > > > Two UI/API questions:
> > > > 1) If a processor does not support dynamic properties, why does the UI let
> > > > them hit [+] New Property?
> > > > 2) If I have an instance of Processor class, I can call
> > > > getSupportedProperties() on it to get a list of the required/optional
> > > > properties. But I don't have a way to know if that processor supports
> > > > dynamic properties or not. Supporting dynamic properties or not was
> > > > something I was planning on reflecting in the documentation. Is there any
> > > > way to detect this currently?
> > > >
> > > > Thanks,
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > I think this is great!
> > > > I would recommend we slate this for the 0.1.0 version, which includes
> > > > putting Controller Services and Reporting Tasks in the UI. But by the same
> > > > token I think that makes your first bulletin for "rev 2" more of a priority
> > > > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > > that's arguably the most important part of the docs (which may make the
> > > > Tags very easy at that point??)
> > > > The rest of what you have listed for "rev 2" i would agree should be there.
> > > >
> > > > Very excited to see this happening!
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > >
> > > > > Mark et al,
> > > > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > > > myself to come up with a solution that makes everyone happy all the time.
> > > > >
> > > > > The goals of this effort will be:
> > > > > - Machine generate as much of the basic(processor description,
> > > > property description, relationship description) info as possible. This
> > > > will make the developers lives better by storing this information in one
> > > > place, and make users lives better by having the "inline" documentation
> > > > match the "usage" documentation. This will also ensure that all the
> > > > documentation is formatted consistently.
> > > > > - Allow the developer to optionally provide additional html and
> > > > images that would be linked from the machine generated documentation
> > > > > - Provide a way for the documentation to be generated statically
> > > > and linked from the Apache NiFi webpage
> > > > >
> > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > - Support documenting ControllerServices and ReportingTasks is a
> > > > similar way
> > > > > -Maybe leverage existing annotations on these types? E.g
> > > > @Tags, @CapabilityDescription...
> > > > > - Provide new 'documentation' annotations that would allow a
> > > > developer to describe the FlowFile attributes a processor uses on input,
> > > > and which attributes a processor modifies on output
> > > > > - Provide new 'documentation' annotations that describe the
> > > > expected input format of the flow file content
> > > > > - Provide new 'documentation' annotations that describe whether
> > > > the processor modifies the flow file content
> > > > > - Provide new 'documentation' annotations that describe the format
> > > > of the outbound flow file content
> > > > > - Provide new 'documentation' annotations that describe other NiFi
> > > > artifacts(processors, controller services, reporting tasks) that are
> > > > related to this one, e.g. "See Also ..."
> > > > > - Consider if/how current Processor annotations should be
> > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > > @EventDriven)
> > > > >
> > > > >
> > > > > High level implementation plan:
> > > > > - Develop a utility class to read an AbstractProcessor(and
> > > > existing HTML documention) and generated the basic documentation with links
> > > > to existing HTML
> > > > > - Integrate this class into the framework such that all
> > > > documentation displayed within a NiFi instance is generated by the utility
> > > > > - Integrate this utility into something that can take a NiFi
> > > > build, dig through the nars, find the
> > > > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > > > HTML documentation so that it can be stored somewhere and linked from the
> > > > Apache NiFi website.
> > > > >
> > > > > If this feels like the wrong path, or if you have any other ideas
> > > > please speak up. Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: RE: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > > > and it's a pain. The down side i think to doing it via a Maven plugin is
> > > > that if the user wants to modify it, they have to rebuild, pull the
> > > > generated html out of the target directory, and then modify it and put it
> > > > in the src/main/resources/docs. Then, if they change something they have to
> > > > do this again. If it's done on-the-fly, rather than building static content
> > > > with a plugin, that doesn't have to happen.
> > > > > But you may come up with something much better than I'm envisioning :)
> > > > Having a util to do it is a good call though. Then it could be used in a
> > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > Thanks-Mark
> > > > >
> > > > > > From: dbress@onyxconsults.com
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > >
> > > > > > Mark,
> > > > > > Sorry.. I missed your reply in the thread.
> > > > > >
> > > > > > Are you saying that going forward, you want to auto generate the
> > > > doc and allow the developer to include a "more info" link? I wanted to
> > > > confirm that it doesn't work that way now.
> > > > > >
> > > > > > I'll think about this a little more. Might write some code to
> > > > generate the documentation from an AbstractProcessor class, then decide
> > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > >
> > > > > > Matt and I discussed this approach before but ultimately decided that
> > > > we like the idea of just auto generating the docs on demand with a "More
> > > > Info" type of link that lets you add more in html format if you care to do
> > > > so.
> > > > > >
> > > > > > Others' thoughts and ideas are welcome though!
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Just pulled down the code out of Git and got everything up and
> > > > running on Windows. This was pretty simple, and the UI looks great under
> > > > chrome.
> > > > > > >
> > > > > > >
> > > > > > > I was thinking of writing a maven plugin to generate the processor
> > > > usage HTML documentation.
> > > > > > >
> > > > > > >
> > > > > > > Looking through the existing processor documentation, it seems
> > > > like most of this information comes directly from the code in the
> > > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > > Relationships. Then if your processor only requires this level of
> > > > documentation in the HTML format, you could tie this plugin to the build,
> > > > and keep your inline processor documentation in sync with the HTML
> > > > documentation.
> > > > > > >
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > >
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Mark,
No worries. On the dynamic property tip, I see what you are saying. I was hoping to provide the same level of detail in describing dynamic properties as non-dynamic ones, but right now I see no way to do that. Thats fine, I'll put that part(providing detailed documentation of the dynamic properties) on the back burner.
On the Validator tip, for each PropertyDescriptor I am getting the list of AllowableValues so that I can say "This property has X, Y and Z for valid values". If there are no AllowableValues present I would like to look at the validators and say "This property requires a positive integer". or "This requires a <data size> followed by a <data unit>". Although typing this all out now makes it seem like it might be too verbose and cluttered. I think I'll forget this idea and just work with AllowableValues.
Thanks again,
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Tuesday, January 20, 2015 12:16 PM
To: dev@nifi.incubator.apache.org
Subject: RE: Processor documentation plugin
Dan,
Sorry, I never did respond to this e-mail.
You cannot get the dynamic PropertyDescriptor because the AbstractConfigurableComponent (the base abstract class from which every processor, controller service, reporting task that I know of extends) provides the following method signature:
protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName)
I.e., there is not a PropertyDescriptor for dynamic properties but rather the descriptor can be different for every dynamic property. For example, consider a Processor that allows dynamic properties but for every property X there must be 2 other properties: X.format (any string) and X.size (a non-zero integer). If there were a single PropertyDescriptor, this wouldn't be possible because all dynamic properties would have the same validators, etc. But by making a property descriptor based on the property name, we can easily achieve this.
Regarding the second note: I don't think Validator can provide information about what it is validating because it doesn't know. For instance, PositiveInteger validator just validates that some value is a positive integer. It does not (and I think should not) know the context of the value.
If anything here doesn't make sense or if you think it could be done better, let me know and we can discuss further.
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Fri, 16 Jan 2015 21:14:41 +0000
>
> Mark & Matt,
> Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
>
> On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Friday, January 16, 2015 11:14 AM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> It might make sense to have an annotation for this:
> @SupportsDynamicProperties
> Perhaps even allow developer to provide a string for how they are used:
> @SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
> I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
>
> > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > Subject: Re: Processor documentation plugin
> > From: matt.c.gilman@gmail.com
> > To: dev@nifi.incubator.apache.org
> >
> > Dan,
> >
> > We discussed item 1 awhile back and even created a ticket for it. However,
> > there was always higher priority work that took precedence. I am not sure
> > if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> > I believe the current mechanism allows the processor to decide if a given
> > dynamic property should be a supported. I think we would need a more
> > general indicator that dynamic properties are supported at all to trigger
> > that button state. This would likely help with item 2 as well.
> >
> > Matt
> >
> > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> >
> > > Mark,
> > > Sounds good. I'm on board. I was thinking of having the
> > > ControllerServices/ReportingTask's documentation include an XML template,
> > > but if those will be configurable via the UI then that will not be needed.
> > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > > accordingly.
> > >
> > >
> > > Matt/Mark,
> > > Two UI/API questions:
> > > 1) If a processor does not support dynamic properties, why does the UI let
> > > them hit [+] New Property?
> > > 2) If I have an instance of Processor class, I can call
> > > getSupportedProperties() on it to get a list of the required/optional
> > > properties. But I don't have a way to know if that processor supports
> > > dynamic properties or not. Supporting dynamic properties or not was
> > > something I was planning on reflecting in the documentation. Is there any
> > > way to detect this currently?
> > >
> > > Thanks,
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > I think this is great!
> > > I would recommend we slate this for the 0.1.0 version, which includes
> > > putting Controller Services and Reporting Tasks in the UI. But by the same
> > > token I think that makes your first bulletin for "rev 2" more of a priority
> > > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > that's arguably the most important part of the docs (which may make the
> > > Tags very easy at that point??)
> > > The rest of what you have listed for "rev 2" i would agree should be there.
> > >
> > > Very excited to see this happening!
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > >
> > > > Mark et al,
> > > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > > myself to come up with a solution that makes everyone happy all the time.
> > > >
> > > > The goals of this effort will be:
> > > > - Machine generate as much of the basic(processor description,
> > > property description, relationship description) info as possible. This
> > > will make the developers lives better by storing this information in one
> > > place, and make users lives better by having the "inline" documentation
> > > match the "usage" documentation. This will also ensure that all the
> > > documentation is formatted consistently.
> > > > - Allow the developer to optionally provide additional html and
> > > images that would be linked from the machine generated documentation
> > > > - Provide a way for the documentation to be generated statically
> > > and linked from the Apache NiFi webpage
> > > >
> > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > - Support documenting ControllerServices and ReportingTasks is a
> > > similar way
> > > > -Maybe leverage existing annotations on these types? E.g
> > > @Tags, @CapabilityDescription...
> > > > - Provide new 'documentation' annotations that would allow a
> > > developer to describe the FlowFile attributes a processor uses on input,
> > > and which attributes a processor modifies on output
> > > > - Provide new 'documentation' annotations that describe the
> > > expected input format of the flow file content
> > > > - Provide new 'documentation' annotations that describe whether
> > > the processor modifies the flow file content
> > > > - Provide new 'documentation' annotations that describe the format
> > > of the outbound flow file content
> > > > - Provide new 'documentation' annotations that describe other NiFi
> > > artifacts(processors, controller services, reporting tasks) that are
> > > related to this one, e.g. "See Also ..."
> > > > - Consider if/how current Processor annotations should be
> > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > @EventDriven)
> > > >
> > > >
> > > > High level implementation plan:
> > > > - Develop a utility class to read an AbstractProcessor(and
> > > existing HTML documention) and generated the basic documentation with links
> > > to existing HTML
> > > > - Integrate this class into the framework such that all
> > > documentation displayed within a NiFi instance is generated by the utility
> > > > - Integrate this utility into something that can take a NiFi
> > > build, dig through the nars, find the
> > > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > > HTML documentation so that it can be stored somewhere and linked from the
> > > Apache NiFi website.
> > > >
> > > > If this feels like the wrong path, or if you have any other ideas
> > > please speak up. Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > > and it's a pain. The down side i think to doing it via a Maven plugin is
> > > that if the user wants to modify it, they have to rebuild, pull the
> > > generated html out of the target directory, and then modify it and put it
> > > in the src/main/resources/docs. Then, if they change something they have to
> > > do this again. If it's done on-the-fly, rather than building static content
> > > with a plugin, that doesn't have to happen.
> > > > But you may come up with something much better than I'm envisioning :)
> > > Having a util to do it is a good call though. Then it could be used in a
> > > maven plugin, by the app, or wherever is most appropriate.
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > >
> > > > > Mark,
> > > > > Sorry.. I missed your reply in the thread.
> > > > >
> > > > > Are you saying that going forward, you want to auto generate the
> > > doc and allow the developer to include a "more info" link? I wanted to
> > > confirm that it doesn't work that way now.
> > > > >
> > > > > I'll think about this a little more. Might write some code to
> > > generate the documentation from an AbstractProcessor class, then decide
> > > later where to integrate it(maven plugin, or somewhere else)
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > >
> > > > > Matt and I discussed this approach before but ultimately decided that
> > > we like the idea of just auto generating the docs on demand with a "More
> > > Info" type of link that lets you add more in html format if you care to do
> > > so.
> > > > >
> > > > > Others' thoughts and ideas are welcome though!
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Just pulled down the code out of Git and got everything up and
> > > running on Windows. This was pretty simple, and the UI looks great under
> > > chrome.
> > > > > >
> > > > > >
> > > > > > I was thinking of writing a maven plugin to generate the processor
> > > usage HTML documentation.
> > > > > >
> > > > > >
> > > > > > Looking through the existing processor documentation, it seems
> > > like most of this information comes directly from the code in the
> > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > Relationships. Then if your processor only requires this level of
> > > documentation in the HTML format, you could tie this plugin to the build,
> > > and keep your inline processor documentation in sync with the HTML
> > > documentation.
> > > > > >
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > >
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
Sorry, I never did respond to this e-mail.
You cannot get the dynamic PropertyDescriptor because the AbstractConfigurableComponent (the base abstract class from which every processor, controller service, reporting task that I know of extends) provides the following method signature:
protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(final String propertyDescriptorName)
I.e., there is not a PropertyDescriptor for dynamic properties but rather the descriptor can be different for every dynamic property. For example, consider a Processor that allows dynamic properties but for every property X there must be 2 other properties: X.format (any string) and X.size (a non-zero integer). If there were a single PropertyDescriptor, this wouldn't be possible because all dynamic properties would have the same validators, etc. But by making a property descriptor based on the property name, we can easily achieve this.
Regarding the second note: I don't think Validator can provide information about what it is validating because it doesn't know. For instance, PositiveInteger validator just validates that some value is a positive integer. It does not (and I think should not) know the context of the value.
If anything here doesn't make sense or if you think it could be done better, let me know and we can discuss further.
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Fri, 16 Jan 2015 21:14:41 +0000
>
> Mark & Matt,
> Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
>
> On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Friday, January 16, 2015 11:14 AM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> It might make sense to have an annotation for this:
> @SupportsDynamicProperties
> Perhaps even allow developer to provide a string for how they are used:
> @SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
> I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
>
> > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > Subject: Re: Processor documentation plugin
> > From: matt.c.gilman@gmail.com
> > To: dev@nifi.incubator.apache.org
> >
> > Dan,
> >
> > We discussed item 1 awhile back and even created a ticket for it. However,
> > there was always higher priority work that took precedence. I am not sure
> > if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> > I believe the current mechanism allows the processor to decide if a given
> > dynamic property should be a supported. I think we would need a more
> > general indicator that dynamic properties are supported at all to trigger
> > that button state. This would likely help with item 2 as well.
> >
> > Matt
> >
> > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> >
> > > Mark,
> > > Sounds good. I'm on board. I was thinking of having the
> > > ControllerServices/ReportingTask's documentation include an XML template,
> > > but if those will be configurable via the UI then that will not be needed.
> > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > > accordingly.
> > >
> > >
> > > Matt/Mark,
> > > Two UI/API questions:
> > > 1) If a processor does not support dynamic properties, why does the UI let
> > > them hit [+] New Property?
> > > 2) If I have an instance of Processor class, I can call
> > > getSupportedProperties() on it to get a list of the required/optional
> > > properties. But I don't have a way to know if that processor supports
> > > dynamic properties or not. Supporting dynamic properties or not was
> > > something I was planning on reflecting in the documentation. Is there any
> > > way to detect this currently?
> > >
> > > Thanks,
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > I think this is great!
> > > I would recommend we slate this for the 0.1.0 version, which includes
> > > putting Controller Services and Reporting Tasks in the UI. But by the same
> > > token I think that makes your first bulletin for "rev 2" more of a priority
> > > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > that's arguably the most important part of the docs (which may make the
> > > Tags very easy at that point??)
> > > The rest of what you have listed for "rev 2" i would agree should be there.
> > >
> > > Very excited to see this happening!
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > >
> > > > Mark et al,
> > > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > > myself to come up with a solution that makes everyone happy all the time.
> > > >
> > > > The goals of this effort will be:
> > > > - Machine generate as much of the basic(processor description,
> > > property description, relationship description) info as possible. This
> > > will make the developers lives better by storing this information in one
> > > place, and make users lives better by having the "inline" documentation
> > > match the "usage" documentation. This will also ensure that all the
> > > documentation is formatted consistently.
> > > > - Allow the developer to optionally provide additional html and
> > > images that would be linked from the machine generated documentation
> > > > - Provide a way for the documentation to be generated statically
> > > and linked from the Apache NiFi webpage
> > > >
> > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > - Support documenting ControllerServices and ReportingTasks is a
> > > similar way
> > > > -Maybe leverage existing annotations on these types? E.g
> > > @Tags, @CapabilityDescription...
> > > > - Provide new 'documentation' annotations that would allow a
> > > developer to describe the FlowFile attributes a processor uses on input,
> > > and which attributes a processor modifies on output
> > > > - Provide new 'documentation' annotations that describe the
> > > expected input format of the flow file content
> > > > - Provide new 'documentation' annotations that describe whether
> > > the processor modifies the flow file content
> > > > - Provide new 'documentation' annotations that describe the format
> > > of the outbound flow file content
> > > > - Provide new 'documentation' annotations that describe other NiFi
> > > artifacts(processors, controller services, reporting tasks) that are
> > > related to this one, e.g. "See Also ..."
> > > > - Consider if/how current Processor annotations should be
> > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > @EventDriven)
> > > >
> > > >
> > > > High level implementation plan:
> > > > - Develop a utility class to read an AbstractProcessor(and
> > > existing HTML documention) and generated the basic documentation with links
> > > to existing HTML
> > > > - Integrate this class into the framework such that all
> > > documentation displayed within a NiFi instance is generated by the utility
> > > > - Integrate this utility into something that can take a NiFi
> > > build, dig through the nars, find the
> > > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > > HTML documentation so that it can be stored somewhere and linked from the
> > > Apache NiFi website.
> > > >
> > > > If this feels like the wrong path, or if you have any other ideas
> > > please speak up. Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > > and it's a pain. The down side i think to doing it via a Maven plugin is
> > > that if the user wants to modify it, they have to rebuild, pull the
> > > generated html out of the target directory, and then modify it and put it
> > > in the src/main/resources/docs. Then, if they change something they have to
> > > do this again. If it's done on-the-fly, rather than building static content
> > > with a plugin, that doesn't have to happen.
> > > > But you may come up with something much better than I'm envisioning :)
> > > Having a util to do it is a good call though. Then it could be used in a
> > > maven plugin, by the app, or wherever is most appropriate.
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > >
> > > > > Mark,
> > > > > Sorry.. I missed your reply in the thread.
> > > > >
> > > > > Are you saying that going forward, you want to auto generate the
> > > doc and allow the developer to include a "more info" link? I wanted to
> > > confirm that it doesn't work that way now.
> > > > >
> > > > > I'll think about this a little more. Might write some code to
> > > generate the documentation from an AbstractProcessor class, then decide
> > > later where to integrate it(maven plugin, or somewhere else)
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > >
> > > > > Matt and I discussed this approach before but ultimately decided that
> > > we like the idea of just auto generating the docs on demand with a "More
> > > Info" type of link that lets you add more in html format if you care to do
> > > so.
> > > > >
> > > > > Others' thoughts and ideas are welcome though!
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Just pulled down the code out of Git and got everything up and
> > > running on Windows. This was pretty simple, and the UI looks great under
> > > chrome.
> > > > > >
> > > > > >
> > > > > > I was thinking of writing a maven plugin to generate the processor
> > > usage HTML documentation.
> > > > > >
> > > > > >
> > > > > > Looking through the existing processor documentation, it seems
> > > like most of this information comes directly from the code in the
> > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > Relationships. Then if your processor only requires this level of
> > > documentation in the HTML format, you could tie this plugin to the build,
> > > and keep your inline processor documentation in sync with the HTML
> > > documentation.
> > > > > >
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > >
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Drew,
The vast majority of the documentation for Processors already exists within the NiFi API. For example, each Property of a Processor is defined with a PropertyDescription that contians a a name, description, whether or not the Expression Language is supported, etc. We did this via the API specifically because we want to use this information from within the User Interface to provide specific information when processors are being configured. Not sure that JavaDoc doclets can meet that need... then again, I don't know a whole lot about doclets. If you think they may provide this sort of functionality, I could look more into it.
Thanks-Mark
> From: drew@apache.org
> Date: Mon, 19 Jan 2015 12:54:43 -0500
> Subject: Re: Processor documentation plugin
> To: dev@nifi.incubator.apache.org
>
> Hi All,
>
> Getting in on this thread a bit late. This is interesting work.
>
> Is this functionality still slated to be rolled into a maven plugin? Could
> this be something that's implemented as a javadoc doclet, or does that api
> tie everything too closely to the static approach?
>
> Drew
>
> On Mon, Jan 19, 2015 at 11:49 AM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > Mark,
> > I've got the API for generating
> > Processor/ControllerService/ReportingTask documentation, and now am looking
> > at integrating that into the Framework during startup. I've got a few
> > questions about how to tie everything together.
> >
> > 1) Where should my code reside in the tree? I was going to create a new
> > jar/project under framework-bundle/framework called documentation.
> > 2) My stuff requires initialized instances of ConfigurableComponents. The
> > reason being most Processors set up their properties in their init()
> > methods. So what's the best way to get an instance of a
> > Processor/ControllerService/Reporting task? I was planning on kicking my
> > stuff out of the nifi-runtime/NiFi class. To get things going I've been
> > using stubbed out versions of the InitializationContext interfaces that I
> > wrote. Should I get the instances from the FlowController?
> >
> > I've got it tied in right now in a maybe hacky way, but the docs looks
> > good. I went through all the existing html, and deleted anything that
> > didn't add additional detail. Anything that did provide additional detail
> > I renamed to "additionalDetails.html" which my code detects and links from
> > the generated documentation. Also having NIFI-6 and NIFI-4 resolved will
> > make the documentation look better for ControllerServices and
> > ReportingTasks.
> >
> > Once I get it integrated and tested a bit I'll add the patch to the
> > ticket. Thanks for your help!
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Daniel Bress <db...@onyxconsults.com>
> > Sent: Friday, January 16, 2015 4:14 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> >
> > Mark & Matt,
> > Cool. While I like the idea of @SupportsDynamicProperties, especially
> > if it has a description, I'd also like a guaranteed way to get the dynamic
> > PropertyDescriptor so I can interrogate it further. I'd be interested in
> > knowing whether it supports expression language, and/or how the value of
> > the property is validated.
> >
> > On that note, I was thinking it may be helpful to add either an annotation
> > or a method to the Validator interface that could provide a text
> > description about what it is validating. Is this useful or overkill?
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Friday, January 16, 2015 11:14 AM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > It might make sense to have an annotation for this:
> > @SupportsDynamicProperties
> > Perhaps even allow developer to provide a string for how they are used:
> > @SupportsDynamicProperties("Name of dynamic property becomes attribute
> > name; value becomes attribute value.")
> > I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I
> > have not pushed anything to a branch or anything yet. Basically, I
> > deprecated the existing annotations and created 3 new packages for them.
> > Instead of org.apache.nifi.processor.annotation, we will have:
> >
> > org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> > So the @SupportsDynamicProperties could go under 'behavior' if that makes
> > sense.
> >
> > > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > > Subject: Re: Processor documentation plugin
> > > From: matt.c.gilman@gmail.com
> > > To: dev@nifi.incubator.apache.org
> > >
> > > Dan,
> > >
> > > We discussed item 1 awhile back and even created a ticket for it.
> > However,
> > > there was always higher priority work that took precedence. I am not sure
> > > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> > should.
> > > I believe the current mechanism allows the processor to decide if a given
> > > dynamic property should be a supported. I think we would need a more
> > > general indicator that dynamic properties are supported at all to trigger
> > > that button state. This would likely help with item 2 as well.
> > >
> > > Matt
> > >
> > > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > > wrote:
> > >
> > > > Mark,
> > > > Sounds good. I'm on board. I was thinking of having the
> > > > ControllerServices/ReportingTask's documentation include an XML
> > template,
> > > > but if those will be configurable via the UI then that will not be
> > needed.
> > > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to
> > work
> > > > accordingly.
> > > >
> > > >
> > > > Matt/Mark,
> > > > Two UI/API questions:
> > > > 1) If a processor does not support dynamic properties, why does the UI
> > let
> > > > them hit [+] New Property?
> > > > 2) If I have an instance of Processor class, I can call
> > > > getSupportedProperties() on it to get a list of the required/optional
> > > > properties. But I don't have a way to know if that processor supports
> > > > dynamic properties or not. Supporting dynamic properties or not was
> > > > something I was planning on reflecting in the documentation. Is there
> > any
> > > > way to detect this currently?
> > > >
> > > > Thanks,
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > I think this is great!
> > > > I would recommend we slate this for the 0.1.0 version, which includes
> > > > putting Controller Services and Reporting Tasks in the UI. But by the
> > same
> > > > token I think that makes your first bulletin for "rev 2" more of a
> > priority
> > > > for "rev 1" if we can get it there. I could pitch in and help if
> > needed.
> > > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > > that's arguably the most important part of the docs (which may make the
> > > > Tags very easy at that point??)
> > > > The rest of what you have listed for "rev 2" i would agree should be
> > there.
> > > >
> > > > Very excited to see this happening!
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > > >
> > > > > Mark et al,
> > > > > Ok, I think I hear everyone's concerns and ideas. I will
> > challenge
> > > > myself to come up with a solution that makes everyone happy all the
> > time.
> > > > >
> > > > > The goals of this effort will be:
> > > > > - Machine generate as much of the basic(processor description,
> > > > property description, relationship description) info as possible. This
> > > > will make the developers lives better by storing this information in
> > one
> > > > place, and make users lives better by having the "inline" documentation
> > > > match the "usage" documentation. This will also ensure that all the
> > > > documentation is formatted consistently.
> > > > > - Allow the developer to optionally provide additional html and
> > > > images that would be linked from the machine generated documentation
> > > > > - Provide a way for the documentation to be generated
> > statically
> > > > and linked from the Apache NiFi webpage
> > > > >
> > > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > > - Support documenting ControllerServices and ReportingTasks is
> > a
> > > > similar way
> > > > > -Maybe leverage existing annotations on these types? E.g
> > > > @Tags, @CapabilityDescription...
> > > > > - Provide new 'documentation' annotations that would allow a
> > > > developer to describe the FlowFile attributes a processor uses on
> > input,
> > > > and which attributes a processor modifies on output
> > > > > - Provide new 'documentation' annotations that describe the
> > > > expected input format of the flow file content
> > > > > - Provide new 'documentation' annotations that describe whether
> > > > the processor modifies the flow file content
> > > > > - Provide new 'documentation' annotations that describe the
> > format
> > > > of the outbound flow file content
> > > > > - Provide new 'documentation' annotations that describe other
> > NiFi
> > > > artifacts(processors, controller services, reporting tasks) that are
> > > > related to this one, e.g. "See Also ..."
> > > > > - Consider if/how current Processor annotations should be
> > > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > > @EventDriven)
> > > > >
> > > > >
> > > > > High level implementation plan:
> > > > > - Develop a utility class to read an AbstractProcessor(and
> > > > existing HTML documention) and generated the basic documentation with
> > links
> > > > to existing HTML
> > > > > - Integrate this class into the framework such that all
> > > > documentation displayed within a NiFi instance is generated by the
> > utility
> > > > > - Integrate this utility into something that can take a NiFi
> > > > build, dig through the nars, find the
> > > > Processors/ControllerServices/ReportingTasks and generate their
> > appropriate
> > > > HTML documentation so that it can be stored somewhere and linked from
> > the
> > > > Apache NiFi website.
> > > > >
> > > > > If this feels like the wrong path, or if you have any other ideas
> > > > please speak up. Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: RE: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > > Yes, that's what I'm saying. Right now, it's all done manually by
> > hand,
> > > > and it's a pain. The down side i think to doing it via a Maven plugin
> > is
> > > > that if the user wants to modify it, they have to rebuild, pull the
> > > > generated html out of the target directory, and then modify it and put
> > it
> > > > in the src/main/resources/docs. Then, if they change something they
> > have to
> > > > do this again. If it's done on-the-fly, rather than building static
> > content
> > > > with a plugin, that doesn't have to happen.
> > > > > But you may come up with something much better than I'm envisioning
> > :)
> > > > Having a util to do it is a good call though. Then it could be used in
> > a
> > > > maven plugin, by the app, or wherever is most appropriate.
> > > > > Thanks-Mark
> > > > >
> > > > > > From: dbress@onyxconsults.com
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > > >
> > > > > > Mark,
> > > > > > Sorry.. I missed your reply in the thread.
> > > > > >
> > > > > > Are you saying that going forward, you want to auto generate the
> > > > doc and allow the developer to include a "more info" link? I wanted to
> > > > confirm that it doesn't work that way now.
> > > > > >
> > > > > > I'll think about this a little more. Might write some code to
> > > > generate the documentation from an AbstractProcessor class, then decide
> > > > later where to integrate it(maven plugin, or somewhere else)
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > > > > >
> > > > > > ________________________________________
> > > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > > To: dev@nifi.incubator.apache.org
> > > > > > Subject: Re: Processor documentation plugin
> > > > > >
> > > > > > Dan,
> > > > > >
> > > > > > Matt and I discussed this approach before but ultimately decided
> > that
> > > > we like the idea of just auto generating the docs on demand with a
> > "More
> > > > Info" type of link that lets you add more in html format if you care
> > to do
> > > > so.
> > > > > >
> > > > > > Others' thoughts and ideas are welcome though!
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> > dbress@onyxconsults.com>
> > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Just pulled down the code out of Git and got everything up and
> > > > running on Windows. This was pretty simple, and the UI looks great
> > under
> > > > chrome.
> > > > > > >
> > > > > > >
> > > > > > > I was thinking of writing a maven plugin to generate the
> > processor
> > > > usage HTML documentation.
> > > > > > >
> > > > > > >
> > > > > > > Looking through the existing processor documentation, it seems
> > > > like most of this information comes directly from the code in the
> > > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > > Relationships. Then if your processor only requires this level of
> > > > documentation in the HTML format, you could tie this plugin to the
> > build,
> > > > and keep your inline processor documentation in sync with the HTML
> > > > documentation.
> > > > > > >
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > >
> > > > > > > Dan Bress
> > > > > > > Software Engineer
> > > > > > > ONYX Consulting Services
> > > >
> >
Re: Processor documentation plugin
Posted by Drew Farris <dr...@apache.org>.
Hi All,
Getting in on this thread a bit late. This is interesting work.
Is this functionality still slated to be rolled into a maven plugin? Could
this be something that's implemented as a javadoc doclet, or does that api
tie everything too closely to the static approach?
Drew
On Mon, Jan 19, 2015 at 11:49 AM, Daniel Bress <db...@onyxconsults.com>
wrote:
> Mark,
> I've got the API for generating
> Processor/ControllerService/ReportingTask documentation, and now am looking
> at integrating that into the Framework during startup. I've got a few
> questions about how to tie everything together.
>
> 1) Where should my code reside in the tree? I was going to create a new
> jar/project under framework-bundle/framework called documentation.
> 2) My stuff requires initialized instances of ConfigurableComponents. The
> reason being most Processors set up their properties in their init()
> methods. So what's the best way to get an instance of a
> Processor/ControllerService/Reporting task? I was planning on kicking my
> stuff out of the nifi-runtime/NiFi class. To get things going I've been
> using stubbed out versions of the InitializationContext interfaces that I
> wrote. Should I get the instances from the FlowController?
>
> I've got it tied in right now in a maybe hacky way, but the docs looks
> good. I went through all the existing html, and deleted anything that
> didn't add additional detail. Anything that did provide additional detail
> I renamed to "additionalDetails.html" which my code detects and links from
> the generated documentation. Also having NIFI-6 and NIFI-4 resolved will
> make the documentation look better for ControllerServices and
> ReportingTasks.
>
> Once I get it integrated and tested a bit I'll add the patch to the
> ticket. Thanks for your help!
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Daniel Bress <db...@onyxconsults.com>
> Sent: Friday, January 16, 2015 4:14 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Mark & Matt,
> Cool. While I like the idea of @SupportsDynamicProperties, especially
> if it has a description, I'd also like a guaranteed way to get the dynamic
> PropertyDescriptor so I can interrogate it further. I'd be interested in
> knowing whether it supports expression language, and/or how the value of
> the property is validated.
>
> On that note, I was thinking it may be helpful to add either an annotation
> or a method to the Validator interface that could provide a text
> description about what it is validating. Is this useful or overkill?
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Friday, January 16, 2015 11:14 AM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> It might make sense to have an annotation for this:
> @SupportsDynamicProperties
> Perhaps even allow developer to provide a string for how they are used:
> @SupportsDynamicProperties("Name of dynamic property becomes attribute
> name; value becomes attribute value.")
> I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I
> have not pushed anything to a branch or anything yet. Basically, I
> deprecated the existing annotations and created 3 new packages for them.
> Instead of org.apache.nifi.processor.annotation, we will have:
>
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> So the @SupportsDynamicProperties could go under 'behavior' if that makes
> sense.
>
> > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > Subject: Re: Processor documentation plugin
> > From: matt.c.gilman@gmail.com
> > To: dev@nifi.incubator.apache.org
> >
> > Dan,
> >
> > We discussed item 1 awhile back and even created a ticket for it.
> However,
> > there was always higher priority work that took precedence. I am not sure
> > if that ticket was ever created in JIRA for Apache NiFi. If not, we
> should.
> > I believe the current mechanism allows the processor to decide if a given
> > dynamic property should be a supported. I think we would need a more
> > general indicator that dynamic properties are supported at all to trigger
> > that button state. This would likely help with item 2 as well.
> >
> > Matt
> >
> > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> >
> > > Mark,
> > > Sounds good. I'm on board. I was thinking of having the
> > > ControllerServices/ReportingTask's documentation include an XML
> template,
> > > but if those will be configurable via the UI then that will not be
> needed.
> > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to
> work
> > > accordingly.
> > >
> > >
> > > Matt/Mark,
> > > Two UI/API questions:
> > > 1) If a processor does not support dynamic properties, why does the UI
> let
> > > them hit [+] New Property?
> > > 2) If I have an instance of Processor class, I can call
> > > getSupportedProperties() on it to get a list of the required/optional
> > > properties. But I don't have a way to know if that processor supports
> > > dynamic properties or not. Supporting dynamic properties or not was
> > > something I was planning on reflecting in the documentation. Is there
> any
> > > way to detect this currently?
> > >
> > > Thanks,
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > I think this is great!
> > > I would recommend we slate this for the 0.1.0 version, which includes
> > > putting Controller Services and Reporting Tasks in the UI. But by the
> same
> > > token I think that makes your first bulletin for "rev 2" more of a
> priority
> > > for "rev 1" if we can get it there. I could pitch in and help if
> needed.
> > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > that's arguably the most important part of the docs (which may make the
> > > Tags very easy at that point??)
> > > The rest of what you have listed for "rev 2" i would agree should be
> there.
> > >
> > > Very excited to see this happening!
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > >
> > > > Mark et al,
> > > > Ok, I think I hear everyone's concerns and ideas. I will
> challenge
> > > myself to come up with a solution that makes everyone happy all the
> time.
> > > >
> > > > The goals of this effort will be:
> > > > - Machine generate as much of the basic(processor description,
> > > property description, relationship description) info as possible. This
> > > will make the developers lives better by storing this information in
> one
> > > place, and make users lives better by having the "inline" documentation
> > > match the "usage" documentation. This will also ensure that all the
> > > documentation is formatted consistently.
> > > > - Allow the developer to optionally provide additional html and
> > > images that would be linked from the machine generated documentation
> > > > - Provide a way for the documentation to be generated
> statically
> > > and linked from the Apache NiFi webpage
> > > >
> > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > - Support documenting ControllerServices and ReportingTasks is
> a
> > > similar way
> > > > -Maybe leverage existing annotations on these types? E.g
> > > @Tags, @CapabilityDescription...
> > > > - Provide new 'documentation' annotations that would allow a
> > > developer to describe the FlowFile attributes a processor uses on
> input,
> > > and which attributes a processor modifies on output
> > > > - Provide new 'documentation' annotations that describe the
> > > expected input format of the flow file content
> > > > - Provide new 'documentation' annotations that describe whether
> > > the processor modifies the flow file content
> > > > - Provide new 'documentation' annotations that describe the
> format
> > > of the outbound flow file content
> > > > - Provide new 'documentation' annotations that describe other
> NiFi
> > > artifacts(processors, controller services, reporting tasks) that are
> > > related to this one, e.g. "See Also ..."
> > > > - Consider if/how current Processor annotations should be
> > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > @EventDriven)
> > > >
> > > >
> > > > High level implementation plan:
> > > > - Develop a utility class to read an AbstractProcessor(and
> > > existing HTML documention) and generated the basic documentation with
> links
> > > to existing HTML
> > > > - Integrate this class into the framework such that all
> > > documentation displayed within a NiFi instance is generated by the
> utility
> > > > - Integrate this utility into something that can take a NiFi
> > > build, dig through the nars, find the
> > > Processors/ControllerServices/ReportingTasks and generate their
> appropriate
> > > HTML documentation so that it can be stored somewhere and linked from
> the
> > > Apache NiFi website.
> > > >
> > > > If this feels like the wrong path, or if you have any other ideas
> > > please speak up. Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > Yes, that's what I'm saying. Right now, it's all done manually by
> hand,
> > > and it's a pain. The down side i think to doing it via a Maven plugin
> is
> > > that if the user wants to modify it, they have to rebuild, pull the
> > > generated html out of the target directory, and then modify it and put
> it
> > > in the src/main/resources/docs. Then, if they change something they
> have to
> > > do this again. If it's done on-the-fly, rather than building static
> content
> > > with a plugin, that doesn't have to happen.
> > > > But you may come up with something much better than I'm envisioning
> :)
> > > Having a util to do it is a good call though. Then it could be used in
> a
> > > maven plugin, by the app, or wherever is most appropriate.
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > >
> > > > > Mark,
> > > > > Sorry.. I missed your reply in the thread.
> > > > >
> > > > > Are you saying that going forward, you want to auto generate the
> > > doc and allow the developer to include a "more info" link? I wanted to
> > > confirm that it doesn't work that way now.
> > > > >
> > > > > I'll think about this a little more. Might write some code to
> > > generate the documentation from an AbstractProcessor class, then decide
> > > later where to integrate it(maven plugin, or somewhere else)
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > >
> > > > > Matt and I discussed this approach before but ultimately decided
> that
> > > we like the idea of just auto generating the docs on demand with a
> "More
> > > Info" type of link that lets you add more in html format if you care
> to do
> > > so.
> > > > >
> > > > > Others' thoughts and ideas are welcome though!
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <
> dbress@onyxconsults.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Just pulled down the code out of Git and got everything up and
> > > running on Windows. This was pretty simple, and the UI looks great
> under
> > > chrome.
> > > > > >
> > > > > >
> > > > > > I was thinking of writing a maven plugin to generate the
> processor
> > > usage HTML documentation.
> > > > > >
> > > > > >
> > > > > > Looking through the existing processor documentation, it seems
> > > like most of this information comes directly from the code in the
> > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > Relationships. Then if your processor only requires this level of
> > > documentation in the HTML format, you could tie this plugin to the
> build,
> > > and keep your inline processor documentation in sync with the HTML
> > > documentation.
> > > > > >
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > >
>
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
Excellent. The FlowController definitely can handle this. However, it's not cool to require that a FlowController be available for this if we want to extract the documentation for the website. I would actually very much consider just providing a "mock" Initialization Context. The context basically provides an identifier, a logger, and controller services. For your purposes, you don't care about the identifier or anything that is logged really. Access to Controller Services really isn't important, either. It may result in invalid processors, but that's fine because these instances won't even be added to the flow.
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Mon, 19 Jan 2015 16:49:47 +0000
>
> Mark,
> I've got the API for generating Processor/ControllerService/ReportingTask documentation, and now am looking at integrating that into the Framework during startup. I've got a few questions about how to tie everything together.
>
> 1) Where should my code reside in the tree? I was going to create a new jar/project under framework-bundle/framework called documentation.
> 2) My stuff requires initialized instances of ConfigurableComponents. The reason being most Processors set up their properties in their init() methods. So what's the best way to get an instance of a Processor/ControllerService/Reporting task? I was planning on kicking my stuff out of the nifi-runtime/NiFi class. To get things going I've been using stubbed out versions of the InitializationContext interfaces that I wrote. Should I get the instances from the FlowController?
>
> I've got it tied in right now in a maybe hacky way, but the docs looks good. I went through all the existing html, and deleted anything that didn't add additional detail. Anything that did provide additional detail I renamed to "additionalDetails.html" which my code detects and links from the generated documentation. Also having NIFI-6 and NIFI-4 resolved will make the documentation look better for ControllerServices and ReportingTasks.
>
> Once I get it integrated and tested a bit I'll add the patch to the ticket. Thanks for your help!
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Daniel Bress <db...@onyxconsults.com>
> Sent: Friday, January 16, 2015 4:14 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Mark & Matt,
> Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
>
> On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Friday, January 16, 2015 11:14 AM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> It might make sense to have an annotation for this:
> @SupportsDynamicProperties
> Perhaps even allow developer to provide a string for how they are used:
> @SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
> I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
> org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
> So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
>
> > Date: Fri, 16 Jan 2015 11:09:28 -0500
> > Subject: Re: Processor documentation plugin
> > From: matt.c.gilman@gmail.com
> > To: dev@nifi.incubator.apache.org
> >
> > Dan,
> >
> > We discussed item 1 awhile back and even created a ticket for it. However,
> > there was always higher priority work that took precedence. I am not sure
> > if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> > I believe the current mechanism allows the processor to decide if a given
> > dynamic property should be a supported. I think we would need a more
> > general indicator that dynamic properties are supported at all to trigger
> > that button state. This would likely help with item 2 as well.
> >
> > Matt
> >
> > On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> >
> > > Mark,
> > > Sounds good. I'm on board. I was thinking of having the
> > > ControllerServices/ReportingTask's documentation include an XML template,
> > > but if those will be configurable via the UI then that will not be needed.
> > > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > > accordingly.
> > >
> > >
> > > Matt/Mark,
> > > Two UI/API questions:
> > > 1) If a processor does not support dynamic properties, why does the UI let
> > > them hit [+] New Property?
> > > 2) If I have an instance of Processor class, I can call
> > > getSupportedProperties() on it to get a list of the required/optional
> > > properties. But I don't have a way to know if that processor supports
> > > dynamic properties or not. Supporting dynamic properties or not was
> > > something I was planning on reflecting in the documentation. Is there any
> > > way to detect this currently?
> > >
> > > Thanks,
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Wednesday, January 14, 2015 8:51 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > I think this is great!
> > > I would recommend we slate this for the 0.1.0 version, which includes
> > > putting Controller Services and Reporting Tasks in the UI. But by the same
> > > token I think that makes your first bulletin for "rev 2" more of a priority
> > > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > > I'd also recommend you put the Capability Description in for Rev 1 as
> > > that's arguably the most important part of the docs (which may make the
> > > Tags very easy at that point??)
> > > The rest of what you have listed for "rev 2" i would agree should be there.
> > >
> > > Very excited to see this happening!
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > > >
> > > > Mark et al,
> > > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > > myself to come up with a solution that makes everyone happy all the time.
> > > >
> > > > The goals of this effort will be:
> > > > - Machine generate as much of the basic(processor description,
> > > property description, relationship description) info as possible. This
> > > will make the developers lives better by storing this information in one
> > > place, and make users lives better by having the "inline" documentation
> > > match the "usage" documentation. This will also ensure that all the
> > > documentation is formatted consistently.
> > > > - Allow the developer to optionally provide additional html and
> > > images that would be linked from the machine generated documentation
> > > > - Provide a way for the documentation to be generated statically
> > > and linked from the Apache NiFi webpage
> > > >
> > > > A few other 'nice-to-have' goals(probably for rev 2):
> > > > - Support documenting ControllerServices and ReportingTasks is a
> > > similar way
> > > > -Maybe leverage existing annotations on these types? E.g
> > > @Tags, @CapabilityDescription...
> > > > - Provide new 'documentation' annotations that would allow a
> > > developer to describe the FlowFile attributes a processor uses on input,
> > > and which attributes a processor modifies on output
> > > > - Provide new 'documentation' annotations that describe the
> > > expected input format of the flow file content
> > > > - Provide new 'documentation' annotations that describe whether
> > > the processor modifies the flow file content
> > > > - Provide new 'documentation' annotations that describe the format
> > > of the outbound flow file content
> > > > - Provide new 'documentation' annotations that describe other NiFi
> > > artifacts(processors, controller services, reporting tasks) that are
> > > related to this one, e.g. "See Also ..."
> > > > - Consider if/how current Processor annotations should be
> > > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > > @EventDriven)
> > > >
> > > >
> > > > High level implementation plan:
> > > > - Develop a utility class to read an AbstractProcessor(and
> > > existing HTML documention) and generated the basic documentation with links
> > > to existing HTML
> > > > - Integrate this class into the framework such that all
> > > documentation displayed within a NiFi instance is generated by the utility
> > > > - Integrate this utility into something that can take a NiFi
> > > build, dig through the nars, find the
> > > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > > HTML documentation so that it can be stored somewhere and linked from the
> > > Apache NiFi website.
> > > >
> > > > If this feels like the wrong path, or if you have any other ideas
> > > please speak up. Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Monday, January 12, 2015 3:40 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: RE: Processor documentation plugin
> > > >
> > > > Dan,
> > > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > > and it's a pain. The down side i think to doing it via a Maven plugin is
> > > that if the user wants to modify it, they have to rebuild, pull the
> > > generated html out of the target directory, and then modify it and put it
> > > in the src/main/resources/docs. Then, if they change something they have to
> > > do this again. If it's done on-the-fly, rather than building static content
> > > with a plugin, that doesn't have to happen.
> > > > But you may come up with something much better than I'm envisioning :)
> > > Having a util to do it is a good call though. Then it could be used in a
> > > maven plugin, by the app, or wherever is most appropriate.
> > > > Thanks-Mark
> > > >
> > > > > From: dbress@onyxconsults.com
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > > >
> > > > > Mark,
> > > > > Sorry.. I missed your reply in the thread.
> > > > >
> > > > > Are you saying that going forward, you want to auto generate the
> > > doc and allow the developer to include a "more info" link? I wanted to
> > > confirm that it doesn't work that way now.
> > > > >
> > > > > I'll think about this a little more. Might write some code to
> > > generate the documentation from an AbstractProcessor class, then decide
> > > later where to integrate it(maven plugin, or somewhere else)
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> > > > >
> > > > > ________________________________________
> > > > > From: Mark Payne <ma...@hotmail.com>
> > > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > > To: dev@nifi.incubator.apache.org
> > > > > Subject: Re: Processor documentation plugin
> > > > >
> > > > > Dan,
> > > > >
> > > > > Matt and I discussed this approach before but ultimately decided that
> > > we like the idea of just auto generating the docs on demand with a "More
> > > Info" type of link that lets you add more in html format if you care to do
> > > so.
> > > > >
> > > > > Others' thoughts and ideas are welcome though!
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > > wrote:
> > > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > Just pulled down the code out of Git and got everything up and
> > > running on Windows. This was pretty simple, and the UI looks great under
> > > chrome.
> > > > > >
> > > > > >
> > > > > > I was thinking of writing a maven plugin to generate the processor
> > > usage HTML documentation.
> > > > > >
> > > > > >
> > > > > > Looking through the existing processor documentation, it seems
> > > like most of this information comes directly from the code in the
> > > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > > Relationships. Then if your processor only requires this level of
> > > documentation in the HTML format, you could tie this plugin to the build,
> > > and keep your inline processor documentation in sync with the HTML
> > > documentation.
> > > > > >
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > >
> > > > > > Dan Bress
> > > > > > Software Engineer
> > > > > > ONYX Consulting Services
> > >
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Mark,
I've got the API for generating Processor/ControllerService/ReportingTask documentation, and now am looking at integrating that into the Framework during startup. I've got a few questions about how to tie everything together.
1) Where should my code reside in the tree? I was going to create a new jar/project under framework-bundle/framework called documentation.
2) My stuff requires initialized instances of ConfigurableComponents. The reason being most Processors set up their properties in their init() methods. So what's the best way to get an instance of a Processor/ControllerService/Reporting task? I was planning on kicking my stuff out of the nifi-runtime/NiFi class. To get things going I've been using stubbed out versions of the InitializationContext interfaces that I wrote. Should I get the instances from the FlowController?
I've got it tied in right now in a maybe hacky way, but the docs looks good. I went through all the existing html, and deleted anything that didn't add additional detail. Anything that did provide additional detail I renamed to "additionalDetails.html" which my code detects and links from the generated documentation. Also having NIFI-6 and NIFI-4 resolved will make the documentation look better for ControllerServices and ReportingTasks.
Once I get it integrated and tested a bit I'll add the patch to the ticket. Thanks for your help!
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Daniel Bress <db...@onyxconsults.com>
Sent: Friday, January 16, 2015 4:14 PM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Mark & Matt,
Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Friday, January 16, 2015 11:14 AM
To: dev@nifi.incubator.apache.org
Subject: RE: Processor documentation plugin
It might make sense to have an annotation for this:
@SupportsDynamicProperties
Perhaps even allow developer to provide a string for how they are used:
@SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
> Date: Fri, 16 Jan 2015 11:09:28 -0500
> Subject: Re: Processor documentation plugin
> From: matt.c.gilman@gmail.com
> To: dev@nifi.incubator.apache.org
>
> Dan,
>
> We discussed item 1 awhile back and even created a ticket for it. However,
> there was always higher priority work that took precedence. I am not sure
> if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> I believe the current mechanism allows the processor to decide if a given
> dynamic property should be a supported. I think we would need a more
> general indicator that dynamic properties are supported at all to trigger
> that button state. This would likely help with item 2 as well.
>
> Matt
>
> On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > Mark,
> > Sounds good. I'm on board. I was thinking of having the
> > ControllerServices/ReportingTask's documentation include an XML template,
> > but if those will be configurable via the UI then that will not be needed.
> > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > accordingly.
> >
> >
> > Matt/Mark,
> > Two UI/API questions:
> > 1) If a processor does not support dynamic properties, why does the UI let
> > them hit [+] New Property?
> > 2) If I have an instance of Processor class, I can call
> > getSupportedProperties() on it to get a list of the required/optional
> > properties. But I don't have a way to know if that processor supports
> > dynamic properties or not. Supporting dynamic properties or not was
> > something I was planning on reflecting in the documentation. Is there any
> > way to detect this currently?
> >
> > Thanks,
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Wednesday, January 14, 2015 8:51 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > I think this is great!
> > I would recommend we slate this for the 0.1.0 version, which includes
> > putting Controller Services and Reporting Tasks in the UI. But by the same
> > token I think that makes your first bulletin for "rev 2" more of a priority
> > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > I'd also recommend you put the Capability Description in for Rev 1 as
> > that's arguably the most important part of the docs (which may make the
> > Tags very easy at that point??)
> > The rest of what you have listed for "rev 2" i would agree should be there.
> >
> > Very excited to see this happening!
> > Thanks-Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > >
> > > Mark et al,
> > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > myself to come up with a solution that makes everyone happy all the time.
> > >
> > > The goals of this effort will be:
> > > - Machine generate as much of the basic(processor description,
> > property description, relationship description) info as possible. This
> > will make the developers lives better by storing this information in one
> > place, and make users lives better by having the "inline" documentation
> > match the "usage" documentation. This will also ensure that all the
> > documentation is formatted consistently.
> > > - Allow the developer to optionally provide additional html and
> > images that would be linked from the machine generated documentation
> > > - Provide a way for the documentation to be generated statically
> > and linked from the Apache NiFi webpage
> > >
> > > A few other 'nice-to-have' goals(probably for rev 2):
> > > - Support documenting ControllerServices and ReportingTasks is a
> > similar way
> > > -Maybe leverage existing annotations on these types? E.g
> > @Tags, @CapabilityDescription...
> > > - Provide new 'documentation' annotations that would allow a
> > developer to describe the FlowFile attributes a processor uses on input,
> > and which attributes a processor modifies on output
> > > - Provide new 'documentation' annotations that describe the
> > expected input format of the flow file content
> > > - Provide new 'documentation' annotations that describe whether
> > the processor modifies the flow file content
> > > - Provide new 'documentation' annotations that describe the format
> > of the outbound flow file content
> > > - Provide new 'documentation' annotations that describe other NiFi
> > artifacts(processors, controller services, reporting tasks) that are
> > related to this one, e.g. "See Also ..."
> > > - Consider if/how current Processor annotations should be
> > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > @EventDriven)
> > >
> > >
> > > High level implementation plan:
> > > - Develop a utility class to read an AbstractProcessor(and
> > existing HTML documention) and generated the basic documentation with links
> > to existing HTML
> > > - Integrate this class into the framework such that all
> > documentation displayed within a NiFi instance is generated by the utility
> > > - Integrate this utility into something that can take a NiFi
> > build, dig through the nars, find the
> > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > HTML documentation so that it can be stored somewhere and linked from the
> > Apache NiFi website.
> > >
> > > If this feels like the wrong path, or if you have any other ideas
> > please speak up. Thanks!
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Monday, January 12, 2015 3:40 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > and it's a pain. The down side i think to doing it via a Maven plugin is
> > that if the user wants to modify it, they have to rebuild, pull the
> > generated html out of the target directory, and then modify it and put it
> > in the src/main/resources/docs. Then, if they change something they have to
> > do this again. If it's done on-the-fly, rather than building static content
> > with a plugin, that doesn't have to happen.
> > > But you may come up with something much better than I'm envisioning :)
> > Having a util to do it is a good call though. Then it could be used in a
> > maven plugin, by the app, or wherever is most appropriate.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > >
> > > > Mark,
> > > > Sorry.. I missed your reply in the thread.
> > > >
> > > > Are you saying that going forward, you want to auto generate the
> > doc and allow the developer to include a "more info" link? I wanted to
> > confirm that it doesn't work that way now.
> > > >
> > > > I'll think about this a little more. Might write some code to
> > generate the documentation from an AbstractProcessor class, then decide
> > later where to integrate it(maven plugin, or somewhere else)
> > > >
> > > > Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > >
> > > > Dan,
> > > >
> > > > Matt and I discussed this approach before but ultimately decided that
> > we like the idea of just auto generating the docs on demand with a "More
> > Info" type of link that lets you add more in html format if you care to do
> > so.
> > > >
> > > > Others' thoughts and ideas are welcome though!
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Just pulled down the code out of Git and got everything up and
> > running on Windows. This was pretty simple, and the UI looks great under
> > chrome.
> > > > >
> > > > >
> > > > > I was thinking of writing a maven plugin to generate the processor
> > usage HTML documentation.
> > > > >
> > > > >
> > > > > Looking through the existing processor documentation, it seems
> > like most of this information comes directly from the code in the
> > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > Relationships. Then if your processor only requires this level of
> > documentation in the HTML format, you could tie this plugin to the build,
> > and keep your inline processor documentation in sync with the HTML
> > documentation.
> > > > >
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> >
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Mark & Matt,
Cool. While I like the idea of @SupportsDynamicProperties, especially if it has a description, I'd also like a guaranteed way to get the dynamic PropertyDescriptor so I can interrogate it further. I'd be interested in knowing whether it supports expression language, and/or how the value of the property is validated.
On that note, I was thinking it may be helpful to add either an annotation or a method to the Validator interface that could provide a text description about what it is validating. Is this useful or overkill?
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Friday, January 16, 2015 11:14 AM
To: dev@nifi.incubator.apache.org
Subject: RE: Processor documentation plugin
It might make sense to have an annotation for this:
@SupportsDynamicProperties
Perhaps even allow developer to provide a string for how they are used:
@SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
> Date: Fri, 16 Jan 2015 11:09:28 -0500
> Subject: Re: Processor documentation plugin
> From: matt.c.gilman@gmail.com
> To: dev@nifi.incubator.apache.org
>
> Dan,
>
> We discussed item 1 awhile back and even created a ticket for it. However,
> there was always higher priority work that took precedence. I am not sure
> if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> I believe the current mechanism allows the processor to decide if a given
> dynamic property should be a supported. I think we would need a more
> general indicator that dynamic properties are supported at all to trigger
> that button state. This would likely help with item 2 as well.
>
> Matt
>
> On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > Mark,
> > Sounds good. I'm on board. I was thinking of having the
> > ControllerServices/ReportingTask's documentation include an XML template,
> > but if those will be configurable via the UI then that will not be needed.
> > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > accordingly.
> >
> >
> > Matt/Mark,
> > Two UI/API questions:
> > 1) If a processor does not support dynamic properties, why does the UI let
> > them hit [+] New Property?
> > 2) If I have an instance of Processor class, I can call
> > getSupportedProperties() on it to get a list of the required/optional
> > properties. But I don't have a way to know if that processor supports
> > dynamic properties or not. Supporting dynamic properties or not was
> > something I was planning on reflecting in the documentation. Is there any
> > way to detect this currently?
> >
> > Thanks,
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Wednesday, January 14, 2015 8:51 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > I think this is great!
> > I would recommend we slate this for the 0.1.0 version, which includes
> > putting Controller Services and Reporting Tasks in the UI. But by the same
> > token I think that makes your first bulletin for "rev 2" more of a priority
> > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > I'd also recommend you put the Capability Description in for Rev 1 as
> > that's arguably the most important part of the docs (which may make the
> > Tags very easy at that point??)
> > The rest of what you have listed for "rev 2" i would agree should be there.
> >
> > Very excited to see this happening!
> > Thanks-Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > >
> > > Mark et al,
> > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > myself to come up with a solution that makes everyone happy all the time.
> > >
> > > The goals of this effort will be:
> > > - Machine generate as much of the basic(processor description,
> > property description, relationship description) info as possible. This
> > will make the developers lives better by storing this information in one
> > place, and make users lives better by having the "inline" documentation
> > match the "usage" documentation. This will also ensure that all the
> > documentation is formatted consistently.
> > > - Allow the developer to optionally provide additional html and
> > images that would be linked from the machine generated documentation
> > > - Provide a way for the documentation to be generated statically
> > and linked from the Apache NiFi webpage
> > >
> > > A few other 'nice-to-have' goals(probably for rev 2):
> > > - Support documenting ControllerServices and ReportingTasks is a
> > similar way
> > > -Maybe leverage existing annotations on these types? E.g
> > @Tags, @CapabilityDescription...
> > > - Provide new 'documentation' annotations that would allow a
> > developer to describe the FlowFile attributes a processor uses on input,
> > and which attributes a processor modifies on output
> > > - Provide new 'documentation' annotations that describe the
> > expected input format of the flow file content
> > > - Provide new 'documentation' annotations that describe whether
> > the processor modifies the flow file content
> > > - Provide new 'documentation' annotations that describe the format
> > of the outbound flow file content
> > > - Provide new 'documentation' annotations that describe other NiFi
> > artifacts(processors, controller services, reporting tasks) that are
> > related to this one, e.g. "See Also ..."
> > > - Consider if/how current Processor annotations should be
> > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > @EventDriven)
> > >
> > >
> > > High level implementation plan:
> > > - Develop a utility class to read an AbstractProcessor(and
> > existing HTML documention) and generated the basic documentation with links
> > to existing HTML
> > > - Integrate this class into the framework such that all
> > documentation displayed within a NiFi instance is generated by the utility
> > > - Integrate this utility into something that can take a NiFi
> > build, dig through the nars, find the
> > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > HTML documentation so that it can be stored somewhere and linked from the
> > Apache NiFi website.
> > >
> > > If this feels like the wrong path, or if you have any other ideas
> > please speak up. Thanks!
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Monday, January 12, 2015 3:40 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > and it's a pain. The down side i think to doing it via a Maven plugin is
> > that if the user wants to modify it, they have to rebuild, pull the
> > generated html out of the target directory, and then modify it and put it
> > in the src/main/resources/docs. Then, if they change something they have to
> > do this again. If it's done on-the-fly, rather than building static content
> > with a plugin, that doesn't have to happen.
> > > But you may come up with something much better than I'm envisioning :)
> > Having a util to do it is a good call though. Then it could be used in a
> > maven plugin, by the app, or wherever is most appropriate.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > >
> > > > Mark,
> > > > Sorry.. I missed your reply in the thread.
> > > >
> > > > Are you saying that going forward, you want to auto generate the
> > doc and allow the developer to include a "more info" link? I wanted to
> > confirm that it doesn't work that way now.
> > > >
> > > > I'll think about this a little more. Might write some code to
> > generate the documentation from an AbstractProcessor class, then decide
> > later where to integrate it(maven plugin, or somewhere else)
> > > >
> > > > Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > >
> > > > Dan,
> > > >
> > > > Matt and I discussed this approach before but ultimately decided that
> > we like the idea of just auto generating the docs on demand with a "More
> > Info" type of link that lets you add more in html format if you care to do
> > so.
> > > >
> > > > Others' thoughts and ideas are welcome though!
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Just pulled down the code out of Git and got everything up and
> > running on Windows. This was pretty simple, and the UI looks great under
> > chrome.
> > > > >
> > > > >
> > > > > I was thinking of writing a maven plugin to generate the processor
> > usage HTML documentation.
> > > > >
> > > > >
> > > > > Looking through the existing processor documentation, it seems
> > like most of this information comes directly from the code in the
> > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > Relationships. Then if your processor only requires this level of
> > documentation in the HTML format, you could tie this plugin to the build,
> > and keep your inline processor documentation in sync with the HTML
> > documentation.
> > > > >
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> >
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
It might make sense to have an annotation for this:
@SupportsDynamicProperties
Perhaps even allow developer to provide a string for how they are used:
@SupportsDynamicProperties("Name of dynamic property becomes attribute name; value becomes attribute value.")
I've begun ticket #6: https://issues.apache.org/jira/browse/NIFI-6 but I have not pushed anything to a branch or anything yet. Basically, I deprecated the existing annotations and created 3 new packages for them. Instead of org.apache.nifi.processor.annotation, we will have:
org.apache.nifi.annotation.documentationorg.apache.nifi.annotation.behaviororg.apache.nifi.annotation.lifecycle
So the @SupportsDynamicProperties could go under 'behavior' if that makes sense.
> Date: Fri, 16 Jan 2015 11:09:28 -0500
> Subject: Re: Processor documentation plugin
> From: matt.c.gilman@gmail.com
> To: dev@nifi.incubator.apache.org
>
> Dan,
>
> We discussed item 1 awhile back and even created a ticket for it. However,
> there was always higher priority work that took precedence. I am not sure
> if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
> I believe the current mechanism allows the processor to decide if a given
> dynamic property should be a supported. I think we would need a more
> general indicator that dynamic properties are supported at all to trigger
> that button state. This would likely help with item 2 as well.
>
> Matt
>
> On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
> wrote:
>
> > Mark,
> > Sounds good. I'm on board. I was thinking of having the
> > ControllerServices/ReportingTask's documentation include an XML template,
> > but if those will be configurable via the UI then that will not be needed.
> > I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> > accordingly.
> >
> >
> > Matt/Mark,
> > Two UI/API questions:
> > 1) If a processor does not support dynamic properties, why does the UI let
> > them hit [+] New Property?
> > 2) If I have an instance of Processor class, I can call
> > getSupportedProperties() on it to get a list of the required/optional
> > properties. But I don't have a way to know if that processor supports
> > dynamic properties or not. Supporting dynamic properties or not was
> > something I was planning on reflecting in the documentation. Is there any
> > way to detect this currently?
> >
> > Thanks,
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Wednesday, January 14, 2015 8:51 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > I think this is great!
> > I would recommend we slate this for the 0.1.0 version, which includes
> > putting Controller Services and Reporting Tasks in the UI. But by the same
> > token I think that makes your first bulletin for "rev 2" more of a priority
> > for "rev 1" if we can get it there. I could pitch in and help if needed.
> > I'd also recommend you put the Capability Description in for Rev 1 as
> > that's arguably the most important part of the docs (which may make the
> > Tags very easy at that point??)
> > The rest of what you have listed for "rev 2" i would agree should be there.
> >
> > Very excited to see this happening!
> > Thanks-Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Thu, 15 Jan 2015 01:39:44 +0000
> > >
> > > Mark et al,
> > > Ok, I think I hear everyone's concerns and ideas. I will challenge
> > myself to come up with a solution that makes everyone happy all the time.
> > >
> > > The goals of this effort will be:
> > > - Machine generate as much of the basic(processor description,
> > property description, relationship description) info as possible. This
> > will make the developers lives better by storing this information in one
> > place, and make users lives better by having the "inline" documentation
> > match the "usage" documentation. This will also ensure that all the
> > documentation is formatted consistently.
> > > - Allow the developer to optionally provide additional html and
> > images that would be linked from the machine generated documentation
> > > - Provide a way for the documentation to be generated statically
> > and linked from the Apache NiFi webpage
> > >
> > > A few other 'nice-to-have' goals(probably for rev 2):
> > > - Support documenting ControllerServices and ReportingTasks is a
> > similar way
> > > -Maybe leverage existing annotations on these types? E.g
> > @Tags, @CapabilityDescription...
> > > - Provide new 'documentation' annotations that would allow a
> > developer to describe the FlowFile attributes a processor uses on input,
> > and which attributes a processor modifies on output
> > > - Provide new 'documentation' annotations that describe the
> > expected input format of the flow file content
> > > - Provide new 'documentation' annotations that describe whether
> > the processor modifies the flow file content
> > > - Provide new 'documentation' annotations that describe the format
> > of the outbound flow file content
> > > - Provide new 'documentation' annotations that describe other NiFi
> > artifacts(processors, controller services, reporting tasks) that are
> > related to this one, e.g. "See Also ..."
> > > - Consider if/how current Processor annotations should be
> > displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> > @EventDriven)
> > >
> > >
> > > High level implementation plan:
> > > - Develop a utility class to read an AbstractProcessor(and
> > existing HTML documention) and generated the basic documentation with links
> > to existing HTML
> > > - Integrate this class into the framework such that all
> > documentation displayed within a NiFi instance is generated by the utility
> > > - Integrate this utility into something that can take a NiFi
> > build, dig through the nars, find the
> > Processors/ControllerServices/ReportingTasks and generate their appropriate
> > HTML documentation so that it can be stored somewhere and linked from the
> > Apache NiFi website.
> > >
> > > If this feels like the wrong path, or if you have any other ideas
> > please speak up. Thanks!
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Monday, January 12, 2015 3:40 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: RE: Processor documentation plugin
> > >
> > > Dan,
> > > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> > and it's a pain. The down side i think to doing it via a Maven plugin is
> > that if the user wants to modify it, they have to rebuild, pull the
> > generated html out of the target directory, and then modify it and put it
> > in the src/main/resources/docs. Then, if they change something they have to
> > do this again. If it's done on-the-fly, rather than building static content
> > with a plugin, that doesn't have to happen.
> > > But you may come up with something much better than I'm envisioning :)
> > Having a util to do it is a good call though. Then it could be used in a
> > maven plugin, by the app, or wherever is most appropriate.
> > > Thanks-Mark
> > >
> > > > From: dbress@onyxconsults.com
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > > >
> > > > Mark,
> > > > Sorry.. I missed your reply in the thread.
> > > >
> > > > Are you saying that going forward, you want to auto generate the
> > doc and allow the developer to include a "more info" link? I wanted to
> > confirm that it doesn't work that way now.
> > > >
> > > > I'll think about this a little more. Might write some code to
> > generate the documentation from an AbstractProcessor class, then decide
> > later where to integrate it(maven plugin, or somewhere else)
> > > >
> > > > Thanks!
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
> > > >
> > > > ________________________________________
> > > > From: Mark Payne <ma...@hotmail.com>
> > > > Sent: Sunday, January 11, 2015 1:23 PM
> > > > To: dev@nifi.incubator.apache.org
> > > > Subject: Re: Processor documentation plugin
> > > >
> > > > Dan,
> > > >
> > > > Matt and I discussed this approach before but ultimately decided that
> > we like the idea of just auto generating the docs on demand with a "More
> > Info" type of link that lets you add more in html format if you care to do
> > so.
> > > >
> > > > Others' thoughts and ideas are welcome though!
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> > > > >
> > > > > Hi All,
> > > > >
> > > > > Just pulled down the code out of Git and got everything up and
> > running on Windows. This was pretty simple, and the UI looks great under
> > chrome.
> > > > >
> > > > >
> > > > > I was thinking of writing a maven plugin to generate the processor
> > usage HTML documentation.
> > > > >
> > > > >
> > > > > Looking through the existing processor documentation, it seems
> > like most of this information comes directly from the code in the
> > processors, via @CapabilityDescription, and the PropertyDescriptors and
> > Relationships. Then if your processor only requires this level of
> > documentation in the HTML format, you could tie this plugin to the build,
> > and keep your inline processor documentation in sync with the HTML
> > documentation.
> > > > >
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > Dan Bress
> > > > > Software Engineer
> > > > > ONYX Consulting Services
> >
Re: Processor documentation plugin
Posted by Matt Gilman <ma...@gmail.com>.
Dan,
We discussed item 1 awhile back and even created a ticket for it. However,
there was always higher priority work that took precedence. I am not sure
if that ticket was ever created in JIRA for Apache NiFi. If not, we should.
I believe the current mechanism allows the processor to decide if a given
dynamic property should be a supported. I think we would need a more
general indicator that dynamic properties are supported at all to trigger
that button state. This would likely help with item 2 as well.
Matt
On Fri, Jan 16, 2015 at 10:45 AM, Daniel Bress <db...@onyxconsults.com>
wrote:
> Mark,
> Sounds good. I'm on board. I was thinking of having the
> ControllerServices/ReportingTask's documentation include an XML template,
> but if those will be configurable via the UI then that will not be needed.
> I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work
> accordingly.
>
>
> Matt/Mark,
> Two UI/API questions:
> 1) If a processor does not support dynamic properties, why does the UI let
> them hit [+] New Property?
> 2) If I have an instance of Processor class, I can call
> getSupportedProperties() on it to get a list of the required/optional
> properties. But I don't have a way to know if that processor supports
> dynamic properties or not. Supporting dynamic properties or not was
> something I was planning on reflecting in the documentation. Is there any
> way to detect this currently?
>
> Thanks,
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Wednesday, January 14, 2015 8:51 PM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> I think this is great!
> I would recommend we slate this for the 0.1.0 version, which includes
> putting Controller Services and Reporting Tasks in the UI. But by the same
> token I think that makes your first bulletin for "rev 2" more of a priority
> for "rev 1" if we can get it there. I could pitch in and help if needed.
> I'd also recommend you put the Capability Description in for Rev 1 as
> that's arguably the most important part of the docs (which may make the
> Tags very easy at that point??)
> The rest of what you have listed for "rev 2" i would agree should be there.
>
> Very excited to see this happening!
> Thanks-Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Thu, 15 Jan 2015 01:39:44 +0000
> >
> > Mark et al,
> > Ok, I think I hear everyone's concerns and ideas. I will challenge
> myself to come up with a solution that makes everyone happy all the time.
> >
> > The goals of this effort will be:
> > - Machine generate as much of the basic(processor description,
> property description, relationship description) info as possible. This
> will make the developers lives better by storing this information in one
> place, and make users lives better by having the "inline" documentation
> match the "usage" documentation. This will also ensure that all the
> documentation is formatted consistently.
> > - Allow the developer to optionally provide additional html and
> images that would be linked from the machine generated documentation
> > - Provide a way for the documentation to be generated statically
> and linked from the Apache NiFi webpage
> >
> > A few other 'nice-to-have' goals(probably for rev 2):
> > - Support documenting ControllerServices and ReportingTasks is a
> similar way
> > -Maybe leverage existing annotations on these types? E.g
> @Tags, @CapabilityDescription...
> > - Provide new 'documentation' annotations that would allow a
> developer to describe the FlowFile attributes a processor uses on input,
> and which attributes a processor modifies on output
> > - Provide new 'documentation' annotations that describe the
> expected input format of the flow file content
> > - Provide new 'documentation' annotations that describe whether
> the processor modifies the flow file content
> > - Provide new 'documentation' annotations that describe the format
> of the outbound flow file content
> > - Provide new 'documentation' annotations that describe other NiFi
> artifacts(processors, controller services, reporting tasks) that are
> related to this one, e.g. "See Also ..."
> > - Consider if/how current Processor annotations should be
> displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty...,
> @EventDriven)
> >
> >
> > High level implementation plan:
> > - Develop a utility class to read an AbstractProcessor(and
> existing HTML documention) and generated the basic documentation with links
> to existing HTML
> > - Integrate this class into the framework such that all
> documentation displayed within a NiFi instance is generated by the utility
> > - Integrate this utility into something that can take a NiFi
> build, dig through the nars, find the
> Processors/ControllerServices/ReportingTasks and generate their appropriate
> HTML documentation so that it can be stored somewhere and linked from the
> Apache NiFi website.
> >
> > If this feels like the wrong path, or if you have any other ideas
> please speak up. Thanks!
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Monday, January 12, 2015 3:40 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: RE: Processor documentation plugin
> >
> > Dan,
> > Yes, that's what I'm saying. Right now, it's all done manually by hand,
> and it's a pain. The down side i think to doing it via a Maven plugin is
> that if the user wants to modify it, they have to rebuild, pull the
> generated html out of the target directory, and then modify it and put it
> in the src/main/resources/docs. Then, if they change something they have to
> do this again. If it's done on-the-fly, rather than building static content
> with a plugin, that doesn't have to happen.
> > But you may come up with something much better than I'm envisioning :)
> Having a util to do it is a good call though. Then it could be used in a
> maven plugin, by the app, or wherever is most appropriate.
> > Thanks-Mark
> >
> > > From: dbress@onyxconsults.com
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > > Date: Mon, 12 Jan 2015 20:34:03 +0000
> > >
> > > Mark,
> > > Sorry.. I missed your reply in the thread.
> > >
> > > Are you saying that going forward, you want to auto generate the
> doc and allow the developer to include a "more info" link? I wanted to
> confirm that it doesn't work that way now.
> > >
> > > I'll think about this a little more. Might write some code to
> generate the documentation from an AbstractProcessor class, then decide
> later where to integrate it(maven plugin, or somewhere else)
> > >
> > > Thanks!
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> > >
> > > ________________________________________
> > > From: Mark Payne <ma...@hotmail.com>
> > > Sent: Sunday, January 11, 2015 1:23 PM
> > > To: dev@nifi.incubator.apache.org
> > > Subject: Re: Processor documentation plugin
> > >
> > > Dan,
> > >
> > > Matt and I discussed this approach before but ultimately decided that
> we like the idea of just auto generating the docs on demand with a "More
> Info" type of link that lets you add more in html format if you care to do
> so.
> > >
> > > Others' thoughts and ideas are welcome though!
> > >
> > > Sent from my iPhone
> > >
> > > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Just pulled down the code out of Git and got everything up and
> running on Windows. This was pretty simple, and the UI looks great under
> chrome.
> > > >
> > > >
> > > > I was thinking of writing a maven plugin to generate the processor
> usage HTML documentation.
> > > >
> > > >
> > > > Looking through the existing processor documentation, it seems
> like most of this information comes directly from the code in the
> processors, via @CapabilityDescription, and the PropertyDescriptors and
> Relationships. Then if your processor only requires this level of
> documentation in the HTML format, you could tie this plugin to the build,
> and keep your inline processor documentation in sync with the HTML
> documentation.
> > > >
> > > >
> > > > What do you think?
> > > >
> > > >
> > > > Dan Bress
> > > > Software Engineer
> > > > ONYX Consulting Services
>
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Mark,
Sounds good. I'm on board. I was thinking of having the ControllerServices/ReportingTask's documentation include an XML template, but if those will be configurable via the UI then that will not be needed.
I'm on board with your rev 1 / rev 2 suggestions too. I'll get to work accordingly.
Matt/Mark,
Two UI/API questions:
1) If a processor does not support dynamic properties, why does the UI let them hit [+] New Property?
2) If I have an instance of Processor class, I can call getSupportedProperties() on it to get a list of the required/optional properties. But I don't have a way to know if that processor supports dynamic properties or not. Supporting dynamic properties or not was something I was planning on reflecting in the documentation. Is there any way to detect this currently?
Thanks,
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Wednesday, January 14, 2015 8:51 PM
To: dev@nifi.incubator.apache.org
Subject: RE: Processor documentation plugin
Dan,
I think this is great!
I would recommend we slate this for the 0.1.0 version, which includes putting Controller Services and Reporting Tasks in the UI. But by the same token I think that makes your first bulletin for "rev 2" more of a priority for "rev 1" if we can get it there. I could pitch in and help if needed. I'd also recommend you put the Capability Description in for Rev 1 as that's arguably the most important part of the docs (which may make the Tags very easy at that point??)
The rest of what you have listed for "rev 2" i would agree should be there.
Very excited to see this happening!
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Thu, 15 Jan 2015 01:39:44 +0000
>
> Mark et al,
> Ok, I think I hear everyone's concerns and ideas. I will challenge myself to come up with a solution that makes everyone happy all the time.
>
> The goals of this effort will be:
> - Machine generate as much of the basic(processor description, property description, relationship description) info as possible. This will make the developers lives better by storing this information in one place, and make users lives better by having the "inline" documentation match the "usage" documentation. This will also ensure that all the documentation is formatted consistently.
> - Allow the developer to optionally provide additional html and images that would be linked from the machine generated documentation
> - Provide a way for the documentation to be generated statically and linked from the Apache NiFi webpage
>
> A few other 'nice-to-have' goals(probably for rev 2):
> - Support documenting ControllerServices and ReportingTasks is a similar way
> -Maybe leverage existing annotations on these types? E.g @Tags, @CapabilityDescription...
> - Provide new 'documentation' annotations that would allow a developer to describe the FlowFile attributes a processor uses on input, and which attributes a processor modifies on output
> - Provide new 'documentation' annotations that describe the expected input format of the flow file content
> - Provide new 'documentation' annotations that describe whether the processor modifies the flow file content
> - Provide new 'documentation' annotations that describe the format of the outbound flow file content
> - Provide new 'documentation' annotations that describe other NiFi artifacts(processors, controller services, reporting tasks) that are related to this one, e.g. "See Also ..."
> - Consider if/how current Processor annotations should be displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty..., @EventDriven)
>
>
> High level implementation plan:
> - Develop a utility class to read an AbstractProcessor(and existing HTML documention) and generated the basic documentation with links to existing HTML
> - Integrate this class into the framework such that all documentation displayed within a NiFi instance is generated by the utility
> - Integrate this utility into something that can take a NiFi build, dig through the nars, find the Processors/ControllerServices/ReportingTasks and generate their appropriate HTML documentation so that it can be stored somewhere and linked from the Apache NiFi website.
>
> If this feels like the wrong path, or if you have any other ideas please speak up. Thanks!
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Monday, January 12, 2015 3:40 PM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> Yes, that's what I'm saying. Right now, it's all done manually by hand, and it's a pain. The down side i think to doing it via a Maven plugin is that if the user wants to modify it, they have to rebuild, pull the generated html out of the target directory, and then modify it and put it in the src/main/resources/docs. Then, if they change something they have to do this again. If it's done on-the-fly, rather than building static content with a plugin, that doesn't have to happen.
> But you may come up with something much better than I'm envisioning :) Having a util to do it is a good call though. Then it could be used in a maven plugin, by the app, or wherever is most appropriate.
> Thanks-Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Mon, 12 Jan 2015 20:34:03 +0000
> >
> > Mark,
> > Sorry.. I missed your reply in the thread.
> >
> > Are you saying that going forward, you want to auto generate the doc and allow the developer to include a "more info" link? I wanted to confirm that it doesn't work that way now.
> >
> > I'll think about this a little more. Might write some code to generate the documentation from an AbstractProcessor class, then decide later where to integrate it(maven plugin, or somewhere else)
> >
> > Thanks!
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Sunday, January 11, 2015 1:23 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> >
> > Dan,
> >
> > Matt and I discussed this approach before but ultimately decided that we like the idea of just auto generating the docs on demand with a "More Info" type of link that lets you add more in html format if you care to do so.
> >
> > Others' thoughts and ideas are welcome though!
> >
> > Sent from my iPhone
> >
> > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com> wrote:
> > >
> > > Hi All,
> > >
> > > Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
> > >
> > >
> > > I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
> > >
> > >
> > > Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
> > >
> > >
> > > What do you think?
> > >
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
I think this is great!
I would recommend we slate this for the 0.1.0 version, which includes putting Controller Services and Reporting Tasks in the UI. But by the same token I think that makes your first bulletin for "rev 2" more of a priority for "rev 1" if we can get it there. I could pitch in and help if needed. I'd also recommend you put the Capability Description in for Rev 1 as that's arguably the most important part of the docs (which may make the Tags very easy at that point??)
The rest of what you have listed for "rev 2" i would agree should be there.
Very excited to see this happening!
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Thu, 15 Jan 2015 01:39:44 +0000
>
> Mark et al,
> Ok, I think I hear everyone's concerns and ideas. I will challenge myself to come up with a solution that makes everyone happy all the time.
>
> The goals of this effort will be:
> - Machine generate as much of the basic(processor description, property description, relationship description) info as possible. This will make the developers lives better by storing this information in one place, and make users lives better by having the "inline" documentation match the "usage" documentation. This will also ensure that all the documentation is formatted consistently.
> - Allow the developer to optionally provide additional html and images that would be linked from the machine generated documentation
> - Provide a way for the documentation to be generated statically and linked from the Apache NiFi webpage
>
> A few other 'nice-to-have' goals(probably for rev 2):
> - Support documenting ControllerServices and ReportingTasks is a similar way
> -Maybe leverage existing annotations on these types? E.g @Tags, @CapabilityDescription...
> - Provide new 'documentation' annotations that would allow a developer to describe the FlowFile attributes a processor uses on input, and which attributes a processor modifies on output
> - Provide new 'documentation' annotations that describe the expected input format of the flow file content
> - Provide new 'documentation' annotations that describe whether the processor modifies the flow file content
> - Provide new 'documentation' annotations that describe the format of the outbound flow file content
> - Provide new 'documentation' annotations that describe other NiFi artifacts(processors, controller services, reporting tasks) that are related to this one, e.g. "See Also ..."
> - Consider if/how current Processor annotations should be displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty..., @EventDriven)
>
>
> High level implementation plan:
> - Develop a utility class to read an AbstractProcessor(and existing HTML documention) and generated the basic documentation with links to existing HTML
> - Integrate this class into the framework such that all documentation displayed within a NiFi instance is generated by the utility
> - Integrate this utility into something that can take a NiFi build, dig through the nars, find the Processors/ControllerServices/ReportingTasks and generate their appropriate HTML documentation so that it can be stored somewhere and linked from the Apache NiFi website.
>
> If this feels like the wrong path, or if you have any other ideas please speak up. Thanks!
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Monday, January 12, 2015 3:40 PM
> To: dev@nifi.incubator.apache.org
> Subject: RE: Processor documentation plugin
>
> Dan,
> Yes, that's what I'm saying. Right now, it's all done manually by hand, and it's a pain. The down side i think to doing it via a Maven plugin is that if the user wants to modify it, they have to rebuild, pull the generated html out of the target directory, and then modify it and put it in the src/main/resources/docs. Then, if they change something they have to do this again. If it's done on-the-fly, rather than building static content with a plugin, that doesn't have to happen.
> But you may come up with something much better than I'm envisioning :) Having a util to do it is a good call though. Then it could be used in a maven plugin, by the app, or wherever is most appropriate.
> Thanks-Mark
>
> > From: dbress@onyxconsults.com
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> > Date: Mon, 12 Jan 2015 20:34:03 +0000
> >
> > Mark,
> > Sorry.. I missed your reply in the thread.
> >
> > Are you saying that going forward, you want to auto generate the doc and allow the developer to include a "more info" link? I wanted to confirm that it doesn't work that way now.
> >
> > I'll think about this a little more. Might write some code to generate the documentation from an AbstractProcessor class, then decide later where to integrate it(maven plugin, or somewhere else)
> >
> > Thanks!
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
> >
> > ________________________________________
> > From: Mark Payne <ma...@hotmail.com>
> > Sent: Sunday, January 11, 2015 1:23 PM
> > To: dev@nifi.incubator.apache.org
> > Subject: Re: Processor documentation plugin
> >
> > Dan,
> >
> > Matt and I discussed this approach before but ultimately decided that we like the idea of just auto generating the docs on demand with a "More Info" type of link that lets you add more in html format if you care to do so.
> >
> > Others' thoughts and ideas are welcome though!
> >
> > Sent from my iPhone
> >
> > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com> wrote:
> > >
> > > Hi All,
> > >
> > > Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
> > >
> > >
> > > I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
> > >
> > >
> > > Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
> > >
> > >
> > > What do you think?
> > >
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Mark et al,
Ok, I think I hear everyone's concerns and ideas. I will challenge myself to come up with a solution that makes everyone happy all the time.
The goals of this effort will be:
- Machine generate as much of the basic(processor description, property description, relationship description) info as possible. This will make the developers lives better by storing this information in one place, and make users lives better by having the "inline" documentation match the "usage" documentation. This will also ensure that all the documentation is formatted consistently.
- Allow the developer to optionally provide additional html and images that would be linked from the machine generated documentation
- Provide a way for the documentation to be generated statically and linked from the Apache NiFi webpage
A few other 'nice-to-have' goals(probably for rev 2):
- Support documenting ControllerServices and ReportingTasks is a similar way
-Maybe leverage existing annotations on these types? E.g @Tags, @CapabilityDescription...
- Provide new 'documentation' annotations that would allow a developer to describe the FlowFile attributes a processor uses on input, and which attributes a processor modifies on output
- Provide new 'documentation' annotations that describe the expected input format of the flow file content
- Provide new 'documentation' annotations that describe whether the processor modifies the flow file content
- Provide new 'documentation' annotations that describe the format of the outbound flow file content
- Provide new 'documentation' annotations that describe other NiFi artifacts(processors, controller services, reporting tasks) that are related to this one, e.g. "See Also ..."
- Consider if/how current Processor annotations should be displayed (@SideEffectFree, @TriggerSerially, @TriggerWhenEmpty..., @EventDriven)
High level implementation plan:
- Develop a utility class to read an AbstractProcessor(and existing HTML documention) and generated the basic documentation with links to existing HTML
- Integrate this class into the framework such that all documentation displayed within a NiFi instance is generated by the utility
- Integrate this utility into something that can take a NiFi build, dig through the nars, find the Processors/ControllerServices/ReportingTasks and generate their appropriate HTML documentation so that it can be stored somewhere and linked from the Apache NiFi website.
If this feels like the wrong path, or if you have any other ideas please speak up. Thanks!
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Monday, January 12, 2015 3:40 PM
To: dev@nifi.incubator.apache.org
Subject: RE: Processor documentation plugin
Dan,
Yes, that's what I'm saying. Right now, it's all done manually by hand, and it's a pain. The down side i think to doing it via a Maven plugin is that if the user wants to modify it, they have to rebuild, pull the generated html out of the target directory, and then modify it and put it in the src/main/resources/docs. Then, if they change something they have to do this again. If it's done on-the-fly, rather than building static content with a plugin, that doesn't have to happen.
But you may come up with something much better than I'm envisioning :) Having a util to do it is a good call though. Then it could be used in a maven plugin, by the app, or wherever is most appropriate.
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Mon, 12 Jan 2015 20:34:03 +0000
>
> Mark,
> Sorry.. I missed your reply in the thread.
>
> Are you saying that going forward, you want to auto generate the doc and allow the developer to include a "more info" link? I wanted to confirm that it doesn't work that way now.
>
> I'll think about this a little more. Might write some code to generate the documentation from an AbstractProcessor class, then decide later where to integrate it(maven plugin, or somewhere else)
>
> Thanks!
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Sunday, January 11, 2015 1:23 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
>
> Matt and I discussed this approach before but ultimately decided that we like the idea of just auto generating the docs on demand with a "More Info" type of link that lets you add more in html format if you care to do so.
>
> Others' thoughts and ideas are welcome though!
>
> Sent from my iPhone
>
> > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com> wrote:
> >
> > Hi All,
> >
> > Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
> >
> >
> > I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
> >
> >
> > Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
> >
> >
> > What do you think?
> >
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
RE: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
Yes, that's what I'm saying. Right now, it's all done manually by hand, and it's a pain. The down side i think to doing it via a Maven plugin is that if the user wants to modify it, they have to rebuild, pull the generated html out of the target directory, and then modify it and put it in the src/main/resources/docs. Then, if they change something they have to do this again. If it's done on-the-fly, rather than building static content with a plugin, that doesn't have to happen.
But you may come up with something much better than I'm envisioning :) Having a util to do it is a good call though. Then it could be used in a maven plugin, by the app, or wherever is most appropriate.
Thanks-Mark
> From: dbress@onyxconsults.com
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
> Date: Mon, 12 Jan 2015 20:34:03 +0000
>
> Mark,
> Sorry.. I missed your reply in the thread.
>
> Are you saying that going forward, you want to auto generate the doc and allow the developer to include a "more info" link? I wanted to confirm that it doesn't work that way now.
>
> I'll think about this a little more. Might write some code to generate the documentation from an AbstractProcessor class, then decide later where to integrate it(maven plugin, or somewhere else)
>
> Thanks!
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Mark Payne <ma...@hotmail.com>
> Sent: Sunday, January 11, 2015 1:23 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
>
> Matt and I discussed this approach before but ultimately decided that we like the idea of just auto generating the docs on demand with a "More Info" type of link that lets you add more in html format if you care to do so.
>
> Others' thoughts and ideas are welcome though!
>
> Sent from my iPhone
>
> > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com> wrote:
> >
> > Hi All,
> >
> > Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
> >
> >
> > I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
> >
> >
> > Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
> >
> >
> > What do you think?
> >
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Mark,
Sorry.. I missed your reply in the thread.
Are you saying that going forward, you want to auto generate the doc and allow the developer to include a "more info" link? I wanted to confirm that it doesn't work that way now.
I'll think about this a little more. Might write some code to generate the documentation from an AbstractProcessor class, then decide later where to integrate it(maven plugin, or somewhere else)
Thanks!
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Mark Payne <ma...@hotmail.com>
Sent: Sunday, January 11, 2015 1:23 PM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Dan,
Matt and I discussed this approach before but ultimately decided that we like the idea of just auto generating the docs on demand with a "More Info" type of link that lets you add more in html format if you care to do so.
Others' thoughts and ideas are welcome though!
Sent from my iPhone
> On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com> wrote:
>
> Hi All,
>
> Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
>
>
> I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
>
>
> Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
>
>
> What do you think?
>
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
Re: Processor documentation plugin
Posted by Joe Witt <jo...@gmail.com>.
Dan,
Having the docs available for the apache site is a great point. We'd
definitely want to be able to have them there.
Thanks
Joe
On Sun, Jan 11, 2015 at 1:58 PM, Daniel Bress <db...@onyxconsults.com>
wrote:
> Joe,
> Thanks for welcoming and feedback. I'm definitely looking forward to
> learning from and contributing to Apache NiFi.
>
> I think a good thing about having statically generated content is it
> could be linked from the website(https://nifi.incubator.apache.org/)?
> E.g. these are the processors in the build 0.0.1, and this is how you use
> them. Don't you think you'd want that documentation generated with the
> build and linked from the website at that point?
>
> As far as people providing their own documentation... yeah that needs to
> be sorted. I was thinking maybe the docs could be modified to have a
> generated section that pulls as much as it can from the code, then have a
> user definable section should that processor require more a more in depth
> explanation.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> ________________________________________
> From: Joe Witt <jo...@gmail.com>
> Sent: Sunday, January 11, 2015 1:30 PM
> To: dev@nifi.incubator.apache.org
> Subject: Re: Processor documentation plugin
>
> Dan,
>
> First, very happy to see you're planning to contribute.
>
> So this is a great idea!
>
> A few things to consider here:
> - At what point in the application lifecycle to create automatic
> processor/extension documentation
> 1) Build phase: Best done via a Maven plugin probably.
> 2) Startup Phase: Could be part of the nar loading/extension service
> provider hunt.
> 3) On demand: As a request comes to view-usage or look at the extension
> listing generate the processor/extension docs.
>
> Given that it would likely be purely static content any of these approaches
> are reasonable. My estimation then is that the ease of implementation in
> order is 3, 2, 1. So you might consider approaching this that way. Do
> option 3 since that code would be reusable. If we find that isn't good
> enough then go option 2. If we think we need earliest possible binding
> then option 3. We can just grow through it if needed. But KISS probably
> points at 3 in my opinion.
>
> Also as Benson pointed out in another thread we're likely heading down the
> path of having a separate Git repo for Maven specific plugins/archetypes.
> So you might want to hold off on the maven part specifically until that
> stabilizes too.
>
> Consideration for the generation beyond (when):
> - What if the developer already supplies processor docs? Perhaps
> auto-generation only kicks in if they've not done this. Or maybe we
> provide both (auto generated and developer generated)...something to thing
> about.
>
> Very cool that you're wanting to take a stab at this. I think it will be
> really helpful and it ensures maximum bang for the buck on those
> annotations for users.
>
> Thanks!
> Joe
>
> On Sun, Jan 11, 2015 at 1:23 PM, Mark Payne <ma...@hotmail.com> wrote:
>
> > Dan,
> >
> > Matt and I discussed this approach before but ultimately decided that we
> > like the idea of just auto generating the docs on demand with a "More
> Info"
> > type of link that lets you add more in html format if you care to do so.
> >
> > Others' thoughts and ideas are welcome though!
> >
> > Sent from my iPhone
> >
> > > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> > wrote:
> > >
> > > Hi All,
> > >
> > > Just pulled down the code out of Git and got everything up and
> running
> > on Windows. This was pretty simple, and the UI looks great under chrome.
> > >
> > >
> > > I was thinking of writing a maven plugin to generate the processor
> > usage HTML documentation.
> > >
> > >
> > > Looking through the existing processor documentation, it seems like
> > most of this information comes directly from the code in the processors,
> > via @CapabilityDescription, and the PropertyDescriptors and
> Relationships.
> > Then if your processor only requires this level of documentation in the
> > HTML format, you could tie this plugin to the build, and keep your inline
> > processor documentation in sync with the HTML documentation.
> > >
> > >
> > > What do you think?
> > >
> > >
> > > Dan Bress
> > > Software Engineer
> > > ONYX Consulting Services
> >
>
Re: Processor documentation plugin
Posted by Daniel Bress <db...@onyxconsults.com>.
Joe,
Thanks for welcoming and feedback. I'm definitely looking forward to learning from and contributing to Apache NiFi.
I think a good thing about having statically generated content is it could be linked from the website(https://nifi.incubator.apache.org/)? E.g. these are the processors in the build 0.0.1, and this is how you use them. Don't you think you'd want that documentation generated with the build and linked from the website at that point?
As far as people providing their own documentation... yeah that needs to be sorted. I was thinking maybe the docs could be modified to have a generated section that pulls as much as it can from the code, then have a user definable section should that processor require more a more in depth explanation.
Dan Bress
Software Engineer
ONYX Consulting Services
________________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Sunday, January 11, 2015 1:30 PM
To: dev@nifi.incubator.apache.org
Subject: Re: Processor documentation plugin
Dan,
First, very happy to see you're planning to contribute.
So this is a great idea!
A few things to consider here:
- At what point in the application lifecycle to create automatic
processor/extension documentation
1) Build phase: Best done via a Maven plugin probably.
2) Startup Phase: Could be part of the nar loading/extension service
provider hunt.
3) On demand: As a request comes to view-usage or look at the extension
listing generate the processor/extension docs.
Given that it would likely be purely static content any of these approaches
are reasonable. My estimation then is that the ease of implementation in
order is 3, 2, 1. So you might consider approaching this that way. Do
option 3 since that code would be reusable. If we find that isn't good
enough then go option 2. If we think we need earliest possible binding
then option 3. We can just grow through it if needed. But KISS probably
points at 3 in my opinion.
Also as Benson pointed out in another thread we're likely heading down the
path of having a separate Git repo for Maven specific plugins/archetypes.
So you might want to hold off on the maven part specifically until that
stabilizes too.
Consideration for the generation beyond (when):
- What if the developer already supplies processor docs? Perhaps
auto-generation only kicks in if they've not done this. Or maybe we
provide both (auto generated and developer generated)...something to thing
about.
Very cool that you're wanting to take a stab at this. I think it will be
really helpful and it ensures maximum bang for the buck on those
annotations for users.
Thanks!
Joe
On Sun, Jan 11, 2015 at 1:23 PM, Mark Payne <ma...@hotmail.com> wrote:
> Dan,
>
> Matt and I discussed this approach before but ultimately decided that we
> like the idea of just auto generating the docs on demand with a "More Info"
> type of link that lets you add more in html format if you care to do so.
>
> Others' thoughts and ideas are welcome though!
>
> Sent from my iPhone
>
> > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> wrote:
> >
> > Hi All,
> >
> > Just pulled down the code out of Git and got everything up and running
> on Windows. This was pretty simple, and the UI looks great under chrome.
> >
> >
> > I was thinking of writing a maven plugin to generate the processor
> usage HTML documentation.
> >
> >
> > Looking through the existing processor documentation, it seems like
> most of this information comes directly from the code in the processors,
> via @CapabilityDescription, and the PropertyDescriptors and Relationships.
> Then if your processor only requires this level of documentation in the
> HTML format, you could tie this plugin to the build, and keep your inline
> processor documentation in sync with the HTML documentation.
> >
> >
> > What do you think?
> >
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
>
Re: Processor documentation plugin
Posted by Joe Witt <jo...@gmail.com>.
Dan,
First, very happy to see you're planning to contribute.
So this is a great idea!
A few things to consider here:
- At what point in the application lifecycle to create automatic
processor/extension documentation
1) Build phase: Best done via a Maven plugin probably.
2) Startup Phase: Could be part of the nar loading/extension service
provider hunt.
3) On demand: As a request comes to view-usage or look at the extension
listing generate the processor/extension docs.
Given that it would likely be purely static content any of these approaches
are reasonable. My estimation then is that the ease of implementation in
order is 3, 2, 1. So you might consider approaching this that way. Do
option 3 since that code would be reusable. If we find that isn't good
enough then go option 2. If we think we need earliest possible binding
then option 3. We can just grow through it if needed. But KISS probably
points at 3 in my opinion.
Also as Benson pointed out in another thread we're likely heading down the
path of having a separate Git repo for Maven specific plugins/archetypes.
So you might want to hold off on the maven part specifically until that
stabilizes too.
Consideration for the generation beyond (when):
- What if the developer already supplies processor docs? Perhaps
auto-generation only kicks in if they've not done this. Or maybe we
provide both (auto generated and developer generated)...something to thing
about.
Very cool that you're wanting to take a stab at this. I think it will be
really helpful and it ensures maximum bang for the buck on those
annotations for users.
Thanks!
Joe
On Sun, Jan 11, 2015 at 1:23 PM, Mark Payne <ma...@hotmail.com> wrote:
> Dan,
>
> Matt and I discussed this approach before but ultimately decided that we
> like the idea of just auto generating the docs on demand with a "More Info"
> type of link that lets you add more in html format if you care to do so.
>
> Others' thoughts and ideas are welcome though!
>
> Sent from my iPhone
>
> > On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com>
> wrote:
> >
> > Hi All,
> >
> > Just pulled down the code out of Git and got everything up and running
> on Windows. This was pretty simple, and the UI looks great under chrome.
> >
> >
> > I was thinking of writing a maven plugin to generate the processor
> usage HTML documentation.
> >
> >
> > Looking through the existing processor documentation, it seems like
> most of this information comes directly from the code in the processors,
> via @CapabilityDescription, and the PropertyDescriptors and Relationships.
> Then if your processor only requires this level of documentation in the
> HTML format, you could tie this plugin to the build, and keep your inline
> processor documentation in sync with the HTML documentation.
> >
> >
> > What do you think?
> >
> >
> > Dan Bress
> > Software Engineer
> > ONYX Consulting Services
>
Re: Processor documentation plugin
Posted by Mark Payne <ma...@hotmail.com>.
Dan,
Matt and I discussed this approach before but ultimately decided that we like the idea of just auto generating the docs on demand with a "More Info" type of link that lets you add more in html format if you care to do so.
Others' thoughts and ideas are welcome though!
Sent from my iPhone
> On Jan 11, 2015, at 12:32 PM, Daniel Bress <db...@onyxconsults.com> wrote:
>
> Hi All,
>
> Just pulled down the code out of Git and got everything up and running on Windows. This was pretty simple, and the UI looks great under chrome.
>
>
> I was thinking of writing a maven plugin to generate the processor usage HTML documentation.
>
>
> Looking through the existing processor documentation, it seems like most of this information comes directly from the code in the processors, via @CapabilityDescription, and the PropertyDescriptors and Relationships. Then if your processor only requires this level of documentation in the HTML format, you could tie this plugin to the build, and keep your inline processor documentation in sync with the HTML documentation.
>
>
> What do you think?
>
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services