You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Brian Ghigiarelli <br...@gmail.com> on 2015/08/13 23:22:46 UTC
Custom Controller Service Debugging
NiFi Gurus,
Regarding Controller Services, I'm seeing some strange behavior regarding a
custom Controller Service only showing its ID in a Processor and was
wondering if anyone's encountered similar issues. Seeing this on the
latest develop branch, 0.3.0-SNAPSHOT.
Steps taken:
1. Create instance of Controller Service in the Controller Settings >
Controller Services tab
1a. Enable that instance of the Controller Service
2. Add new Processor on canvas that will use Controller Service
3. Configure Processor
3a. Existing Controller Service does not show in property options. Only "No
value" and "Create new service..."
3b. Clicking "Create new service..." opens modal with the custom Controller
Service as the only item in the dropdown
3c. Clicking "Create" button succeeds, but puts the UUID of the Controller
Service in the Value of the Processor Property.
Observed state:
- Processor is still invalid
- The ID in the Processor Property matches the ID of the second (disabled)
instance of the Controller Service in the Controller Settings modal.
There are no errors in nifi-app.log - only contains log messages from the
StandardControllerServiceProvider that it "Created Controller Service of
type [my-implementation-class] with identifier [UUID-that-shows-in-the-GUI]"
Adding a DetectDuplicate processor with the
DistributedMapCacheClientService works just fine, so it'd hard to point
fingers at the framework. Any thoughts on where to begin debugging and/or
loggers to turn up would be greatly appreciated!
Thanks,
Brian
Re: Custom Controller Service Debugging
Posted by Brian Ghigiarelli <br...@gmail.com>.
Cool - thanks for verifying that, Matt!
https://issues.apache.org/jira/browse/NIFI-854
On Fri, Aug 14, 2015 at 11:43 AM, Matt Gilman <ma...@gmail.com>
wrote:
> Brian,
>
> I see the same behavior regarding Controller Service creation when
> referenced by Processor's in sub Groups. Please file the JIRA ticket and it
> will be addressed in our release. Thanks!
>
> Matt
>
> On Fri, Aug 14, 2015 at 11:30 AM, Brian Ghigiarelli <br...@gmail.com>
> wrote:
>
> > Matt and Bryan,
> >
> > Thanks for the detailed explanation. The class loading aspect makes
> > perfect sense, and yes, importing as disabled is understood as well.
> >
> > I do think I've come across a small bug with automatically creating
> > Controller Services on template imports though. NiFi is not
> automatically
> > creating Controller Services for Processors inside of Process Groups on
> > template imports. Can one of you check me on this before we write up a
> Jira
> > ticket?
> >
> > Steps to reproduce:
> > 1. Start fresh (clean canvas, no controller services)
> > 2. Create Process Group and go into it
> > 3. Add DetectDuplicate processor to Process Group
> > 4. Configure DetectDuplicate processor with a new
> > DistributedMapCacheClientService and enable that service so that the
> > DetectDuplicate processor is valid and ready to run
> > 5. Navigate to root canvas (see the Process Group)
> > 6. Create template from root canvas (name: "Controller Service Test")
> > 7. Delete everything from the canvas and disable + delete the controller
> > service so that we're essentially back to the start
> > 8. Drag and drop the "Controller Service Test" template onto the canvas
> > (expecting that everything will be auto-created)
> > 9. Navigate into the Process Group and notice that the DetectDuplicate
> > processor is invalid because the DistributedMapCacheClientService ID
> cannot
> > be found
> >
> > As an additional test, I created a template with a processor at the
> > top-level and one inside the Process Group that are using the same
> > Controller Service, and that works just fine. My guess is that the
> > top-level processor is used to create the Controller Service and the
> > inner-processor picks it up by that same ID.
> >
> > I can provide a sample template in a Jira ticket if that would help.
> >
> > Thanks,
> > Brian
> >
> > On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Brian,
> > >
> > > Glad it worked!
> > >
> > > I think it was just a combination of things... I had previously spent a
> > lot
> > > of time studying the existing dependency setup in nifi-standard-bundle,
> > as
> > > well as the dev
> > > guide section on NARs [1]. Also, someone else had ran into a similar
> > > problem recently and it turned out to be a dependency problem as well.
> > >
> > > There should be a Maven archetype for Controller Services [2] coming
> soon
> > > which might help nail down the correct dependency setup, also after
> > seeing
> > > your email last night I put together a bare-bones project [3] that has
> a
> > > custom processor using a custom controller service (both created using
> > > the archetypes) to serve as an example.
> > >
> > > -Bryan
> > >
> > > [1]
> > https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
> > > [2] https://issues.apache.org/jira/browse/NIFI-736
> > > [3] https://github.com/bbende/nifi-dependency-example
> > >
> > >
> > >
> > > On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman <ma...@gmail.com>
> > > wrote:
> > >
> > > > Brian,
> > > >
> > > > The NAR dependency stuff all breaks down to providing classpath
> > isolation
> > > > and not polluting the classpath of downstream NARs with dependencies
> > > > required for Controller Service implementation. We try to keep
> > Controller
> > > > Service interfaces lean by not depending on external libs in the
> > > interface
> > > > (generally a good practice anyways). Both the implementation of the
> > > > Controller Service and the any Processor NAR that want to consume the
> > > > Controller Service needs to depend on the Controller Service API NAR
> > > (where
> > > > the interface lives). This established a common parent ClassLoader
> > which
> > > > allows the service to be access across NARs. If your building
> Processor
> > > > NARs that depend on custom Controller Services and standard
> Controller
> > > > Services, you'll want to have your custom Controller Services API NAR
> > > > depend on the standard Controller Services API NAR. The NAR
> dependency
> > > tree
> > > > (if you will) maps to a ClassLoader hierarchy within NiFi.
> > > >
> > > > Controller Services being added in a disabled state is remaining
> > > consistent
> > > > with Processors (and other components) in template. Also, because
> > > sensitive
> > > > properties are not included in templates, they may require the user
> to
> > > > complete the configuration prior to enabling.
> > > >
> > > > Matt
> > > >
> > > > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli <
> > brianghig@gmail.com>
> > > > wrote:
> > > >
> > > > > Bryan,
> > > > >
> > > > > Thank you!! Adding a NAR dependency to my standard-processors NAR
> did
> > > the
> > > > > trick to get NiFi to recognize the service for these processors and
> > > > > auto-create a new instance of the Controller Service on template
> > > > import. I
> > > > > owe ya a beer or six the next time our paths cross.
> > > > >
> > > > > How did you figure out the required NAR dependency? Any great
> > debugging
> > > > > tricks to share, or was it your existing depth of NiFi knowledge
> that
> > > led
> > > > > you to that solution?
> > > > >
> > > > > As a follow-on to that, I'm seeing that Controller Services created
> > via
> > > > > Template instantiation are created in "Disabled" state. Is that
> > > > expected?
> > > > > Is there any way to drag-and-drop a template with controller
> services
> > > > onto
> > > > > a canvas and have the newly created controller services enabled?
> > > > >
> > > > > Thanks again,
> > > > > Brian
> > > > >
> > > > > On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > >
> > > > > > Brian,
> > > > > >
> > > > > > I just ran into this exact scenario today while testing a custom
> > > > > controller
> > > > > > service.
> > > > > >
> > > > > > The cause of the problem was that my custom service NAR, did not
> > > have a
> > > > > NAR
> > > > > > dependency on the service API NAR because I had accidentally
> marked
> > > the
> > > > > API
> > > > > > NAR as provided in the pom file. Once I removed the provided
> scope
> > > > > > everything worked.
> > > > > >
> > > > > > You can check the MANIFEST file in your custom service NAR...
> under
> > > > your
> > > > > > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it
> > > should
> > > > > > have a "Nar-Dependency-Id" on your API NAR. If it is missing
> that,
> > > that
> > > > > is
> > > > > > likely the problem.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <
> > > > brianghig@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > NiFi Gurus,
> > > > > > >
> > > > > > > Regarding Controller Services, I'm seeing some strange behavior
> > > > > > regarding a
> > > > > > > custom Controller Service only showing its ID in a Processor
> and
> > > was
> > > > > > > wondering if anyone's encountered similar issues. Seeing this
> on
> > > the
> > > > > > > latest develop branch, 0.3.0-SNAPSHOT.
> > > > > > >
> > > > > > > Steps taken:
> > > > > > > 1. Create instance of Controller Service in the Controller
> > > Settings >
> > > > > > > Controller Services tab
> > > > > > > 1a. Enable that instance of the Controller Service
> > > > > > > 2. Add new Processor on canvas that will use Controller Service
> > > > > > > 3. Configure Processor
> > > > > > > 3a. Existing Controller Service does not show in property
> > options.
> > > > Only
> > > > > > "No
> > > > > > > value" and "Create new service..."
> > > > > > > 3b. Clicking "Create new service..." opens modal with the
> custom
> > > > > > Controller
> > > > > > > Service as the only item in the dropdown
> > > > > > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > > > > > Controller
> > > > > > > Service in the Value of the Processor Property.
> > > > > > >
> > > > > > > Observed state:
> > > > > > > - Processor is still invalid
> > > > > > > - The ID in the Processor Property matches the ID of the second
> > > > > > (disabled)
> > > > > > > instance of the Controller Service in the Controller Settings
> > > modal.
> > > > > > >
> > > > > > > There are no errors in nifi-app.log - only contains log
> messages
> > > from
> > > > > the
> > > > > > > StandardControllerServiceProvider that it "Created Controller
> > > Service
> > > > > of
> > > > > > > type [my-implementation-class] with identifier
> > > > > > > [UUID-that-shows-in-the-GUI]"
> > > > > > >
> > > > > > > Adding a DetectDuplicate processor with the
> > > > > > > DistributedMapCacheClientService works just fine, so it'd hard
> to
> > > > point
> > > > > > > fingers at the framework. Any thoughts on where to begin
> > debugging
> > > > > > and/or
> > > > > > > loggers to turn up would be greatly appreciated!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Brian
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Brian Ghigiarelli
> > > > > 570-878-9139
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Brian Ghigiarelli
> > 570-878-9139
> >
>
--
Brian Ghigiarelli
570-878-9139
Re: Custom Controller Service Debugging
Posted by Matt Gilman <ma...@gmail.com>.
Brian,
I see the same behavior regarding Controller Service creation when
referenced by Processor's in sub Groups. Please file the JIRA ticket and it
will be addressed in our release. Thanks!
Matt
On Fri, Aug 14, 2015 at 11:30 AM, Brian Ghigiarelli <br...@gmail.com>
wrote:
> Matt and Bryan,
>
> Thanks for the detailed explanation. The class loading aspect makes
> perfect sense, and yes, importing as disabled is understood as well.
>
> I do think I've come across a small bug with automatically creating
> Controller Services on template imports though. NiFi is not automatically
> creating Controller Services for Processors inside of Process Groups on
> template imports. Can one of you check me on this before we write up a Jira
> ticket?
>
> Steps to reproduce:
> 1. Start fresh (clean canvas, no controller services)
> 2. Create Process Group and go into it
> 3. Add DetectDuplicate processor to Process Group
> 4. Configure DetectDuplicate processor with a new
> DistributedMapCacheClientService and enable that service so that the
> DetectDuplicate processor is valid and ready to run
> 5. Navigate to root canvas (see the Process Group)
> 6. Create template from root canvas (name: "Controller Service Test")
> 7. Delete everything from the canvas and disable + delete the controller
> service so that we're essentially back to the start
> 8. Drag and drop the "Controller Service Test" template onto the canvas
> (expecting that everything will be auto-created)
> 9. Navigate into the Process Group and notice that the DetectDuplicate
> processor is invalid because the DistributedMapCacheClientService ID cannot
> be found
>
> As an additional test, I created a template with a processor at the
> top-level and one inside the Process Group that are using the same
> Controller Service, and that works just fine. My guess is that the
> top-level processor is used to create the Controller Service and the
> inner-processor picks it up by that same ID.
>
> I can provide a sample template in a Jira ticket if that would help.
>
> Thanks,
> Brian
>
> On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende <bb...@gmail.com> wrote:
>
> > Brian,
> >
> > Glad it worked!
> >
> > I think it was just a combination of things... I had previously spent a
> lot
> > of time studying the existing dependency setup in nifi-standard-bundle,
> as
> > well as the dev
> > guide section on NARs [1]. Also, someone else had ran into a similar
> > problem recently and it turned out to be a dependency problem as well.
> >
> > There should be a Maven archetype for Controller Services [2] coming soon
> > which might help nail down the correct dependency setup, also after
> seeing
> > your email last night I put together a bare-bones project [3] that has a
> > custom processor using a custom controller service (both created using
> > the archetypes) to serve as an example.
> >
> > -Bryan
> >
> > [1]
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
> > [2] https://issues.apache.org/jira/browse/NIFI-736
> > [3] https://github.com/bbende/nifi-dependency-example
> >
> >
> >
> > On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman <ma...@gmail.com>
> > wrote:
> >
> > > Brian,
> > >
> > > The NAR dependency stuff all breaks down to providing classpath
> isolation
> > > and not polluting the classpath of downstream NARs with dependencies
> > > required for Controller Service implementation. We try to keep
> Controller
> > > Service interfaces lean by not depending on external libs in the
> > interface
> > > (generally a good practice anyways). Both the implementation of the
> > > Controller Service and the any Processor NAR that want to consume the
> > > Controller Service needs to depend on the Controller Service API NAR
> > (where
> > > the interface lives). This established a common parent ClassLoader
> which
> > > allows the service to be access across NARs. If your building Processor
> > > NARs that depend on custom Controller Services and standard Controller
> > > Services, you'll want to have your custom Controller Services API NAR
> > > depend on the standard Controller Services API NAR. The NAR dependency
> > tree
> > > (if you will) maps to a ClassLoader hierarchy within NiFi.
> > >
> > > Controller Services being added in a disabled state is remaining
> > consistent
> > > with Processors (and other components) in template. Also, because
> > sensitive
> > > properties are not included in templates, they may require the user to
> > > complete the configuration prior to enabling.
> > >
> > > Matt
> > >
> > > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli <
> brianghig@gmail.com>
> > > wrote:
> > >
> > > > Bryan,
> > > >
> > > > Thank you!! Adding a NAR dependency to my standard-processors NAR did
> > the
> > > > trick to get NiFi to recognize the service for these processors and
> > > > auto-create a new instance of the Controller Service on template
> > > import. I
> > > > owe ya a beer or six the next time our paths cross.
> > > >
> > > > How did you figure out the required NAR dependency? Any great
> debugging
> > > > tricks to share, or was it your existing depth of NiFi knowledge that
> > led
> > > > you to that solution?
> > > >
> > > > As a follow-on to that, I'm seeing that Controller Services created
> via
> > > > Template instantiation are created in "Disabled" state. Is that
> > > expected?
> > > > Is there any way to drag-and-drop a template with controller services
> > > onto
> > > > a canvas and have the newly created controller services enabled?
> > > >
> > > > Thanks again,
> > > > Brian
> > > >
> > > > On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bb...@gmail.com>
> wrote:
> > > >
> > > > > Brian,
> > > > >
> > > > > I just ran into this exact scenario today while testing a custom
> > > > controller
> > > > > service.
> > > > >
> > > > > The cause of the problem was that my custom service NAR, did not
> > have a
> > > > NAR
> > > > > dependency on the service API NAR because I had accidentally marked
> > the
> > > > API
> > > > > NAR as provided in the pom file. Once I removed the provided scope
> > > > > everything worked.
> > > > >
> > > > > You can check the MANIFEST file in your custom service NAR... under
> > > your
> > > > > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it
> > should
> > > > > have a "Nar-Dependency-Id" on your API NAR. If it is missing that,
> > that
> > > > is
> > > > > likely the problem.
> > > > >
> > > > > -Bryan
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <
> > > brianghig@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > NiFi Gurus,
> > > > > >
> > > > > > Regarding Controller Services, I'm seeing some strange behavior
> > > > > regarding a
> > > > > > custom Controller Service only showing its ID in a Processor and
> > was
> > > > > > wondering if anyone's encountered similar issues. Seeing this on
> > the
> > > > > > latest develop branch, 0.3.0-SNAPSHOT.
> > > > > >
> > > > > > Steps taken:
> > > > > > 1. Create instance of Controller Service in the Controller
> > Settings >
> > > > > > Controller Services tab
> > > > > > 1a. Enable that instance of the Controller Service
> > > > > > 2. Add new Processor on canvas that will use Controller Service
> > > > > > 3. Configure Processor
> > > > > > 3a. Existing Controller Service does not show in property
> options.
> > > Only
> > > > > "No
> > > > > > value" and "Create new service..."
> > > > > > 3b. Clicking "Create new service..." opens modal with the custom
> > > > > Controller
> > > > > > Service as the only item in the dropdown
> > > > > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > > > > Controller
> > > > > > Service in the Value of the Processor Property.
> > > > > >
> > > > > > Observed state:
> > > > > > - Processor is still invalid
> > > > > > - The ID in the Processor Property matches the ID of the second
> > > > > (disabled)
> > > > > > instance of the Controller Service in the Controller Settings
> > modal.
> > > > > >
> > > > > > There are no errors in nifi-app.log - only contains log messages
> > from
> > > > the
> > > > > > StandardControllerServiceProvider that it "Created Controller
> > Service
> > > > of
> > > > > > type [my-implementation-class] with identifier
> > > > > > [UUID-that-shows-in-the-GUI]"
> > > > > >
> > > > > > Adding a DetectDuplicate processor with the
> > > > > > DistributedMapCacheClientService works just fine, so it'd hard to
> > > point
> > > > > > fingers at the framework. Any thoughts on where to begin
> debugging
> > > > > and/or
> > > > > > loggers to turn up would be greatly appreciated!
> > > > > >
> > > > > > Thanks,
> > > > > > Brian
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Brian Ghigiarelli
> > > > 570-878-9139
> > > >
> > >
> >
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
>
Re: Custom Controller Service Debugging
Posted by Brian Ghigiarelli <br...@gmail.com>.
Matt and Bryan,
Thanks for the detailed explanation. The class loading aspect makes
perfect sense, and yes, importing as disabled is understood as well.
I do think I've come across a small bug with automatically creating
Controller Services on template imports though. NiFi is not automatically
creating Controller Services for Processors inside of Process Groups on
template imports. Can one of you check me on this before we write up a Jira
ticket?
Steps to reproduce:
1. Start fresh (clean canvas, no controller services)
2. Create Process Group and go into it
3. Add DetectDuplicate processor to Process Group
4. Configure DetectDuplicate processor with a new
DistributedMapCacheClientService and enable that service so that the
DetectDuplicate processor is valid and ready to run
5. Navigate to root canvas (see the Process Group)
6. Create template from root canvas (name: "Controller Service Test")
7. Delete everything from the canvas and disable + delete the controller
service so that we're essentially back to the start
8. Drag and drop the "Controller Service Test" template onto the canvas
(expecting that everything will be auto-created)
9. Navigate into the Process Group and notice that the DetectDuplicate
processor is invalid because the DistributedMapCacheClientService ID cannot
be found
As an additional test, I created a template with a processor at the
top-level and one inside the Process Group that are using the same
Controller Service, and that works just fine. My guess is that the
top-level processor is used to create the Controller Service and the
inner-processor picks it up by that same ID.
I can provide a sample template in a Jira ticket if that would help.
Thanks,
Brian
On Fri, Aug 14, 2015 at 8:24 AM, Bryan Bende <bb...@gmail.com> wrote:
> Brian,
>
> Glad it worked!
>
> I think it was just a combination of things... I had previously spent a lot
> of time studying the existing dependency setup in nifi-standard-bundle, as
> well as the dev
> guide section on NARs [1]. Also, someone else had ran into a similar
> problem recently and it turned out to be a dependency problem as well.
>
> There should be a Maven archetype for Controller Services [2] coming soon
> which might help nail down the correct dependency setup, also after seeing
> your email last night I put together a bare-bones project [3] that has a
> custom processor using a custom controller service (both created using
> the archetypes) to serve as an example.
>
> -Bryan
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
> [2] https://issues.apache.org/jira/browse/NIFI-736
> [3] https://github.com/bbende/nifi-dependency-example
>
>
>
> On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman <ma...@gmail.com>
> wrote:
>
> > Brian,
> >
> > The NAR dependency stuff all breaks down to providing classpath isolation
> > and not polluting the classpath of downstream NARs with dependencies
> > required for Controller Service implementation. We try to keep Controller
> > Service interfaces lean by not depending on external libs in the
> interface
> > (generally a good practice anyways). Both the implementation of the
> > Controller Service and the any Processor NAR that want to consume the
> > Controller Service needs to depend on the Controller Service API NAR
> (where
> > the interface lives). This established a common parent ClassLoader which
> > allows the service to be access across NARs. If your building Processor
> > NARs that depend on custom Controller Services and standard Controller
> > Services, you'll want to have your custom Controller Services API NAR
> > depend on the standard Controller Services API NAR. The NAR dependency
> tree
> > (if you will) maps to a ClassLoader hierarchy within NiFi.
> >
> > Controller Services being added in a disabled state is remaining
> consistent
> > with Processors (and other components) in template. Also, because
> sensitive
> > properties are not included in templates, they may require the user to
> > complete the configuration prior to enabling.
> >
> > Matt
> >
> > On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli <br...@gmail.com>
> > wrote:
> >
> > > Bryan,
> > >
> > > Thank you!! Adding a NAR dependency to my standard-processors NAR did
> the
> > > trick to get NiFi to recognize the service for these processors and
> > > auto-create a new instance of the Controller Service on template
> > import. I
> > > owe ya a beer or six the next time our paths cross.
> > >
> > > How did you figure out the required NAR dependency? Any great debugging
> > > tricks to share, or was it your existing depth of NiFi knowledge that
> led
> > > you to that solution?
> > >
> > > As a follow-on to that, I'm seeing that Controller Services created via
> > > Template instantiation are created in "Disabled" state. Is that
> > expected?
> > > Is there any way to drag-and-drop a template with controller services
> > onto
> > > a canvas and have the newly created controller services enabled?
> > >
> > > Thanks again,
> > > Brian
> > >
> > > On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bb...@gmail.com> wrote:
> > >
> > > > Brian,
> > > >
> > > > I just ran into this exact scenario today while testing a custom
> > > controller
> > > > service.
> > > >
> > > > The cause of the problem was that my custom service NAR, did not
> have a
> > > NAR
> > > > dependency on the service API NAR because I had accidentally marked
> the
> > > API
> > > > NAR as provided in the pom file. Once I removed the provided scope
> > > > everything worked.
> > > >
> > > > You can check the MANIFEST file in your custom service NAR... under
> > your
> > > > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it
> should
> > > > have a "Nar-Dependency-Id" on your API NAR. If it is missing that,
> that
> > > is
> > > > likely the problem.
> > > >
> > > > -Bryan
> > > >
> > > >
> > > >
> > > > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <
> > brianghig@gmail.com>
> > > > wrote:
> > > >
> > > > > NiFi Gurus,
> > > > >
> > > > > Regarding Controller Services, I'm seeing some strange behavior
> > > > regarding a
> > > > > custom Controller Service only showing its ID in a Processor and
> was
> > > > > wondering if anyone's encountered similar issues. Seeing this on
> the
> > > > > latest develop branch, 0.3.0-SNAPSHOT.
> > > > >
> > > > > Steps taken:
> > > > > 1. Create instance of Controller Service in the Controller
> Settings >
> > > > > Controller Services tab
> > > > > 1a. Enable that instance of the Controller Service
> > > > > 2. Add new Processor on canvas that will use Controller Service
> > > > > 3. Configure Processor
> > > > > 3a. Existing Controller Service does not show in property options.
> > Only
> > > > "No
> > > > > value" and "Create new service..."
> > > > > 3b. Clicking "Create new service..." opens modal with the custom
> > > > Controller
> > > > > Service as the only item in the dropdown
> > > > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > > > Controller
> > > > > Service in the Value of the Processor Property.
> > > > >
> > > > > Observed state:
> > > > > - Processor is still invalid
> > > > > - The ID in the Processor Property matches the ID of the second
> > > > (disabled)
> > > > > instance of the Controller Service in the Controller Settings
> modal.
> > > > >
> > > > > There are no errors in nifi-app.log - only contains log messages
> from
> > > the
> > > > > StandardControllerServiceProvider that it "Created Controller
> Service
> > > of
> > > > > type [my-implementation-class] with identifier
> > > > > [UUID-that-shows-in-the-GUI]"
> > > > >
> > > > > Adding a DetectDuplicate processor with the
> > > > > DistributedMapCacheClientService works just fine, so it'd hard to
> > point
> > > > > fingers at the framework. Any thoughts on where to begin debugging
> > > > and/or
> > > > > loggers to turn up would be greatly appreciated!
> > > > >
> > > > > Thanks,
> > > > > Brian
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Brian Ghigiarelli
> > > 570-878-9139
> > >
> >
>
--
Brian Ghigiarelli
570-878-9139
Re: Custom Controller Service Debugging
Posted by Bryan Bende <bb...@gmail.com>.
Brian,
Glad it worked!
I think it was just a combination of things... I had previously spent a lot
of time studying the existing dependency setup in nifi-standard-bundle, as
well as the dev
guide section on NARs [1]. Also, someone else had ran into a similar
problem recently and it turned out to be a dependency problem as well.
There should be a Maven archetype for Controller Services [2] coming soon
which might help nail down the correct dependency setup, also after seeing
your email last night I put together a bare-bones project [3] that has a
custom processor using a custom controller service (both created using
the archetypes) to serve as an example.
-Bryan
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#nars
[2] https://issues.apache.org/jira/browse/NIFI-736
[3] https://github.com/bbende/nifi-dependency-example
On Fri, Aug 14, 2015 at 8:17 AM, Matt Gilman <ma...@gmail.com>
wrote:
> Brian,
>
> The NAR dependency stuff all breaks down to providing classpath isolation
> and not polluting the classpath of downstream NARs with dependencies
> required for Controller Service implementation. We try to keep Controller
> Service interfaces lean by not depending on external libs in the interface
> (generally a good practice anyways). Both the implementation of the
> Controller Service and the any Processor NAR that want to consume the
> Controller Service needs to depend on the Controller Service API NAR (where
> the interface lives). This established a common parent ClassLoader which
> allows the service to be access across NARs. If your building Processor
> NARs that depend on custom Controller Services and standard Controller
> Services, you'll want to have your custom Controller Services API NAR
> depend on the standard Controller Services API NAR. The NAR dependency tree
> (if you will) maps to a ClassLoader hierarchy within NiFi.
>
> Controller Services being added in a disabled state is remaining consistent
> with Processors (and other components) in template. Also, because sensitive
> properties are not included in templates, they may require the user to
> complete the configuration prior to enabling.
>
> Matt
>
> On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli <br...@gmail.com>
> wrote:
>
> > Bryan,
> >
> > Thank you!! Adding a NAR dependency to my standard-processors NAR did the
> > trick to get NiFi to recognize the service for these processors and
> > auto-create a new instance of the Controller Service on template
> import. I
> > owe ya a beer or six the next time our paths cross.
> >
> > How did you figure out the required NAR dependency? Any great debugging
> > tricks to share, or was it your existing depth of NiFi knowledge that led
> > you to that solution?
> >
> > As a follow-on to that, I'm seeing that Controller Services created via
> > Template instantiation are created in "Disabled" state. Is that
> expected?
> > Is there any way to drag-and-drop a template with controller services
> onto
> > a canvas and have the newly created controller services enabled?
> >
> > Thanks again,
> > Brian
> >
> > On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bb...@gmail.com> wrote:
> >
> > > Brian,
> > >
> > > I just ran into this exact scenario today while testing a custom
> > controller
> > > service.
> > >
> > > The cause of the problem was that my custom service NAR, did not have a
> > NAR
> > > dependency on the service API NAR because I had accidentally marked the
> > API
> > > NAR as provided in the pom file. Once I removed the provided scope
> > > everything worked.
> > >
> > > You can check the MANIFEST file in your custom service NAR... under
> your
> > > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
> > > have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that
> > is
> > > likely the problem.
> > >
> > > -Bryan
> > >
> > >
> > >
> > > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <
> brianghig@gmail.com>
> > > wrote:
> > >
> > > > NiFi Gurus,
> > > >
> > > > Regarding Controller Services, I'm seeing some strange behavior
> > > regarding a
> > > > custom Controller Service only showing its ID in a Processor and was
> > > > wondering if anyone's encountered similar issues. Seeing this on the
> > > > latest develop branch, 0.3.0-SNAPSHOT.
> > > >
> > > > Steps taken:
> > > > 1. Create instance of Controller Service in the Controller Settings >
> > > > Controller Services tab
> > > > 1a. Enable that instance of the Controller Service
> > > > 2. Add new Processor on canvas that will use Controller Service
> > > > 3. Configure Processor
> > > > 3a. Existing Controller Service does not show in property options.
> Only
> > > "No
> > > > value" and "Create new service..."
> > > > 3b. Clicking "Create new service..." opens modal with the custom
> > > Controller
> > > > Service as the only item in the dropdown
> > > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > > Controller
> > > > Service in the Value of the Processor Property.
> > > >
> > > > Observed state:
> > > > - Processor is still invalid
> > > > - The ID in the Processor Property matches the ID of the second
> > > (disabled)
> > > > instance of the Controller Service in the Controller Settings modal.
> > > >
> > > > There are no errors in nifi-app.log - only contains log messages from
> > the
> > > > StandardControllerServiceProvider that it "Created Controller Service
> > of
> > > > type [my-implementation-class] with identifier
> > > > [UUID-that-shows-in-the-GUI]"
> > > >
> > > > Adding a DetectDuplicate processor with the
> > > > DistributedMapCacheClientService works just fine, so it'd hard to
> point
> > > > fingers at the framework. Any thoughts on where to begin debugging
> > > and/or
> > > > loggers to turn up would be greatly appreciated!
> > > >
> > > > Thanks,
> > > > Brian
> > > >
> > >
> >
> >
> >
> > --
> > Brian Ghigiarelli
> > 570-878-9139
> >
>
Re: Custom Controller Service Debugging
Posted by Matt Gilman <ma...@gmail.com>.
Brian,
The NAR dependency stuff all breaks down to providing classpath isolation
and not polluting the classpath of downstream NARs with dependencies
required for Controller Service implementation. We try to keep Controller
Service interfaces lean by not depending on external libs in the interface
(generally a good practice anyways). Both the implementation of the
Controller Service and the any Processor NAR that want to consume the
Controller Service needs to depend on the Controller Service API NAR (where
the interface lives). This established a common parent ClassLoader which
allows the service to be access across NARs. If your building Processor
NARs that depend on custom Controller Services and standard Controller
Services, you'll want to have your custom Controller Services API NAR
depend on the standard Controller Services API NAR. The NAR dependency tree
(if you will) maps to a ClassLoader hierarchy within NiFi.
Controller Services being added in a disabled state is remaining consistent
with Processors (and other components) in template. Also, because sensitive
properties are not included in templates, they may require the user to
complete the configuration prior to enabling.
Matt
On Fri, Aug 14, 2015 at 8:01 AM, Brian Ghigiarelli <br...@gmail.com>
wrote:
> Bryan,
>
> Thank you!! Adding a NAR dependency to my standard-processors NAR did the
> trick to get NiFi to recognize the service for these processors and
> auto-create a new instance of the Controller Service on template import. I
> owe ya a beer or six the next time our paths cross.
>
> How did you figure out the required NAR dependency? Any great debugging
> tricks to share, or was it your existing depth of NiFi knowledge that led
> you to that solution?
>
> As a follow-on to that, I'm seeing that Controller Services created via
> Template instantiation are created in "Disabled" state. Is that expected?
> Is there any way to drag-and-drop a template with controller services onto
> a canvas and have the newly created controller services enabled?
>
> Thanks again,
> Brian
>
> On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> > Brian,
> >
> > I just ran into this exact scenario today while testing a custom
> controller
> > service.
> >
> > The cause of the problem was that my custom service NAR, did not have a
> NAR
> > dependency on the service API NAR because I had accidentally marked the
> API
> > NAR as provided in the pom file. Once I removed the provided scope
> > everything worked.
> >
> > You can check the MANIFEST file in your custom service NAR... under your
> > NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
> > have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that
> is
> > likely the problem.
> >
> > -Bryan
> >
> >
> >
> > On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <br...@gmail.com>
> > wrote:
> >
> > > NiFi Gurus,
> > >
> > > Regarding Controller Services, I'm seeing some strange behavior
> > regarding a
> > > custom Controller Service only showing its ID in a Processor and was
> > > wondering if anyone's encountered similar issues. Seeing this on the
> > > latest develop branch, 0.3.0-SNAPSHOT.
> > >
> > > Steps taken:
> > > 1. Create instance of Controller Service in the Controller Settings >
> > > Controller Services tab
> > > 1a. Enable that instance of the Controller Service
> > > 2. Add new Processor on canvas that will use Controller Service
> > > 3. Configure Processor
> > > 3a. Existing Controller Service does not show in property options. Only
> > "No
> > > value" and "Create new service..."
> > > 3b. Clicking "Create new service..." opens modal with the custom
> > Controller
> > > Service as the only item in the dropdown
> > > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> > Controller
> > > Service in the Value of the Processor Property.
> > >
> > > Observed state:
> > > - Processor is still invalid
> > > - The ID in the Processor Property matches the ID of the second
> > (disabled)
> > > instance of the Controller Service in the Controller Settings modal.
> > >
> > > There are no errors in nifi-app.log - only contains log messages from
> the
> > > StandardControllerServiceProvider that it "Created Controller Service
> of
> > > type [my-implementation-class] with identifier
> > > [UUID-that-shows-in-the-GUI]"
> > >
> > > Adding a DetectDuplicate processor with the
> > > DistributedMapCacheClientService works just fine, so it'd hard to point
> > > fingers at the framework. Any thoughts on where to begin debugging
> > and/or
> > > loggers to turn up would be greatly appreciated!
> > >
> > > Thanks,
> > > Brian
> > >
> >
>
>
>
> --
> Brian Ghigiarelli
> 570-878-9139
>
Re: Custom Controller Service Debugging
Posted by Brian Ghigiarelli <br...@gmail.com>.
Bryan,
Thank you!! Adding a NAR dependency to my standard-processors NAR did the
trick to get NiFi to recognize the service for these processors and
auto-create a new instance of the Controller Service on template import. I
owe ya a beer or six the next time our paths cross.
How did you figure out the required NAR dependency? Any great debugging
tricks to share, or was it your existing depth of NiFi knowledge that led
you to that solution?
As a follow-on to that, I'm seeing that Controller Services created via
Template instantiation are created in "Disabled" state. Is that expected?
Is there any way to drag-and-drop a template with controller services onto
a canvas and have the newly created controller services enabled?
Thanks again,
Brian
On Thu, Aug 13, 2015 at 5:57 PM, Bryan Bende <bb...@gmail.com> wrote:
> Brian,
>
> I just ran into this exact scenario today while testing a custom controller
> service.
>
> The cause of the problem was that my custom service NAR, did not have a NAR
> dependency on the service API NAR because I had accidentally marked the API
> NAR as provided in the pom file. Once I removed the provided scope
> everything worked.
>
> You can check the MANIFEST file in your custom service NAR... under your
> NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
> have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that is
> likely the problem.
>
> -Bryan
>
>
>
> On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <br...@gmail.com>
> wrote:
>
> > NiFi Gurus,
> >
> > Regarding Controller Services, I'm seeing some strange behavior
> regarding a
> > custom Controller Service only showing its ID in a Processor and was
> > wondering if anyone's encountered similar issues. Seeing this on the
> > latest develop branch, 0.3.0-SNAPSHOT.
> >
> > Steps taken:
> > 1. Create instance of Controller Service in the Controller Settings >
> > Controller Services tab
> > 1a. Enable that instance of the Controller Service
> > 2. Add new Processor on canvas that will use Controller Service
> > 3. Configure Processor
> > 3a. Existing Controller Service does not show in property options. Only
> "No
> > value" and "Create new service..."
> > 3b. Clicking "Create new service..." opens modal with the custom
> Controller
> > Service as the only item in the dropdown
> > 3c. Clicking "Create" button succeeds, but puts the UUID of the
> Controller
> > Service in the Value of the Processor Property.
> >
> > Observed state:
> > - Processor is still invalid
> > - The ID in the Processor Property matches the ID of the second
> (disabled)
> > instance of the Controller Service in the Controller Settings modal.
> >
> > There are no errors in nifi-app.log - only contains log messages from the
> > StandardControllerServiceProvider that it "Created Controller Service of
> > type [my-implementation-class] with identifier
> > [UUID-that-shows-in-the-GUI]"
> >
> > Adding a DetectDuplicate processor with the
> > DistributedMapCacheClientService works just fine, so it'd hard to point
> > fingers at the framework. Any thoughts on where to begin debugging
> and/or
> > loggers to turn up would be greatly appreciated!
> >
> > Thanks,
> > Brian
> >
>
--
Brian Ghigiarelli
570-878-9139
Re: Custom Controller Service Debugging
Posted by Bryan Bende <bb...@gmail.com>.
Brian,
I just ran into this exact scenario today while testing a custom controller
service.
The cause of the problem was that my custom service NAR, did not have a NAR
dependency on the service API NAR because I had accidentally marked the API
NAR as provided in the pom file. Once I removed the provided scope
everything worked.
You can check the MANIFEST file in your custom service NAR... under your
NIFI_HOME/work/nar/extensions/{your-nar}/META-INF/MANIFEST.MF it should
have a "Nar-Dependency-Id" on your API NAR. If it is missing that, that is
likely the problem.
-Bryan
On Thu, Aug 13, 2015 at 5:22 PM, Brian Ghigiarelli <br...@gmail.com>
wrote:
> NiFi Gurus,
>
> Regarding Controller Services, I'm seeing some strange behavior regarding a
> custom Controller Service only showing its ID in a Processor and was
> wondering if anyone's encountered similar issues. Seeing this on the
> latest develop branch, 0.3.0-SNAPSHOT.
>
> Steps taken:
> 1. Create instance of Controller Service in the Controller Settings >
> Controller Services tab
> 1a. Enable that instance of the Controller Service
> 2. Add new Processor on canvas that will use Controller Service
> 3. Configure Processor
> 3a. Existing Controller Service does not show in property options. Only "No
> value" and "Create new service..."
> 3b. Clicking "Create new service..." opens modal with the custom Controller
> Service as the only item in the dropdown
> 3c. Clicking "Create" button succeeds, but puts the UUID of the Controller
> Service in the Value of the Processor Property.
>
> Observed state:
> - Processor is still invalid
> - The ID in the Processor Property matches the ID of the second (disabled)
> instance of the Controller Service in the Controller Settings modal.
>
> There are no errors in nifi-app.log - only contains log messages from the
> StandardControllerServiceProvider that it "Created Controller Service of
> type [my-implementation-class] with identifier
> [UUID-that-shows-in-the-GUI]"
>
> Adding a DetectDuplicate processor with the
> DistributedMapCacheClientService works just fine, so it'd hard to point
> fingers at the framework. Any thoughts on where to begin debugging and/or
> loggers to turn up would be greatly appreciated!
>
> Thanks,
> Brian
>