You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Bimal Mehta <bi...@gmail.com> on 2019/08/07 19:18:36 UTC
Custom Processor Upgrade
Hi,
If we have a custom processor that was created with NiFi 1.6, what are the
steps we need to follow to make it work in 1.9?
Is there some sort of steps that explains the jar and pom updates we need
to do for making it work in 1.9?
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Thanks all for the support.
We resolved the issue by creating a new nar file itself. It seems the old
version code was using some outdated dependencies which was not getting
fixed when upgrading the processor.
On Thu, Aug 15, 2019 at 2:06 PM James Srinivasan <ja...@gmail.com>
wrote:
> I find strace (or procmon for Windows) very handy to debug such resource
> loading issues.
>
> On Thu, 15 Aug 2019, 19:02 Bryan Bende, <bb...@gmail.com> wrote:
>
>> I was making sure you didn't have any code that was dependent on the
>> internal structure of how the NARs are unpacked.
>>
>> I can't explain why it can't find the application-context.xml since I
>> don't have access to your code, but I don't see why that would be
>> related to moving to NAR_INF from META_INF, nothing should really be
>> relying on that structure.
>>
>> On Thu, Aug 15, 2019 at 1:27 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >
>> > Its inside one of the jars within the NAR_INF folder. Does it need to
>> be somehwere else?
>> > Also I think we extended the AbstractNiFiProcessor from the custom kylo
>> processor while migrating it as kylo processor was not working as is in our
>> environment.. Will check that and have it packaged in the processors module.
>> >
>> > On Thu, Aug 15, 2019 at 1:50 AM Bryan Bende <bb...@gmail.com> wrote:
>> >>
>> >> Where is application-context.xml in your NAR?
>> >>
>> >> And how are you trying to load it in
>> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor ?
>> >>
>> >> I would expect it to be packaged into the jar that contains your
>> processors, most likely in src/main/resources of the processors module
>> which then ends up at the root of the jar.
>> >>
>> >> On Wed, Aug 14, 2019 at 5:36 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >>>
>> >>> Ahh, seems like a Springboot error.
>> >>> Is it to do with upgraded Jetty server ?
>> >>>
>> >>> Caused by:
>> org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected
>> exception parsing XML document from class path resource
>> [application-context.xml]; nested exception is
>> org.springframework.beans.FatalBeanException: Class
>> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
>> http://www.springframework.org/schema/context] does not implement the
>> [org.springframework.beans.factory.xml.NamespaceHandler] interface
>> >>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
>> >>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
>> >>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
>> >>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
>> >>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
>> >>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
>> >>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
>> >>> at
>> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
>> >>> at
>> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
>> >>> at
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
>> >>> at
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
>> >>> at
>> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
>> >>> at
>> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
>> >>> at
>> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
>> >>> at
>> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
>> >>> at
>> org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
>> >>> at
>> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
>> >>> ... 50 common frames omitted
>> >>> Caused by: org.springframework.beans.FatalBeanException: Class
>> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
>> http://www.springframework.org/schema/context] does not implement the
>> [org.springframework.beans.factory.xml.NamespaceHandler] interface
>> >>> at
>> org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
>> >>> at
>> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
>> >>> at
>> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
>> >>> at
>> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
>> >>> at
>> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
>> >>> at
>> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
>> >>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
>> >>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
>> >>> ... 66 common frames omitted
>> >>>
>> >>> On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende <bb...@gmail.com> wrote:
>> >>>>
>> >>>> You have to add another instance of the processor which should
>> >>>> generate the same stracktrace you sent earlier, except this time
>> there
>> >>>> should be a second part to it with "Caused by.... " and then more of
>> >>>> the stacktrace that wasn't there before.
>> >>>>
>> >>>> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >>>> >
>> >>>> > Hi Bryan,
>> >>>> >
>> >>>> > I did what you said.
>> >>>> > This is what I got
>> >>>> >
>> >>>> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
>> o.a.n.controller.AbstractComponentNode Computed validation errors with
>> Validation Context StandardValidationContext[componentId=
>> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
>> properties={PropertyDescriptor[Header Line Count]=1,
>> PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
>> Processor' validated against 'Any Property' is invalid because Processor is
>> of type org.apache.nifi.init.InitiateScan, but this is not a valid
>> Processor type]
>> >>>> > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
>> o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
>> >>>> >
>> >>>> > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com>
>> wrote:
>> >>>> >>
>> >>>> >> Can you edit logback.xml and add the following, the get the
>> stacktrace again?
>> >>>> >>
>> >>>> >> <logger name="org.apache.nifi.controller" level="DEBUG" />
>> >>>> >>
>> >>>> >> This should include a root cause exception which we are missing
>> right now.
>> >>>> >>
>> >>>> >> I think it takes about 20-30 seconds for logback to pick up the
>> edits
>> >>>> >> to logback.xml.
>> >>>> >>
>> >>>> >> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >>>> >> >
>> >>>> >> > For the custom processor we have, we are extending the
>> AbstractNifiProcessor.java.
>> >>>> >> > The processor is used to scan metadata of an incoming flow file.
>> >>>> >> > The error we get in logs is as below.
>> >>>> >> > 2019-08-13 23:21:21,529 ERROR [main]
>> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
>> org.apache.nifi.init.InitiateScan for ID
>> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
>> >>>> >> >
>> org.apache.nifi.controller.exception.ProcessorInstantiationException:
>> org.apache.nifi.init.InitiateScan
>> >>>> >> > at
>> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
>> >>>> >> > at
>> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
>> >>>> >> > at
>> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
>> >>>> >> > at
>> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
>> >>>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
>> >>>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> >>>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> >>>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
>> >>>> >> > at
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
>> >>>> >> > at
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
>> >>>> >> > at
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
>> >>>> >> > at
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
>> >>>> >> > at
>> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
>> >>>> >> > at
>> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
>> >>>> >> > at
>> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
>> >>>> >> > at
>> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
>> >>>> >> > at
>> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
>> >>>> >> > at
>> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
>> >>>> >> > at
>> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >>>> >> > at
>> org.eclipse.jetty.server.Server.start(Server.java:419)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>> >>>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >>>> >> > at
>> org.eclipse.jetty.server.Server.doStart(Server.java:386)
>> >>>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >>>> >> > at
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
>> >>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
>> >>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
>> >>>> >> > at org.apache.nifi.NiFi.main(NiFi.java:297)
>> >>>> >> >
>> >>>> >> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com>
>> wrote:
>> >>>> >> >>
>> >>>> >> >> Without access to the code for your NAR I can only really
>> guess, but it sounds like an exception is happening when trying to call
>> the constructor of your processor and then it bounces into creating a ghost
>> processor.
>> >>>> >> >>
>> >>>> >> >> What is in the logs at the time you get the ghost processor?
>> >>>> >> >>
>> >>>> >> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >>>> >> >>>
>> >>>> >> >>> Does that mean I need to recreate the processor? Or there is
>> some workaround?
>> >>>> >> >>>
>> >>>> >> >>> The processor gets unpacked and its bundled dependencies go
>> in NAR_INF.
>> >>>> >> >>> However when I drag the processor on the canvas, it comes
>> with a yellow triangle (and gives the error message I stated above) and
>> properties are missing as well.
>> >>>> >> >>>
>> >>>> >> >>>
>> >>>> >> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <
>> bbende@gmail.com> wrote:
>> >>>> >> >>>>
>> >>>> >> >>>> I don’t remember all the reasoning behind the change, but it
>> had to do with an issue when we upgraded Jetty...
>> >>>> >> >>>>
>> >>>> >> >>>> https://issues.apache.org/jira/browse/NIFI-5479
>> >>>> >> >>>>
>> >>>> >> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >>>> >> >>>>>
>> >>>> >> >>>>> Yes it does show as an option.
>> >>>> >> >>>>> One thing I noticed is that the when the nar is unpacked,
>> the bundled dependencies are inside META_INF in the work folder in NiFi
>> 1.6.0, however in NiFI 1.9.0 they go inside NAR_INF.
>> >>>> >> >>>>> Why does this happen?
>> >>>> >> >>>>> It seems the custom processor that we have uses Springboot,
>> and references applicationcontext file which was inside META_INF when it
>> was built. However I cant see that file anymore in the unpacked nar.
>> >>>> >> >>>>>
>> >>>> >> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <
>> bbende@gmail.com> wrote:
>> >>>> >> >>>>>>
>> >>>> >> >>>>>> Does that custom processor type show as an option if you
>> try to add a new processor to the canvas?
>> >>>> >> >>>>>>
>> >>>> >> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>> Hi Mike and Bryan,
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>> One of my custom processors appears as inactive in NiFi
>> with a yellow triangle error.
>> >>>> >> >>>>>>> When I hover over it I see a message saying 'Missing
>> Processor' validated against 'Any Property' is invalid. This is not a valid
>> processor.
>> >>>> >> >>>>>>> In the log it seems to invoke GhostProcessor.java which
>> is giving the above error when restarting nifi.
>> >>>> >> >>>>>>> This custom processor sits (with my other processors) in
>> my custom_lib folder and I have provided that path in the nifi properties
>> file as
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>> Not sure what I missed?
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>> Do I need to make entry of this custom processor
>> somewhere?
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>>
>> >>>> >> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >>>> >> >>>>>>>>
>> >>>> >> >>>>>>>> Thanks Mike and Bryan.
>> >>>> >> >>>>>>>> Yes it seems my template was still referring the old
>> version.
>> >>>> >> >>>>>>>> I will have it updated now and will reimport.
>> >>>> >> >>>>>>>> Also the version of NiFi we are using is the one that
>> comes with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will
>> reach out to Cloudera and see if we can get it upgraded.
>> >>>> >> >>>>>>>>
>> >>>> >> >>>>>>>>
>> >>>> >> >>>>>>>>
>> >>>> >> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <
>> bbende@gmail.com> wrote:
>> >>>> >> >>>>>>>>>
>> >>>> >> >>>>>>>>> What is in the template for the bundle coordinates of
>> your processor?
>> >>>> >> >>>>>>>>> and does that match the coordinates of the NAR that is
>> deployed?
>> >>>> >> >>>>>>>>>
>> >>>> >> >>>>>>>>> Example:
>> >>>> >> >>>>>>>>>
>> >>>> >> >>>>>>>>> <bundle>
>> >>>> >> >>>>>>>>> <group>org.apache.nifi</group>
>> >>>> >> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>> >>>> >> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>> >>>> >> >>>>>>>>> </bundle>
>> >>>> >> >>>>>>>>>
>> >>>> >> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and
>> your template
>> >>>> >> >>>>>>>>> references 1.0.0, then you'll need to update your
>> template.
>> >>>> >> >>>>>>>>>
>> >>>> >> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
>> mikerthomsen@gmail.com> wrote:
>> >>>> >> >>>>>>>>> >
>> >>>> >> >>>>>>>>> > If it's happening immediately upon trying to import
>> the template, I believe that's the error message saying that the 1.9
>> instance cannot find the NAR file which provided the processor. Also, if
>> you're referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to
>> the latter because there are a few critical bugs fixed in 1.9.2.
>> >>>> >> >>>>>>>>> >
>> >>>> >> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >>>> >> >>>>>>>>> >>
>> >>>> >> >>>>>>>>> >> Thanks Bryan.
>> >>>> >> >>>>>>>>> >> My custom processors are part of a template. However
>> when I try to import my template in NiFi 1.9, I get an error message saying
>> >>>> >> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance.
>> I did update all the dependencies to NiFi 1.9 and even the plugins. We are
>> using a Cloudera distributed version of NiFi 1.9.
>> >>>> >> >>>>>>>>> >> Any idea why is this happening?
>> >>>> >> >>>>>>>>> >>
>> >>>> >> >>>>>>>>> >> Thanks
>> >>>> >> >>>>>>>>> >>
>> >>>> >> >>>>>>>>> >>
>> >>>> >> >>>>>>>>> >>
>> >>>> >> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <
>> bbende@gmail.com> wrote:
>> >>>> >> >>>>>>>>> >>>
>> >>>> >> >>>>>>>>> >>> Hello,
>> >>>> >> >>>>>>>>> >>>
>> >>>> >> >>>>>>>>> >>> Most likely your processor built against 1.6 would
>> run fine in 1.9,
>> >>>> >> >>>>>>>>> >>> but to make sure you just need to update any nifi
>> dependencies in your
>> >>>> >> >>>>>>>>> >>> poms to 1.9.2.
>> >>>> >> >>>>>>>>> >>>
>> >>>> >> >>>>>>>>> >>> If you created your project from the archetype and
>> didn't change
>> >>>> >> >>>>>>>>> >>> anything, then this should just be changing the
>> parent in the root pom
>> >>>> >> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
>> >>>> >> >>>>>>>>> >>>
>> >>>> >> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend
>> on nifi-api you
>> >>>> >> >>>>>>>>> >>> need to change.
>> >>>> >> >>>>>>>>> >>>
>> >>>> >> >>>>>>>>> >>> -Bryan
>> >>>> >> >>>>>>>>> >>>
>> >>>> >> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >>>> >> >>>>>>>>> >>> >
>> >>>> >> >>>>>>>>> >>> > Hi,
>> >>>> >> >>>>>>>>> >>> >
>> >>>> >> >>>>>>>>> >>> > If we have a custom processor that was created
>> with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>> >>>> >> >>>>>>>>> >>> > Is there some sort of steps that explains the jar
>> and pom updates we need to do for making it work in 1.9?
>> >>>> >> >>>>>>
>> >>>> >> >>>>>> --
>> >>>> >> >>>>>> Sent from Gmail Mobile
>> >>>> >> >>>>
>> >>>> >> >>>> --
>> >>>> >> >>>> Sent from Gmail Mobile
>> >>>> >> >>
>> >>>> >> >> --
>> >>>> >> >> Sent from Gmail Mobile
>> >>
>> >> --
>> >> Sent from Gmail Mobile
>>
>
Re: Custom Processor Upgrade
Posted by James Srinivasan <ja...@gmail.com>.
I find strace (or procmon for Windows) very handy to debug such resource
loading issues.
On Thu, 15 Aug 2019, 19:02 Bryan Bende, <bb...@gmail.com> wrote:
> I was making sure you didn't have any code that was dependent on the
> internal structure of how the NARs are unpacked.
>
> I can't explain why it can't find the application-context.xml since I
> don't have access to your code, but I don't see why that would be
> related to moving to NAR_INF from META_INF, nothing should really be
> relying on that structure.
>
> On Thu, Aug 15, 2019 at 1:27 PM Bimal Mehta <bi...@gmail.com> wrote:
> >
> > Its inside one of the jars within the NAR_INF folder. Does it need to be
> somehwere else?
> > Also I think we extended the AbstractNiFiProcessor from the custom kylo
> processor while migrating it as kylo processor was not working as is in our
> environment.. Will check that and have it packaged in the processors module.
> >
> > On Thu, Aug 15, 2019 at 1:50 AM Bryan Bende <bb...@gmail.com> wrote:
> >>
> >> Where is application-context.xml in your NAR?
> >>
> >> And how are you trying to load it in
> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor ?
> >>
> >> I would expect it to be packaged into the jar that contains your
> processors, most likely in src/main/resources of the processors module
> which then ends up at the root of the jar.
> >>
> >> On Wed, Aug 14, 2019 at 5:36 PM Bimal Mehta <bi...@gmail.com> wrote:
> >>>
> >>> Ahh, seems like a Springboot error.
> >>> Is it to do with upgraded Jetty server ?
> >>>
> >>> Caused by:
> org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected
> exception parsing XML document from class path resource
> [application-context.xml]; nested exception is
> org.springframework.beans.FatalBeanException: Class
> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
> http://www.springframework.org/schema/context] does not implement the
> [org.springframework.beans.factory.xml.NamespaceHandler] interface
> >>> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
> >>> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
> >>> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
> >>> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
> >>> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
> >>> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
> >>> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
> >>> at
> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
> >>> at
> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
> >>> at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
> >>> at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
> >>> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
> >>> at
> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
> >>> at
> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
> >>> at
> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
> >>> at
> org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
> >>> at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
> >>> ... 50 common frames omitted
> >>> Caused by: org.springframework.beans.FatalBeanException: Class
> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
> http://www.springframework.org/schema/context] does not implement the
> [org.springframework.beans.factory.xml.NamespaceHandler] interface
> >>> at
> org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
> >>> at
> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
> >>> at
> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
> >>> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
> >>> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
> >>> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
> >>> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
> >>> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
> >>> ... 66 common frames omitted
> >>>
> >>> On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende <bb...@gmail.com> wrote:
> >>>>
> >>>> You have to add another instance of the processor which should
> >>>> generate the same stracktrace you sent earlier, except this time there
> >>>> should be a second part to it with "Caused by.... " and then more of
> >>>> the stacktrace that wasn't there before.
> >>>>
> >>>> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>> >
> >>>> > Hi Bryan,
> >>>> >
> >>>> > I did what you said.
> >>>> > This is what I got
> >>>> >
> >>>> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
> o.a.n.controller.AbstractComponentNode Computed validation errors with
> Validation Context StandardValidationContext[componentId=
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
> properties={PropertyDescriptor[Header Line Count]=1,
> PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
> Processor' validated against 'Any Property' is invalid because Processor is
> of type org.apache.nifi.init.InitiateScan, but this is not a valid
> Processor type]
> >>>> > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
> o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
> >>>> >
> >>>> > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com>
> wrote:
> >>>> >>
> >>>> >> Can you edit logback.xml and add the following, the get the
> stacktrace again?
> >>>> >>
> >>>> >> <logger name="org.apache.nifi.controller" level="DEBUG" />
> >>>> >>
> >>>> >> This should include a root cause exception which we are missing
> right now.
> >>>> >>
> >>>> >> I think it takes about 20-30 seconds for logback to pick up the
> edits
> >>>> >> to logback.xml.
> >>>> >>
> >>>> >> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>> >> >
> >>>> >> > For the custom processor we have, we are extending the
> AbstractNifiProcessor.java.
> >>>> >> > The processor is used to scan metadata of an incoming flow file.
> >>>> >> > The error we get in logs is as below.
> >>>> >> > 2019-08-13 23:21:21,529 ERROR [main]
> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
> org.apache.nifi.init.InitiateScan for ID
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
> >>>> >> >
> org.apache.nifi.controller.exception.ProcessorInstantiationException:
> org.apache.nifi.init.InitiateScan
> >>>> >> > at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
> >>>> >> > at
> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
> >>>> >> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
> >>>> >> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
> >>>> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
> >>>> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> >>>> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> >>>> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
> >>>> >> > at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
> >>>> >> > at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
> >>>> >> > at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
> >>>> >> > at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
> >>>> >> > at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> >>>> >> > at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
> >>>> >> > at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
> >>>> >> > at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
> >>>> >> > at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
> >>>> >> > at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
> >>>> >> > at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
> >>>> >> > at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
> >>>> >> > at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
> >>>> >> > at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
> >>>> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> >>>> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >>>> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> >>>> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >>>> >> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
> >>>> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> >>>> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >>>> >> > at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
> >>>> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> >>>> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >>>> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >>>> >> > at org.eclipse.jetty.server.Server.start(Server.java:419)
> >>>> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> >>>> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >>>> >> > at
> org.eclipse.jetty.server.Server.doStart(Server.java:386)
> >>>> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >>>> >> > at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
> >>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
> >>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
> >>>> >> > at org.apache.nifi.NiFi.main(NiFi.java:297)
> >>>> >> >
> >>>> >> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com>
> wrote:
> >>>> >> >>
> >>>> >> >> Without access to the code for your NAR I can only really
> guess, but it sounds like an exception is happening when trying to call
> the constructor of your processor and then it bounces into creating a ghost
> processor.
> >>>> >> >>
> >>>> >> >> What is in the logs at the time you get the ghost processor?
> >>>> >> >>
> >>>> >> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>> >> >>>
> >>>> >> >>> Does that mean I need to recreate the processor? Or there is
> some workaround?
> >>>> >> >>>
> >>>> >> >>> The processor gets unpacked and its bundled dependencies go in
> NAR_INF.
> >>>> >> >>> However when I drag the processor on the canvas, it comes with
> a yellow triangle (and gives the error message I stated above) and
> properties are missing as well.
> >>>> >> >>>
> >>>> >> >>>
> >>>> >> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com>
> wrote:
> >>>> >> >>>>
> >>>> >> >>>> I don’t remember all the reasoning behind the change, but it
> had to do with an issue when we upgraded Jetty...
> >>>> >> >>>>
> >>>> >> >>>> https://issues.apache.org/jira/browse/NIFI-5479
> >>>> >> >>>>
> >>>> >> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>> >> >>>>>
> >>>> >> >>>>> Yes it does show as an option.
> >>>> >> >>>>> One thing I noticed is that the when the nar is unpacked,
> the bundled dependencies are inside META_INF in the work folder in NiFi
> 1.6.0, however in NiFI 1.9.0 they go inside NAR_INF.
> >>>> >> >>>>> Why does this happen?
> >>>> >> >>>>> It seems the custom processor that we have uses Springboot,
> and references applicationcontext file which was inside META_INF when it
> was built. However I cant see that file anymore in the unpacked nar.
> >>>> >> >>>>>
> >>>> >> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <
> bbende@gmail.com> wrote:
> >>>> >> >>>>>>
> >>>> >> >>>>>> Does that custom processor type show as an option if you
> try to add a new processor to the canvas?
> >>>> >> >>>>>>
> >>>> >> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>> >> >>>>>>>
> >>>> >> >>>>>>> Hi Mike and Bryan,
> >>>> >> >>>>>>>
> >>>> >> >>>>>>> One of my custom processors appears as inactive in NiFi
> with a yellow triangle error.
> >>>> >> >>>>>>> When I hover over it I see a message saying 'Missing
> Processor' validated against 'Any Property' is invalid. This is not a valid
> processor.
> >>>> >> >>>>>>> In the log it seems to invoke GhostProcessor.java which is
> giving the above error when restarting nifi.
> >>>> >> >>>>>>> This custom processor sits (with my other processors) in
> my custom_lib folder and I have provided that path in the nifi properties
> file as
> >>>> >> >>>>>>>
> >>>> >> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
> >>>> >> >>>>>>>
> >>>> >> >>>>>>>
> >>>> >> >>>>>>> Not sure what I missed?
> >>>> >> >>>>>>>
> >>>> >> >>>>>>> Do I need to make entry of this custom processor somewhere?
> >>>> >> >>>>>>>
> >>>> >> >>>>>>>
> >>>> >> >>>>>>>
> >>>> >> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>> >> >>>>>>>>
> >>>> >> >>>>>>>> Thanks Mike and Bryan.
> >>>> >> >>>>>>>> Yes it seems my template was still referring the old
> version.
> >>>> >> >>>>>>>> I will have it updated now and will reimport.
> >>>> >> >>>>>>>> Also the version of NiFi we are using is the one that
> comes with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will
> reach out to Cloudera and see if we can get it upgraded.
> >>>> >> >>>>>>>>
> >>>> >> >>>>>>>>
> >>>> >> >>>>>>>>
> >>>> >> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <
> bbende@gmail.com> wrote:
> >>>> >> >>>>>>>>>
> >>>> >> >>>>>>>>> What is in the template for the bundle coordinates of
> your processor?
> >>>> >> >>>>>>>>> and does that match the coordinates of the NAR that is
> deployed?
> >>>> >> >>>>>>>>>
> >>>> >> >>>>>>>>> Example:
> >>>> >> >>>>>>>>>
> >>>> >> >>>>>>>>> <bundle>
> >>>> >> >>>>>>>>> <group>org.apache.nifi</group>
> >>>> >> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
> >>>> >> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
> >>>> >> >>>>>>>>> </bundle>
> >>>> >> >>>>>>>>>
> >>>> >> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and
> your template
> >>>> >> >>>>>>>>> references 1.0.0, then you'll need to update your
> template.
> >>>> >> >>>>>>>>>
> >>>> >> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
> mikerthomsen@gmail.com> wrote:
> >>>> >> >>>>>>>>> >
> >>>> >> >>>>>>>>> > If it's happening immediately upon trying to import
> the template, I believe that's the error message saying that the 1.9
> instance cannot find the NAR file which provided the processor. Also, if
> you're referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to
> the latter because there are a few critical bugs fixed in 1.9.2.
> >>>> >> >>>>>>>>> >
> >>>> >> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>> >> >>>>>>>>> >>
> >>>> >> >>>>>>>>> >> Thanks Bryan.
> >>>> >> >>>>>>>>> >> My custom processors are part of a template. However
> when I try to import my template in NiFi 1.9, I get an error message saying
> >>>> >> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I
> did update all the dependencies to NiFi 1.9 and even the plugins. We are
> using a Cloudera distributed version of NiFi 1.9.
> >>>> >> >>>>>>>>> >> Any idea why is this happening?
> >>>> >> >>>>>>>>> >>
> >>>> >> >>>>>>>>> >> Thanks
> >>>> >> >>>>>>>>> >>
> >>>> >> >>>>>>>>> >>
> >>>> >> >>>>>>>>> >>
> >>>> >> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <
> bbende@gmail.com> wrote:
> >>>> >> >>>>>>>>> >>>
> >>>> >> >>>>>>>>> >>> Hello,
> >>>> >> >>>>>>>>> >>>
> >>>> >> >>>>>>>>> >>> Most likely your processor built against 1.6 would
> run fine in 1.9,
> >>>> >> >>>>>>>>> >>> but to make sure you just need to update any nifi
> dependencies in your
> >>>> >> >>>>>>>>> >>> poms to 1.9.2.
> >>>> >> >>>>>>>>> >>>
> >>>> >> >>>>>>>>> >>> If you created your project from the archetype and
> didn't change
> >>>> >> >>>>>>>>> >>> anything, then this should just be changing the
> parent in the root pom
> >>>> >> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
> >>>> >> >>>>>>>>> >>>
> >>>> >> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend
> on nifi-api you
> >>>> >> >>>>>>>>> >>> need to change.
> >>>> >> >>>>>>>>> >>>
> >>>> >> >>>>>>>>> >>> -Bryan
> >>>> >> >>>>>>>>> >>>
> >>>> >> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>> >> >>>>>>>>> >>> >
> >>>> >> >>>>>>>>> >>> > Hi,
> >>>> >> >>>>>>>>> >>> >
> >>>> >> >>>>>>>>> >>> > If we have a custom processor that was created
> with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
> >>>> >> >>>>>>>>> >>> > Is there some sort of steps that explains the jar
> and pom updates we need to do for making it work in 1.9?
> >>>> >> >>>>>>
> >>>> >> >>>>>> --
> >>>> >> >>>>>> Sent from Gmail Mobile
> >>>> >> >>>>
> >>>> >> >>>> --
> >>>> >> >>>> Sent from Gmail Mobile
> >>>> >> >>
> >>>> >> >> --
> >>>> >> >> Sent from Gmail Mobile
> >>
> >> --
> >> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
I was making sure you didn't have any code that was dependent on the
internal structure of how the NARs are unpacked.
I can't explain why it can't find the application-context.xml since I
don't have access to your code, but I don't see why that would be
related to moving to NAR_INF from META_INF, nothing should really be
relying on that structure.
On Thu, Aug 15, 2019 at 1:27 PM Bimal Mehta <bi...@gmail.com> wrote:
>
> Its inside one of the jars within the NAR_INF folder. Does it need to be somehwere else?
> Also I think we extended the AbstractNiFiProcessor from the custom kylo processor while migrating it as kylo processor was not working as is in our environment.. Will check that and have it packaged in the processors module.
>
> On Thu, Aug 15, 2019 at 1:50 AM Bryan Bende <bb...@gmail.com> wrote:
>>
>> Where is application-context.xml in your NAR?
>>
>> And how are you trying to load it in com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor ?
>>
>> I would expect it to be packaged into the jar that contains your processors, most likely in src/main/resources of the processors module which then ends up at the root of the jar.
>>
>> On Wed, Aug 14, 2019 at 5:36 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>
>>> Ahh, seems like a Springboot error.
>>> Is it to do with upgraded Jetty server ?
>>>
>>> Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from class path resource [application-context.xml]; nested exception is org.springframework.beans.FatalBeanException: Class [org.springframework.context.config.ContextNamespaceHandler] for namespace [http://www.springframework.org/schema/context] does not implement the [org.springframework.beans.factory.xml.NamespaceHandler] interface
>>> at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
>>> at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
>>> at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
>>> at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
>>> at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
>>> at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
>>> at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
>>> at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
>>> at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
>>> at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
>>> at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
>>> at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
>>> at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
>>> at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
>>> at com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
>>> at org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
>>> at org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
>>> ... 50 common frames omitted
>>> Caused by: org.springframework.beans.FatalBeanException: Class [org.springframework.context.config.ContextNamespaceHandler] for namespace [http://www.springframework.org/schema/context] does not implement the [org.springframework.beans.factory.xml.NamespaceHandler] interface
>>> at org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
>>> at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
>>> at org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
>>> at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
>>> at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
>>> at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
>>> at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
>>> at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
>>> ... 66 common frames omitted
>>>
>>> On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> You have to add another instance of the processor which should
>>>> generate the same stracktrace you sent earlier, except this time there
>>>> should be a second part to it with "Caused by.... " and then more of
>>>> the stacktrace that wasn't there before.
>>>>
>>>> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >
>>>> > Hi Bryan,
>>>> >
>>>> > I did what you said.
>>>> > This is what I got
>>>> >
>>>> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3] o.a.n.controller.AbstractComponentNode Computed validation errors with Validation Context StandardValidationContext[componentId= 6fbe2407-7799-3908-f4c4-bf2f8940bf1e , properties={PropertyDescriptor[Header Line Count]=1, PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing Processor' validated against 'Any Property' is invalid because Processor is of type org.apache.nifi.init.InitiateScan, but this is not a valid Processor type]
>>>> > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6] o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
>>>> >
>>>> > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com> wrote:
>>>> >>
>>>> >> Can you edit logback.xml and add the following, the get the stacktrace again?
>>>> >>
>>>> >> <logger name="org.apache.nifi.controller" level="DEBUG" />
>>>> >>
>>>> >> This should include a root cause exception which we are missing right now.
>>>> >>
>>>> >> I think it takes about 20-30 seconds for logback to pick up the edits
>>>> >> to logback.xml.
>>>> >>
>>>> >> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >
>>>> >> > For the custom processor we have, we are extending the AbstractNifiProcessor.java.
>>>> >> > The processor is used to scan metadata of an incoming flow file.
>>>> >> > The error we get in logs is as below.
>>>> >> > 2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder Could not create Processor of type org.apache.nifi.init.InitiateScan for ID 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
>>>> >> > org.apache.nifi.controller.exception.ProcessorInstantiationException: org.apache.nifi.init.InitiateScan
>>>> >> > at org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
>>>> >> > at org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
>>>> >> > at org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
>>>> >> > at org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
>>>> >> > at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
>>>> >> > at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>>>> >> > at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>>>> >> > at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
>>>> >> > at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
>>>> >> > at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
>>>> >> > at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
>>>> >> > at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
>>>> >> > at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
>>>> >> > at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
>>>> >> > at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
>>>> >> > at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
>>>> >> > at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
>>>> >> > at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
>>>> >> > at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
>>>> >> > at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
>>>> >> > at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
>>>> >> > at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>>>> >> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>>>> >> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>>> >> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>>>> >> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>>> >> > at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
>>>> >> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>>>> >> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>>> >> > at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
>>>> >> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>>>> >> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>>> >> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>>> >> > at org.eclipse.jetty.server.Server.start(Server.java:419)
>>>> >> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>>>> >> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>>> >> > at org.eclipse.jetty.server.Server.doStart(Server.java:386)
>>>> >> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>>> >> > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
>>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
>>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
>>>> >> > at org.apache.nifi.NiFi.main(NiFi.java:297)
>>>> >> >
>>>> >> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com> wrote:
>>>> >> >>
>>>> >> >> Without access to the code for your NAR I can only really guess, but it sounds like an exception is happening when trying to call the constructor of your processor and then it bounces into creating a ghost processor.
>>>> >> >>
>>>> >> >> What is in the logs at the time you get the ghost processor?
>>>> >> >>
>>>> >> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >>>
>>>> >> >>> Does that mean I need to recreate the processor? Or there is some workaround?
>>>> >> >>>
>>>> >> >>> The processor gets unpacked and its bundled dependencies go in NAR_INF.
>>>> >> >>> However when I drag the processor on the canvas, it comes with a yellow triangle (and gives the error message I stated above) and properties are missing as well.
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
>>>> >> >>>>
>>>> >> >>>> I don’t remember all the reasoning behind the change, but it had to do with an issue when we upgraded Jetty...
>>>> >> >>>>
>>>> >> >>>> https://issues.apache.org/jira/browse/NIFI-5479
>>>> >> >>>>
>>>> >> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >>>>>
>>>> >> >>>>> Yes it does show as an option.
>>>> >> >>>>> One thing I noticed is that the when the nar is unpacked, the bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0, however in NiFI 1.9.0 they go inside NAR_INF.
>>>> >> >>>>> Why does this happen?
>>>> >> >>>>> It seems the custom processor that we have uses Springboot, and references applicationcontext file which was inside META_INF when it was built. However I cant see that file anymore in the unpacked nar.
>>>> >> >>>>>
>>>> >> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>>>> >> >>>>>>
>>>> >> >>>>>> Does that custom processor type show as an option if you try to add a new processor to the canvas?
>>>> >> >>>>>>
>>>> >> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >>>>>>>
>>>> >> >>>>>>> Hi Mike and Bryan,
>>>> >> >>>>>>>
>>>> >> >>>>>>> One of my custom processors appears as inactive in NiFi with a yellow triangle error.
>>>> >> >>>>>>> When I hover over it I see a message saying 'Missing Processor' validated against 'Any Property' is invalid. This is not a valid processor.
>>>> >> >>>>>>> In the log it seems to invoke GhostProcessor.java which is giving the above error when restarting nifi.
>>>> >> >>>>>>> This custom processor sits (with my other processors) in my custom_lib folder and I have provided that path in the nifi properties file as
>>>> >> >>>>>>>
>>>> >> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
>>>> >> >>>>>>>
>>>> >> >>>>>>>
>>>> >> >>>>>>> Not sure what I missed?
>>>> >> >>>>>>>
>>>> >> >>>>>>> Do I need to make entry of this custom processor somewhere?
>>>> >> >>>>>>>
>>>> >> >>>>>>>
>>>> >> >>>>>>>
>>>> >> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >>>>>>>>
>>>> >> >>>>>>>> Thanks Mike and Bryan.
>>>> >> >>>>>>>> Yes it seems my template was still referring the old version.
>>>> >> >>>>>>>> I will have it updated now and will reimport.
>>>> >> >>>>>>>> Also the version of NiFi we are using is the one that comes with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera and see if we can get it upgraded.
>>>> >> >>>>>>>>
>>>> >> >>>>>>>>
>>>> >> >>>>>>>>
>>>> >> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> What is in the template for the bundle coordinates of your processor?
>>>> >> >>>>>>>>> and does that match the coordinates of the NAR that is deployed?
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> Example:
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> <bundle>
>>>> >> >>>>>>>>> <group>org.apache.nifi</group>
>>>> >> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>> >> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>>>> >> >>>>>>>>> </bundle>
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>> >> >>>>>>>>> references 1.0.0, then you'll need to update your template.
>>>> >> >>>>>>>>>
>>>> >> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com> wrote:
>>>> >> >>>>>>>>> >
>>>> >> >>>>>>>>> > If it's happening immediately upon trying to import the template, I believe that's the error message saying that the 1.9 instance cannot find the NAR file which provided the processor. Also, if you're referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because there are a few critical bugs fixed in 1.9.2.
>>>> >> >>>>>>>>> >
>>>> >> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >>>>>>>>> >>
>>>> >> >>>>>>>>> >> Thanks Bryan.
>>>> >> >>>>>>>>> >> My custom processors are part of a template. However when I try to import my template in NiFi 1.9, I get an error message saying
>>>> >> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update all the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera distributed version of NiFi 1.9.
>>>> >> >>>>>>>>> >> Any idea why is this happening?
>>>> >> >>>>>>>>> >>
>>>> >> >>>>>>>>> >> Thanks
>>>> >> >>>>>>>>> >>
>>>> >> >>>>>>>>> >>
>>>> >> >>>>>>>>> >>
>>>> >> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>>>> >> >>>>>>>>> >>>
>>>> >> >>>>>>>>> >>> Hello,
>>>> >> >>>>>>>>> >>>
>>>> >> >>>>>>>>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>>>> >> >>>>>>>>> >>> but to make sure you just need to update any nifi dependencies in your
>>>> >> >>>>>>>>> >>> poms to 1.9.2.
>>>> >> >>>>>>>>> >>>
>>>> >> >>>>>>>>> >>> If you created your project from the archetype and didn't change
>>>> >> >>>>>>>>> >>> anything, then this should just be changing the parent in the root pom
>>>> >> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
>>>> >> >>>>>>>>> >>>
>>>> >> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>>>> >> >>>>>>>>> >>> need to change.
>>>> >> >>>>>>>>> >>>
>>>> >> >>>>>>>>> >>> -Bryan
>>>> >> >>>>>>>>> >>>
>>>> >> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>> >> >>>>>>>>> >>> >
>>>> >> >>>>>>>>> >>> > Hi,
>>>> >> >>>>>>>>> >>> >
>>>> >> >>>>>>>>> >>> > If we have a custom processor that was created with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>>>> >> >>>>>>>>> >>> > Is there some sort of steps that explains the jar and pom updates we need to do for making it work in 1.9?
>>>> >> >>>>>>
>>>> >> >>>>>> --
>>>> >> >>>>>> Sent from Gmail Mobile
>>>> >> >>>>
>>>> >> >>>> --
>>>> >> >>>> Sent from Gmail Mobile
>>>> >> >>
>>>> >> >> --
>>>> >> >> Sent from Gmail Mobile
>>
>> --
>> Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Its inside one of the jars within the NAR_INF folder. Does it need to be
somehwere else?
Also I think we extended the AbstractNiFiProcessor from the custom kylo
processor while migrating it as kylo processor was not working as is in our
environment.. Will check that and have it packaged in the processors module.
On Thu, Aug 15, 2019 at 1:50 AM Bryan Bende <bb...@gmail.com> wrote:
> Where is application-context.xml in your NAR?
>
> And how are you trying to load it in com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor
> ?
>
> I would expect it to be packaged into the jar that contains your
> processors, most likely in src/main/resources of the processors module
> which then ends up at the root of the jar.
>
> On Wed, Aug 14, 2019 at 5:36 PM Bimal Mehta <bi...@gmail.com> wrote:
>
>> Ahh, seems like a Springboot error.
>> Is it to do with upgraded Jetty server ?
>>
>> Caused by:
>> org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected
>> exception parsing XML document from class path resource
>> [application-context.xml]; nested exception is
>> org.springframework.beans.FatalBeanException: Class
>> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
>> http://www.springframework.org/schema/context] does not implement the
>> [org.springframework.beans.factory.xml.NamespaceHandler] interface
>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
>> at
>> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
>> at
>> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
>> at
>> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
>> at
>> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
>> at
>> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
>> at
>> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
>> at
>> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
>> at
>> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
>> at
>> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
>> at
>> org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
>> at
>> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
>> ... 50 common frames omitted
>> Caused by: org.springframework.beans.FatalBeanException: Class
>> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
>> http://www.springframework.org/schema/context] does not implement the
>> [org.springframework.beans.factory.xml.NamespaceHandler] interface
>> at
>> org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
>> at
>> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
>> at
>> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
>> at
>> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
>> at
>> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
>> at
>> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
>> at
>> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
>> ... 66 common frames omitted
>>
>> On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende <bb...@gmail.com> wrote:
>>
>>> You have to add another instance of the processor which should
>>> generate the same stracktrace you sent earlier, except this time there
>>> should be a second part to it with "Caused by.... " and then more of
>>> the stacktrace that wasn't there before.
>>>
>>> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com> wrote:
>>> >
>>> > Hi Bryan,
>>> >
>>> > I did what you said.
>>> > This is what I got
>>> >
>>> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
>>> o.a.n.controller.AbstractComponentNode Computed validation errors with
>>> Validation Context StandardValidationContext[componentId=
>>> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
>>> properties={PropertyDescriptor[Header Line Count]=1,
>>> PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
>>> Processor' validated against 'Any Property' is invalid because Processor is
>>> of type org.apache.nifi.init.InitiateScan, but this is not a valid
>>> Processor type]
>>> > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
>>> o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
>>> >
>>> > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com> wrote:
>>> >>
>>> >> Can you edit logback.xml and add the following, the get the
>>> stacktrace again?
>>> >>
>>> >> <logger name="org.apache.nifi.controller" level="DEBUG" />
>>> >>
>>> >> This should include a root cause exception which we are missing right
>>> now.
>>> >>
>>> >> I think it takes about 20-30 seconds for logback to pick up the edits
>>> >> to logback.xml.
>>> >>
>>> >> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com>
>>> wrote:
>>> >> >
>>> >> > For the custom processor we have, we are extending the
>>> AbstractNifiProcessor.java.
>>> >> > The processor is used to scan metadata of an incoming flow file.
>>> >> > The error we get in logs is as below.
>>> >> > 2019-08-13 23:21:21,529 ERROR [main]
>>> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
>>> org.apache.nifi.init.InitiateScan for ID
>>> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
>>> >> >
>>> org.apache.nifi.controller.exception.ProcessorInstantiationException:
>>> org.apache.nifi.init.InitiateScan
>>> >> > at
>>> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
>>> >> > at
>>> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
>>> >> > at
>>> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
>>> >> > at
>>> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
>>> >> > at
>>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
>>> >> > at
>>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>>> >> > at
>>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>>> >> > at
>>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
>>> >> > at
>>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
>>> >> > at
>>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
>>> >> > at
>>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
>>> >> > at
>>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
>>> >> > at
>>> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
>>> >> > at
>>> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
>>> >> > at
>>> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
>>> >> > at
>>> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
>>> >> > at
>>> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
>>> >> > at
>>> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
>>> >> > at
>>> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
>>> >> > at
>>> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
>>> >> > at
>>> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
>>> >> > at
>>> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>>> >> > at
>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>>> >> > at
>>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>> >> > at
>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>>> >> > at
>>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>> >> > at
>>> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
>>> >> > at
>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>>> >> > at
>>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>> >> > at
>>> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
>>> >> > at
>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>>> >> > at
>>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>> >> > at
>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>>> >> > at org.eclipse.jetty.server.Server.start(Server.java:419)
>>> >> > at
>>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>>> >> > at
>>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>>> >> > at org.eclipse.jetty.server.Server.doStart(Server.java:386)
>>> >> > at
>>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>>> >> > at
>>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
>>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
>>> >> > at org.apache.nifi.NiFi.main(NiFi.java:297)
>>> >> >
>>> >> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com>
>>> wrote:
>>> >> >>
>>> >> >> Without access to the code for your NAR I can only really guess,
>>> but it sounds like an exception is happening when trying to call the
>>> constructor of your processor and then it bounces into creating a ghost
>>> processor.
>>> >> >>
>>> >> >> What is in the logs at the time you get the ghost processor?
>>> >> >>
>>> >> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com>
>>> wrote:
>>> >> >>>
>>> >> >>> Does that mean I need to recreate the processor? Or there is some
>>> workaround?
>>> >> >>>
>>> >> >>> The processor gets unpacked and its bundled dependencies go in
>>> NAR_INF.
>>> >> >>> However when I drag the processor on the canvas, it comes with a
>>> yellow triangle (and gives the error message I stated above) and properties
>>> are missing as well.
>>> >> >>>
>>> >> >>>
>>> >> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com>
>>> wrote:
>>> >> >>>>
>>> >> >>>> I don’t remember all the reasoning behind the change, but it had
>>> to do with an issue when we upgraded Jetty...
>>> >> >>>>
>>> >> >>>> https://issues.apache.org/jira/browse/NIFI-5479
>>> >> >>>>
>>> >> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com>
>>> wrote:
>>> >> >>>>>
>>> >> >>>>> Yes it does show as an option.
>>> >> >>>>> One thing I noticed is that the when the nar is unpacked, the
>>> bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0,
>>> however in NiFI 1.9.0 they go inside NAR_INF.
>>> >> >>>>> Why does this happen?
>>> >> >>>>> It seems the custom processor that we have uses Springboot, and
>>> references applicationcontext file which was inside META_INF when it was
>>> built. However I cant see that file anymore in the unpacked nar.
>>> >> >>>>>
>>> >> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com>
>>> wrote:
>>> >> >>>>>>
>>> >> >>>>>> Does that custom processor type show as an option if you try
>>> to add a new processor to the canvas?
>>> >> >>>>>>
>>> >> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <
>>> bimal007@gmail.com> wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> Hi Mike and Bryan,
>>> >> >>>>>>>
>>> >> >>>>>>> One of my custom processors appears as inactive in NiFi with
>>> a yellow triangle error.
>>> >> >>>>>>> When I hover over it I see a message saying 'Missing
>>> Processor' validated against 'Any Property' is invalid. This is not a valid
>>> processor.
>>> >> >>>>>>> In the log it seems to invoke GhostProcessor.java which is
>>> giving the above error when restarting nifi.
>>> >> >>>>>>> This custom processor sits (with my other processors) in my
>>> custom_lib folder and I have provided that path in the nifi properties
>>> file as
>>> >> >>>>>>>
>>> >> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> Not sure what I missed?
>>> >> >>>>>>>
>>> >> >>>>>>> Do I need to make entry of this custom processor somewhere?
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <
>>> bimal007@gmail.com> wrote:
>>> >> >>>>>>>>
>>> >> >>>>>>>> Thanks Mike and Bryan.
>>> >> >>>>>>>> Yes it seems my template was still referring the old version.
>>> >> >>>>>>>> I will have it updated now and will reimport.
>>> >> >>>>>>>> Also the version of NiFi we are using is the one that comes
>>> with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out
>>> to Cloudera and see if we can get it upgraded.
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com>
>>> wrote:
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> What is in the template for the bundle coordinates of your
>>> processor?
>>> >> >>>>>>>>> and does that match the coordinates of the NAR that is
>>> deployed?
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> Example:
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> <bundle>
>>> >> >>>>>>>>> <group>org.apache.nifi</group>
>>> >> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>> >> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>>> >> >>>>>>>>> </bundle>
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your
>>> template
>>> >> >>>>>>>>> references 1.0.0, then you'll need to update your template.
>>> >> >>>>>>>>>
>>> >> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
>>> mikerthomsen@gmail.com> wrote:
>>> >> >>>>>>>>> >
>>> >> >>>>>>>>> > If it's happening immediately upon trying to import the
>>> template, I believe that's the error message saying that the 1.9 instance
>>> cannot find the NAR file which provided the processor. Also, if you're
>>> referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the
>>> latter because there are a few critical bugs fixed in 1.9.2.
>>> >> >>>>>>>>> >
>>> >> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <
>>> bimal007@gmail.com> wrote:
>>> >> >>>>>>>>> >>
>>> >> >>>>>>>>> >> Thanks Bryan.
>>> >> >>>>>>>>> >> My custom processors are part of a template. However
>>> when I try to import my template in NiFi 1.9, I get an error message saying
>>> >> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I
>>> did update all the dependencies to NiFi 1.9 and even the plugins. We are
>>> using a Cloudera distributed version of NiFi 1.9.
>>> >> >>>>>>>>> >> Any idea why is this happening?
>>> >> >>>>>>>>> >>
>>> >> >>>>>>>>> >> Thanks
>>> >> >>>>>>>>> >>
>>> >> >>>>>>>>> >>
>>> >> >>>>>>>>> >>
>>> >> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <
>>> bbende@gmail.com> wrote:
>>> >> >>>>>>>>> >>>
>>> >> >>>>>>>>> >>> Hello,
>>> >> >>>>>>>>> >>>
>>> >> >>>>>>>>> >>> Most likely your processor built against 1.6 would run
>>> fine in 1.9,
>>> >> >>>>>>>>> >>> but to make sure you just need to update any nifi
>>> dependencies in your
>>> >> >>>>>>>>> >>> poms to 1.9.2.
>>> >> >>>>>>>>> >>>
>>> >> >>>>>>>>> >>> If you created your project from the archetype and
>>> didn't change
>>> >> >>>>>>>>> >>> anything, then this should just be changing the parent
>>> in the root pom
>>> >> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
>>> >> >>>>>>>>> >>>
>>> >> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend on
>>> nifi-api you
>>> >> >>>>>>>>> >>> need to change.
>>> >> >>>>>>>>> >>>
>>> >> >>>>>>>>> >>> -Bryan
>>> >> >>>>>>>>> >>>
>>> >> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <
>>> bimal007@gmail.com> wrote:
>>> >> >>>>>>>>> >>> >
>>> >> >>>>>>>>> >>> > Hi,
>>> >> >>>>>>>>> >>> >
>>> >> >>>>>>>>> >>> > If we have a custom processor that was created with
>>> NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>>> >> >>>>>>>>> >>> > Is there some sort of steps that explains the jar and
>>> pom updates we need to do for making it work in 1.9?
>>> >> >>>>>>
>>> >> >>>>>> --
>>> >> >>>>>> Sent from Gmail Mobile
>>> >> >>>>
>>> >> >>>> --
>>> >> >>>> Sent from Gmail Mobile
>>> >> >>
>>> >> >> --
>>> >> >> Sent from Gmail Mobile
>>>
>> --
> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
Where is application-context.xml in your NAR?
And how are you trying to load it in
com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor
?
I would expect it to be packaged into the jar that contains your
processors, most likely in src/main/resources of the processors module
which then ends up at the root of the jar.
On Wed, Aug 14, 2019 at 5:36 PM Bimal Mehta <bi...@gmail.com> wrote:
> Ahh, seems like a Springboot error.
> Is it to do with upgraded Jetty server ?
>
> Caused by: org.springframework.beans.factory.BeanDefinitionStoreException:
> Unexpected exception parsing XML document from class path resource
> [application-context.xml]; nested exception is
> org.springframework.beans.FatalBeanException: Class
> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
> http://www.springframework.org/schema/context] does not implement the
> [org.springframework.beans.factory.xml.NamespaceHandler] interface
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
> at
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
> at
> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
> at
> org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
> at
> org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
> at
> org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
> at
> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
> at
> org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
> at
> com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
> at
> org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
> at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
> ... 50 common frames omitted
> Caused by: org.springframework.beans.FatalBeanException: Class
> [org.springframework.context.config.ContextNamespaceHandler] for namespace [
> http://www.springframework.org/schema/context] does not implement the
> [org.springframework.beans.factory.xml.NamespaceHandler] interface
> at
> org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
> at
> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
> at
> org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
> at
> org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
> at
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
> ... 66 common frames omitted
>
> On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende <bb...@gmail.com> wrote:
>
>> You have to add another instance of the processor which should
>> generate the same stracktrace you sent earlier, except this time there
>> should be a second part to it with "Caused by.... " and then more of
>> the stacktrace that wasn't there before.
>>
>> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >
>> > Hi Bryan,
>> >
>> > I did what you said.
>> > This is what I got
>> >
>> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
>> o.a.n.controller.AbstractComponentNode Computed validation errors with
>> Validation Context StandardValidationContext[componentId=
>> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
>> properties={PropertyDescriptor[Header Line Count]=1,
>> PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
>> Processor' validated against 'Any Property' is invalid because Processor is
>> of type org.apache.nifi.init.InitiateScan, but this is not a valid
>> Processor type]
>> > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
>> o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
>> >
>> > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com> wrote:
>> >>
>> >> Can you edit logback.xml and add the following, the get the stacktrace
>> again?
>> >>
>> >> <logger name="org.apache.nifi.controller" level="DEBUG" />
>> >>
>> >> This should include a root cause exception which we are missing right
>> now.
>> >>
>> >> I think it takes about 20-30 seconds for logback to pick up the edits
>> >> to logback.xml.
>> >>
>> >> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >> >
>> >> > For the custom processor we have, we are extending the
>> AbstractNifiProcessor.java.
>> >> > The processor is used to scan metadata of an incoming flow file.
>> >> > The error we get in logs is as below.
>> >> > 2019-08-13 23:21:21,529 ERROR [main]
>> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
>> org.apache.nifi.init.InitiateScan for ID
>> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
>> >> >
>> org.apache.nifi.controller.exception.ProcessorInstantiationException:
>> org.apache.nifi.init.InitiateScan
>> >> > at
>> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
>> >> > at
>> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
>> >> > at
>> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
>> >> > at
>> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> >> > at
>> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
>> >> > at
>> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
>> >> > at
>> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
>> >> > at
>> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
>> >> > at
>> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
>> >> > at
>> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
>> >> > at
>> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
>> >> > at
>> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
>> >> > at
>> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
>> >> > at
>> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
>> >> > at
>> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
>> >> > at
>> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >> > at
>> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >> > at
>> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> >> > at org.eclipse.jetty.server.Server.start(Server.java:419)
>> >> > at
>> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>> >> > at
>> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> >> > at org.eclipse.jetty.server.Server.doStart(Server.java:386)
>> >> > at
>> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> >> > at
>> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
>> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
>> >> > at org.apache.nifi.NiFi.main(NiFi.java:297)
>> >> >
>> >> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com>
>> wrote:
>> >> >>
>> >> >> Without access to the code for your NAR I can only really guess,
>> but it sounds like an exception is happening when trying to call the
>> constructor of your processor and then it bounces into creating a ghost
>> processor.
>> >> >>
>> >> >> What is in the logs at the time you get the ghost processor?
>> >> >>
>> >> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >> >>>
>> >> >>> Does that mean I need to recreate the processor? Or there is some
>> workaround?
>> >> >>>
>> >> >>> The processor gets unpacked and its bundled dependencies go in
>> NAR_INF.
>> >> >>> However when I drag the processor on the canvas, it comes with a
>> yellow triangle (and gives the error message I stated above) and properties
>> are missing as well.
>> >> >>>
>> >> >>>
>> >> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com>
>> wrote:
>> >> >>>>
>> >> >>>> I don’t remember all the reasoning behind the change, but it had
>> to do with an issue when we upgraded Jetty...
>> >> >>>>
>> >> >>>> https://issues.apache.org/jira/browse/NIFI-5479
>> >> >>>>
>> >> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >> >>>>>
>> >> >>>>> Yes it does show as an option.
>> >> >>>>> One thing I noticed is that the when the nar is unpacked, the
>> bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0,
>> however in NiFI 1.9.0 they go inside NAR_INF.
>> >> >>>>> Why does this happen?
>> >> >>>>> It seems the custom processor that we have uses Springboot, and
>> references applicationcontext file which was inside META_INF when it was
>> built. However I cant see that file anymore in the unpacked nar.
>> >> >>>>>
>> >> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com>
>> wrote:
>> >> >>>>>>
>> >> >>>>>> Does that custom processor type show as an option if you try to
>> add a new processor to the canvas?
>> >> >>>>>>
>> >> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >> >>>>>>>
>> >> >>>>>>> Hi Mike and Bryan,
>> >> >>>>>>>
>> >> >>>>>>> One of my custom processors appears as inactive in NiFi with a
>> yellow triangle error.
>> >> >>>>>>> When I hover over it I see a message saying 'Missing
>> Processor' validated against 'Any Property' is invalid. This is not a valid
>> processor.
>> >> >>>>>>> In the log it seems to invoke GhostProcessor.java which is
>> giving the above error when restarting nifi.
>> >> >>>>>>> This custom processor sits (with my other processors) in my
>> custom_lib folder and I have provided that path in the nifi properties
>> file as
>> >> >>>>>>>
>> >> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> Not sure what I missed?
>> >> >>>>>>>
>> >> >>>>>>> Do I need to make entry of this custom processor somewhere?
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>>
>> >> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >> >>>>>>>>
>> >> >>>>>>>> Thanks Mike and Bryan.
>> >> >>>>>>>> Yes it seems my template was still referring the old version.
>> >> >>>>>>>> I will have it updated now and will reimport.
>> >> >>>>>>>> Also the version of NiFi we are using is the one that comes
>> with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out
>> to Cloudera and see if we can get it upgraded.
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>>
>> >> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com>
>> wrote:
>> >> >>>>>>>>>
>> >> >>>>>>>>> What is in the template for the bundle coordinates of your
>> processor?
>> >> >>>>>>>>> and does that match the coordinates of the NAR that is
>> deployed?
>> >> >>>>>>>>>
>> >> >>>>>>>>> Example:
>> >> >>>>>>>>>
>> >> >>>>>>>>> <bundle>
>> >> >>>>>>>>> <group>org.apache.nifi</group>
>> >> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>> >> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>> >> >>>>>>>>> </bundle>
>> >> >>>>>>>>>
>> >> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your
>> template
>> >> >>>>>>>>> references 1.0.0, then you'll need to update your template.
>> >> >>>>>>>>>
>> >> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
>> mikerthomsen@gmail.com> wrote:
>> >> >>>>>>>>> >
>> >> >>>>>>>>> > If it's happening immediately upon trying to import the
>> template, I believe that's the error message saying that the 1.9 instance
>> cannot find the NAR file which provided the processor. Also, if you're
>> referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the
>> latter because there are a few critical bugs fixed in 1.9.2.
>> >> >>>>>>>>> >
>> >> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >> >>>>>>>>> >>
>> >> >>>>>>>>> >> Thanks Bryan.
>> >> >>>>>>>>> >> My custom processors are part of a template. However when
>> I try to import my template in NiFi 1.9, I get an error message saying
>> >> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did
>> update all the dependencies to NiFi 1.9 and even the plugins. We are using
>> a Cloudera distributed version of NiFi 1.9.
>> >> >>>>>>>>> >> Any idea why is this happening?
>> >> >>>>>>>>> >>
>> >> >>>>>>>>> >> Thanks
>> >> >>>>>>>>> >>
>> >> >>>>>>>>> >>
>> >> >>>>>>>>> >>
>> >> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <
>> bbende@gmail.com> wrote:
>> >> >>>>>>>>> >>>
>> >> >>>>>>>>> >>> Hello,
>> >> >>>>>>>>> >>>
>> >> >>>>>>>>> >>> Most likely your processor built against 1.6 would run
>> fine in 1.9,
>> >> >>>>>>>>> >>> but to make sure you just need to update any nifi
>> dependencies in your
>> >> >>>>>>>>> >>> poms to 1.9.2.
>> >> >>>>>>>>> >>>
>> >> >>>>>>>>> >>> If you created your project from the archetype and
>> didn't change
>> >> >>>>>>>>> >>> anything, then this should just be changing the parent
>> in the root pom
>> >> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
>> >> >>>>>>>>> >>>
>> >> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend on
>> nifi-api you
>> >> >>>>>>>>> >>> need to change.
>> >> >>>>>>>>> >>>
>> >> >>>>>>>>> >>> -Bryan
>> >> >>>>>>>>> >>>
>> >> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <
>> bimal007@gmail.com> wrote:
>> >> >>>>>>>>> >>> >
>> >> >>>>>>>>> >>> > Hi,
>> >> >>>>>>>>> >>> >
>> >> >>>>>>>>> >>> > If we have a custom processor that was created with
>> NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>> >> >>>>>>>>> >>> > Is there some sort of steps that explains the jar and
>> pom updates we need to do for making it work in 1.9?
>> >> >>>>>>
>> >> >>>>>> --
>> >> >>>>>> Sent from Gmail Mobile
>> >> >>>>
>> >> >>>> --
>> >> >>>> Sent from Gmail Mobile
>> >> >>
>> >> >> --
>> >> >> Sent from Gmail Mobile
>>
> --
Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Ahh, seems like a Springboot error.
Is it to do with upgraded Jetty server ?
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException:
Unexpected exception parsing XML document from class path resource
[application-context.xml]; nested exception is
org.springframework.beans.FatalBeanException: Class
[org.springframework.context.config.ContextNamespaceHandler] for namespace [
http://www.springframework.org/schema/context] does not implement the
[org.springframework.beans.factory.xml.NamespaceHandler] interface
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at
org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at
org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at
org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at
org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at
org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:609)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:510)
at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at
com.thinkbiganalytics.nifi.processor.AbstractNiFiProcessor.init(AbstractNiFiProcessor.java:48)
at
org.apache.nifi.processor.AbstractSessionFactoryProcessor.initialize(AbstractSessionFactoryProcessor.java:63)
at
org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:421)
... 50 common frames omitted
Caused by: org.springframework.beans.FatalBeanException: Class
[org.springframework.context.config.ContextNamespaceHandler] for namespace [
http://www.springframework.org/schema/context] does not implement the
[org.springframework.beans.factory.xml.NamespaceHandler] interface
at
org.springframework.beans.factory.xml.DefaultNamespaceHandlerResolver.resolve(DefaultNamespaceHandlerResolver.java:128)
at
org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1406)
at
org.springframework.beans.factory.xml.BeanDefinitionParserDelegate.parseCustomElement(BeanDefinitionParserDelegate.java:1401)
at
org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.parseBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:168)
at
org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.doRegisterBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:138)
at
org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.registerBeanDefinitions(DefaultBeanDefinitionDocumentReader.java:94)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.registerBeanDefinitions(XmlBeanDefinitionReader.java:508)
at
org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:392)
... 66 common frames omitted
On Wed, Aug 14, 2019 at 4:44 PM Bryan Bende <bb...@gmail.com> wrote:
> You have to add another instance of the processor which should
> generate the same stracktrace you sent earlier, except this time there
> should be a second part to it with "Caused by.... " and then more of
> the stacktrace that wasn't there before.
>
> On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com> wrote:
> >
> > Hi Bryan,
> >
> > I did what you said.
> > This is what I got
> >
> > 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
> o.a.n.controller.AbstractComponentNode Computed validation errors with
> Validation Context StandardValidationContext[componentId=
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
> properties={PropertyDescriptor[Header Line Count]=1,
> PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
> Processor' validated against 'Any Property' is invalid because Processor is
> of type org.apache.nifi.init.InitiateScan, but this is not a valid
> Processor type]
> > 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
> o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
> >
> > On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com> wrote:
> >>
> >> Can you edit logback.xml and add the following, the get the stacktrace
> again?
> >>
> >> <logger name="org.apache.nifi.controller" level="DEBUG" />
> >>
> >> This should include a root cause exception which we are missing right
> now.
> >>
> >> I think it takes about 20-30 seconds for logback to pick up the edits
> >> to logback.xml.
> >>
> >> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >> >
> >> > For the custom processor we have, we are extending the
> AbstractNifiProcessor.java.
> >> > The processor is used to scan metadata of an incoming flow file.
> >> > The error we get in logs is as below.
> >> > 2019-08-13 23:21:21,529 ERROR [main]
> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
> org.apache.nifi.init.InitiateScan for ID
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
> >> >
> org.apache.nifi.controller.exception.ProcessorInstantiationException:
> org.apache.nifi.init.InitiateScan
> >> > at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
> >> > at
> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
> >> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
> >> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> >> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
> >> > at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
> >> > at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
> >> > at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
> >> > at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
> >> > at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> >> > at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
> >> > at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
> >> > at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
> >> > at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
> >> > at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
> >> > at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
> >> > at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
> >> > at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
> >> > at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >> > at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> >> > at org.eclipse.jetty.server.Server.start(Server.java:419)
> >> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> >> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> >> > at org.eclipse.jetty.server.Server.doStart(Server.java:386)
> >> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> >> > at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
> >> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
> >> > at org.apache.nifi.NiFi.main(NiFi.java:297)
> >> >
> >> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com> wrote:
> >> >>
> >> >> Without access to the code for your NAR I can only really guess, but
> it sounds like an exception is happening when trying to call the
> constructor of your processor and then it bounces into creating a ghost
> processor.
> >> >>
> >> >> What is in the logs at the time you get the ghost processor?
> >> >>
> >> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >> >>>
> >> >>> Does that mean I need to recreate the processor? Or there is some
> workaround?
> >> >>>
> >> >>> The processor gets unpacked and its bundled dependencies go in
> NAR_INF.
> >> >>> However when I drag the processor on the canvas, it comes with a
> yellow triangle (and gives the error message I stated above) and properties
> are missing as well.
> >> >>>
> >> >>>
> >> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com>
> wrote:
> >> >>>>
> >> >>>> I don’t remember all the reasoning behind the change, but it had
> to do with an issue when we upgraded Jetty...
> >> >>>>
> >> >>>> https://issues.apache.org/jira/browse/NIFI-5479
> >> >>>>
> >> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >> >>>>>
> >> >>>>> Yes it does show as an option.
> >> >>>>> One thing I noticed is that the when the nar is unpacked, the
> bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0,
> however in NiFI 1.9.0 they go inside NAR_INF.
> >> >>>>> Why does this happen?
> >> >>>>> It seems the custom processor that we have uses Springboot, and
> references applicationcontext file which was inside META_INF when it was
> built. However I cant see that file anymore in the unpacked nar.
> >> >>>>>
> >> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com>
> wrote:
> >> >>>>>>
> >> >>>>>> Does that custom processor type show as an option if you try to
> add a new processor to the canvas?
> >> >>>>>>
> >> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >> >>>>>>>
> >> >>>>>>> Hi Mike and Bryan,
> >> >>>>>>>
> >> >>>>>>> One of my custom processors appears as inactive in NiFi with a
> yellow triangle error.
> >> >>>>>>> When I hover over it I see a message saying 'Missing Processor'
> validated against 'Any Property' is invalid. This is not a valid processor.
> >> >>>>>>> In the log it seems to invoke GhostProcessor.java which is
> giving the above error when restarting nifi.
> >> >>>>>>> This custom processor sits (with my other processors) in my
> custom_lib folder and I have provided that path in the nifi properties
> file as
> >> >>>>>>>
> >> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> Not sure what I missed?
> >> >>>>>>>
> >> >>>>>>> Do I need to make entry of this custom processor somewhere?
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com>
> wrote:
> >> >>>>>>>>
> >> >>>>>>>> Thanks Mike and Bryan.
> >> >>>>>>>> Yes it seems my template was still referring the old version.
> >> >>>>>>>> I will have it updated now and will reimport.
> >> >>>>>>>> Also the version of NiFi we are using is the one that comes
> with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out
> to Cloudera and see if we can get it upgraded.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com>
> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>> What is in the template for the bundle coordinates of your
> processor?
> >> >>>>>>>>> and does that match the coordinates of the NAR that is
> deployed?
> >> >>>>>>>>>
> >> >>>>>>>>> Example:
> >> >>>>>>>>>
> >> >>>>>>>>> <bundle>
> >> >>>>>>>>> <group>org.apache.nifi</group>
> >> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
> >> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
> >> >>>>>>>>> </bundle>
> >> >>>>>>>>>
> >> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your
> template
> >> >>>>>>>>> references 1.0.0, then you'll need to update your template.
> >> >>>>>>>>>
> >> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
> mikerthomsen@gmail.com> wrote:
> >> >>>>>>>>> >
> >> >>>>>>>>> > If it's happening immediately upon trying to import the
> template, I believe that's the error message saying that the 1.9 instance
> cannot find the NAR file which provided the processor. Also, if you're
> referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the
> latter because there are a few critical bugs fixed in 1.9.2.
> >> >>>>>>>>> >
> >> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >> >>>>>>>>> >>
> >> >>>>>>>>> >> Thanks Bryan.
> >> >>>>>>>>> >> My custom processors are part of a template. However when
> I try to import my template in NiFi 1.9, I get an error message saying
> >> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did
> update all the dependencies to NiFi 1.9 and even the plugins. We are using
> a Cloudera distributed version of NiFi 1.9.
> >> >>>>>>>>> >> Any idea why is this happening?
> >> >>>>>>>>> >>
> >> >>>>>>>>> >> Thanks
> >> >>>>>>>>> >>
> >> >>>>>>>>> >>
> >> >>>>>>>>> >>
> >> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <
> bbende@gmail.com> wrote:
> >> >>>>>>>>> >>>
> >> >>>>>>>>> >>> Hello,
> >> >>>>>>>>> >>>
> >> >>>>>>>>> >>> Most likely your processor built against 1.6 would run
> fine in 1.9,
> >> >>>>>>>>> >>> but to make sure you just need to update any nifi
> dependencies in your
> >> >>>>>>>>> >>> poms to 1.9.2.
> >> >>>>>>>>> >>>
> >> >>>>>>>>> >>> If you created your project from the archetype and didn't
> change
> >> >>>>>>>>> >>> anything, then this should just be changing the parent in
> the root pom
> >> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
> >> >>>>>>>>> >>>
> >> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend on
> nifi-api you
> >> >>>>>>>>> >>> need to change.
> >> >>>>>>>>> >>>
> >> >>>>>>>>> >>> -Bryan
> >> >>>>>>>>> >>>
> >> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >> >>>>>>>>> >>> >
> >> >>>>>>>>> >>> > Hi,
> >> >>>>>>>>> >>> >
> >> >>>>>>>>> >>> > If we have a custom processor that was created with
> NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
> >> >>>>>>>>> >>> > Is there some sort of steps that explains the jar and
> pom updates we need to do for making it work in 1.9?
> >> >>>>>>
> >> >>>>>> --
> >> >>>>>> Sent from Gmail Mobile
> >> >>>>
> >> >>>> --
> >> >>>> Sent from Gmail Mobile
> >> >>
> >> >> --
> >> >> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
You have to add another instance of the processor which should
generate the same stracktrace you sent earlier, except this time there
should be a second part to it with "Caused by.... " and then more of
the stacktrace that wasn't there before.
On Wed, Aug 14, 2019 at 4:41 PM Bimal Mehta <bi...@gmail.com> wrote:
>
> Hi Bryan,
>
> I did what you said.
> This is what I got
>
> 2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3] o.a.n.controller.AbstractComponentNode Computed validation errors with Validation Context StandardValidationContext[componentId= 6fbe2407-7799-3908-f4c4-bf2f8940bf1e , properties={PropertyDescriptor[Header Line Count]=1, PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing Processor' validated against 'Any Property' is invalid because Processor is of type org.apache.nifi.init.InitiateScan, but this is not a valid Processor type]
> 2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6] o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
>
> On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com> wrote:
>>
>> Can you edit logback.xml and add the following, the get the stacktrace again?
>>
>> <logger name="org.apache.nifi.controller" level="DEBUG" />
>>
>> This should include a root cause exception which we are missing right now.
>>
>> I think it takes about 20-30 seconds for logback to pick up the edits
>> to logback.xml.
>>
>> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >
>> > For the custom processor we have, we are extending the AbstractNifiProcessor.java.
>> > The processor is used to scan metadata of an incoming flow file.
>> > The error we get in logs is as below.
>> > 2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder Could not create Processor of type org.apache.nifi.init.InitiateScan for ID 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
>> > org.apache.nifi.controller.exception.ProcessorInstantiationException: org.apache.nifi.init.InitiateScan
>> > at org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
>> > at org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
>> > at org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
>> > at org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
>> > at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
>> > at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> > at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
>> > at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
>> > at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
>> > at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
>> > at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
>> > at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
>> > at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
>> > at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
>> > at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
>> > at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
>> > at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
>> > at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
>> > at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
>> > at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
>> > at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
>> > at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
>> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> > at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
>> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> > at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
>> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
>> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
>> > at org.eclipse.jetty.server.Server.start(Server.java:419)
>> > at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
>> > at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
>> > at org.eclipse.jetty.server.Server.doStart(Server.java:386)
>> > at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>> > at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
>> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
>> > at org.apache.nifi.NiFi.main(NiFi.java:297)
>> >
>> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com> wrote:
>> >>
>> >> Without access to the code for your NAR I can only really guess, but it sounds like an exception is happening when trying to call the constructor of your processor and then it bounces into creating a ghost processor.
>> >>
>> >> What is in the logs at the time you get the ghost processor?
>> >>
>> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >>>
>> >>> Does that mean I need to recreate the processor? Or there is some workaround?
>> >>>
>> >>> The processor gets unpacked and its bundled dependencies go in NAR_INF.
>> >>> However when I drag the processor on the canvas, it comes with a yellow triangle (and gives the error message I stated above) and properties are missing as well.
>> >>>
>> >>>
>> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
>> >>>>
>> >>>> I don’t remember all the reasoning behind the change, but it had to do with an issue when we upgraded Jetty...
>> >>>>
>> >>>> https://issues.apache.org/jira/browse/NIFI-5479
>> >>>>
>> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >>>>>
>> >>>>> Yes it does show as an option.
>> >>>>> One thing I noticed is that the when the nar is unpacked, the bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0, however in NiFI 1.9.0 they go inside NAR_INF.
>> >>>>> Why does this happen?
>> >>>>> It seems the custom processor that we have uses Springboot, and references applicationcontext file which was inside META_INF when it was built. However I cant see that file anymore in the unpacked nar.
>> >>>>>
>> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Does that custom processor type show as an option if you try to add a new processor to the canvas?
>> >>>>>>
>> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>> Hi Mike and Bryan,
>> >>>>>>>
>> >>>>>>> One of my custom processors appears as inactive in NiFi with a yellow triangle error.
>> >>>>>>> When I hover over it I see a message saying 'Missing Processor' validated against 'Any Property' is invalid. This is not a valid processor.
>> >>>>>>> In the log it seems to invoke GhostProcessor.java which is giving the above error when restarting nifi.
>> >>>>>>> This custom processor sits (with my other processors) in my custom_lib folder and I have provided that path in the nifi properties file as
>> >>>>>>>
>> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Not sure what I missed?
>> >>>>>>>
>> >>>>>>> Do I need to make entry of this custom processor somewhere?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> Thanks Mike and Bryan.
>> >>>>>>>> Yes it seems my template was still referring the old version.
>> >>>>>>>> I will have it updated now and will reimport.
>> >>>>>>>> Also the version of NiFi we are using is the one that comes with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera and see if we can get it upgraded.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>> What is in the template for the bundle coordinates of your processor?
>> >>>>>>>>> and does that match the coordinates of the NAR that is deployed?
>> >>>>>>>>>
>> >>>>>>>>> Example:
>> >>>>>>>>>
>> >>>>>>>>> <bundle>
>> >>>>>>>>> <group>org.apache.nifi</group>
>> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>> >>>>>>>>> </bundle>
>> >>>>>>>>>
>> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>> >>>>>>>>> references 1.0.0, then you'll need to update your template.
>> >>>>>>>>>
>> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com> wrote:
>> >>>>>>>>> >
>> >>>>>>>>> > If it's happening immediately upon trying to import the template, I believe that's the error message saying that the 1.9 instance cannot find the NAR file which provided the processor. Also, if you're referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because there are a few critical bugs fixed in 1.9.2.
>> >>>>>>>>> >
>> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >>>>>>>>> >>
>> >>>>>>>>> >> Thanks Bryan.
>> >>>>>>>>> >> My custom processors are part of a template. However when I try to import my template in NiFi 1.9, I get an error message saying
>> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update all the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera distributed version of NiFi 1.9.
>> >>>>>>>>> >> Any idea why is this happening?
>> >>>>>>>>> >>
>> >>>>>>>>> >> Thanks
>> >>>>>>>>> >>
>> >>>>>>>>> >>
>> >>>>>>>>> >>
>> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> Hello,
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>> >>>>>>>>> >>> but to make sure you just need to update any nifi dependencies in your
>> >>>>>>>>> >>> poms to 1.9.2.
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> If you created your project from the archetype and didn't change
>> >>>>>>>>> >>> anything, then this should just be changing the parent in the root pom
>> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>> >>>>>>>>> >>> need to change.
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> -Bryan
>> >>>>>>>>> >>>
>> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >>>>>>>>> >>> >
>> >>>>>>>>> >>> > Hi,
>> >>>>>>>>> >>> >
>> >>>>>>>>> >>> > If we have a custom processor that was created with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>> >>>>>>>>> >>> > Is there some sort of steps that explains the jar and pom updates we need to do for making it work in 1.9?
>> >>>>>>
>> >>>>>> --
>> >>>>>> Sent from Gmail Mobile
>> >>>>
>> >>>> --
>> >>>> Sent from Gmail Mobile
>> >>
>> >> --
>> >> Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Hi Bryan,
I did what you said.
This is what I got
2019-08-14 20:16:18,948 DEBUG [Validate Components Thread-3]
o.a.n.controller.AbstractComponentNode Computed validation errors with
Validation Context StandardValidationContext[componentId=
6fbe2407-7799-3908-f4c4-bf2f8940bf1e ,
properties={PropertyDescriptor[Header Line Count]=1,
PropertyDescriptor[Enable processing]=${searchTerm}}]; results = ['Missing
Processor' validated against 'Any Property' is invalid because Processor is
of type org.apache.nifi.init.InitiateScan, but this is not a valid
Processor type]
2019-08-14 20:16:18,946 DEBUG [Timer-Driven Process Thread-6]
o.a.n.c.r.m.SecondPrecisionEventContainer Updated bin 39. Did NOT replace.
On Wed, Aug 14, 2019 at 1:42 PM Bryan Bende <bb...@gmail.com> wrote:
> Can you edit logback.xml and add the following, the get the stacktrace
> again?
>
> <logger name="org.apache.nifi.controller" level="DEBUG" />
>
> This should include a root cause exception which we are missing right now.
>
> I think it takes about 20-30 seconds for logback to pick up the edits
> to logback.xml.
>
> On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com> wrote:
> >
> > For the custom processor we have, we are extending the
> AbstractNifiProcessor.java.
> > The processor is used to scan metadata of an incoming flow file.
> > The error we get in logs is as below.
> > 2019-08-13 23:21:21,529 ERROR [main]
> o.a.nifi.controller.ExtensionBuilder Could not create Processor of type
> org.apache.nifi.init.InitiateScan for ID
> 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
> > org.apache.nifi.controller.exception.ProcessorInstantiationException:
> org.apache.nifi.init.InitiateScan
> > at
> org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
> > at
> org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
> > at
> org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> > at
> org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
> > at
> org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
> > at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
> > at
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
> > at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
> > at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
> > at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
> > at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
> > at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
> > at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
> > at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
> > at
> org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> > at org.eclipse.jetty.server.Server.start(Server.java:419)
> > at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> > at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> > at org.eclipse.jetty.server.Server.doStart(Server.java:386)
> > at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> > at
> org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
> > at org.apache.nifi.NiFi.<init>(NiFi.java:158)
> > at org.apache.nifi.NiFi.<init>(NiFi.java:72)
> > at org.apache.nifi.NiFi.main(NiFi.java:297)
> >
> > On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com> wrote:
> >>
> >> Without access to the code for your NAR I can only really guess, but it
> sounds like an exception is happening when trying to call the constructor
> of your processor and then it bounces into creating a ghost processor.
> >>
> >> What is in the logs at the time you get the ghost processor?
> >>
> >> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>
> >>> Does that mean I need to recreate the processor? Or there is some
> workaround?
> >>>
> >>> The processor gets unpacked and its bundled dependencies go in NAR_INF.
> >>> However when I drag the processor on the canvas, it comes with a
> yellow triangle (and gives the error message I stated above) and properties
> are missing as well.
> >>>
> >>>
> >>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
> >>>>
> >>>> I don’t remember all the reasoning behind the change, but it had to
> do with an issue when we upgraded Jetty...
> >>>>
> >>>> https://issues.apache.org/jira/browse/NIFI-5479
> >>>>
> >>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>>>
> >>>>> Yes it does show as an option.
> >>>>> One thing I noticed is that the when the nar is unpacked, the
> bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0,
> however in NiFI 1.9.0 they go inside NAR_INF.
> >>>>> Why does this happen?
> >>>>> It seems the custom processor that we have uses Springboot, and
> references applicationcontext file which was inside META_INF when it was
> built. However I cant see that file anymore in the unpacked nar.
> >>>>>
> >>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> Does that custom processor type show as an option if you try to add
> a new processor to the canvas?
> >>>>>>
> >>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> Hi Mike and Bryan,
> >>>>>>>
> >>>>>>> One of my custom processors appears as inactive in NiFi with a
> yellow triangle error.
> >>>>>>> When I hover over it I see a message saying 'Missing Processor'
> validated against 'Any Property' is invalid. This is not a valid processor.
> >>>>>>> In the log it seems to invoke GhostProcessor.java which is giving
> the above error when restarting nifi.
> >>>>>>> This custom processor sits (with my other processors) in my
> custom_lib folder and I have provided that path in the nifi properties
> file as
> >>>>>>>
> >>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
> >>>>>>>
> >>>>>>>
> >>>>>>> Not sure what I missed?
> >>>>>>>
> >>>>>>> Do I need to make entry of this custom processor somewhere?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Thanks Mike and Bryan.
> >>>>>>>> Yes it seems my template was still referring the old version.
> >>>>>>>> I will have it updated now and will reimport.
> >>>>>>>> Also the version of NiFi we are using is the one that comes with
> CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out to
> Cloudera and see if we can get it upgraded.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com>
> wrote:
> >>>>>>>>>
> >>>>>>>>> What is in the template for the bundle coordinates of your
> processor?
> >>>>>>>>> and does that match the coordinates of the NAR that is deployed?
> >>>>>>>>>
> >>>>>>>>> Example:
> >>>>>>>>>
> >>>>>>>>> <bundle>
> >>>>>>>>> <group>org.apache.nifi</group>
> >>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
> >>>>>>>>> <version>1.10.0-SNAPSHOT</version>
> >>>>>>>>> </bundle>
> >>>>>>>>>
> >>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your
> template
> >>>>>>>>> references 1.0.0, then you'll need to update your template.
> >>>>>>>>>
> >>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
> mikerthomsen@gmail.com> wrote:
> >>>>>>>>> >
> >>>>>>>>> > If it's happening immediately upon trying to import the
> template, I believe that's the error message saying that the 1.9 instance
> cannot find the NAR file which provided the processor. Also, if you're
> referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the
> latter because there are a few critical bugs fixed in 1.9.2.
> >>>>>>>>> >
> >>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com>
> wrote:
> >>>>>>>>> >>
> >>>>>>>>> >> Thanks Bryan.
> >>>>>>>>> >> My custom processors are part of a template. However when I
> try to import my template in NiFi 1.9, I get an error message saying
> >>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did
> update all the dependencies to NiFi 1.9 and even the plugins. We are using
> a Cloudera distributed version of NiFi 1.9.
> >>>>>>>>> >> Any idea why is this happening?
> >>>>>>>>> >>
> >>>>>>>>> >> Thanks
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >>
> >>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com>
> wrote:
> >>>>>>>>> >>>
> >>>>>>>>> >>> Hello,
> >>>>>>>>> >>>
> >>>>>>>>> >>> Most likely your processor built against 1.6 would run fine
> in 1.9,
> >>>>>>>>> >>> but to make sure you just need to update any nifi
> dependencies in your
> >>>>>>>>> >>> poms to 1.9.2.
> >>>>>>>>> >>>
> >>>>>>>>> >>> If you created your project from the archetype and didn't
> change
> >>>>>>>>> >>> anything, then this should just be changing the parent in
> the root pom
> >>>>>>>>> >>> to the new version of nifi-nar-bundles.
> >>>>>>>>> >>>
> >>>>>>>>> >>> If you set it up yourself, then anywhere you depend on
> nifi-api you
> >>>>>>>>> >>> need to change.
> >>>>>>>>> >>>
> >>>>>>>>> >>> -Bryan
> >>>>>>>>> >>>
> >>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <
> bimal007@gmail.com> wrote:
> >>>>>>>>> >>> >
> >>>>>>>>> >>> > Hi,
> >>>>>>>>> >>> >
> >>>>>>>>> >>> > If we have a custom processor that was created with NiFi
> 1.6, what are the steps we need to follow to make it work in 1.9?
> >>>>>>>>> >>> > Is there some sort of steps that explains the jar and pom
> updates we need to do for making it work in 1.9?
> >>>>>>
> >>>>>> --
> >>>>>> Sent from Gmail Mobile
> >>>>
> >>>> --
> >>>> Sent from Gmail Mobile
> >>
> >> --
> >> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
Can you edit logback.xml and add the following, the get the stacktrace again?
<logger name="org.apache.nifi.controller" level="DEBUG" />
This should include a root cause exception which we are missing right now.
I think it takes about 20-30 seconds for logback to pick up the edits
to logback.xml.
On Wed, Aug 14, 2019 at 12:53 PM Bimal Mehta <bi...@gmail.com> wrote:
>
> For the custom processor we have, we are extending the AbstractNifiProcessor.java.
> The processor is used to scan metadata of an incoming flow file.
> The error we get in logs is as below.
> 2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder Could not create Processor of type org.apache.nifi.init.InitiateScan for ID 6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
> org.apache.nifi.controller.exception.ProcessorInstantiationException: org.apache.nifi.init.InitiateScan
> at org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
> at org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
> at org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
> at org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
> at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
> at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
> at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
> at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
> at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
> at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
> at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
> at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
> at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
> at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
> at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
> at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
> at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
> at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
> at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
> at org.eclipse.jetty.server.Server.start(Server.java:419)
> at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
> at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
> at org.eclipse.jetty.server.Server.doStart(Server.java:386)
> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
> at org.apache.nifi.NiFi.<init>(NiFi.java:158)
> at org.apache.nifi.NiFi.<init>(NiFi.java:72)
> at org.apache.nifi.NiFi.main(NiFi.java:297)
>
> On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com> wrote:
>>
>> Without access to the code for your NAR I can only really guess, but it sounds like an exception is happening when trying to call the constructor of your processor and then it bounces into creating a ghost processor.
>>
>> What is in the logs at the time you get the ghost processor?
>>
>> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>
>>> Does that mean I need to recreate the processor? Or there is some workaround?
>>>
>>> The processor gets unpacked and its bundled dependencies go in NAR_INF.
>>> However when I drag the processor on the canvas, it comes with a yellow triangle (and gives the error message I stated above) and properties are missing as well.
>>>
>>>
>>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>> I don’t remember all the reasoning behind the change, but it had to do with an issue when we upgraded Jetty...
>>>>
>>>> https://issues.apache.org/jira/browse/NIFI-5479
>>>>
>>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>>>
>>>>> Yes it does show as an option.
>>>>> One thing I noticed is that the when the nar is unpacked, the bundled dependencies are inside META_INF in the work folder in NiFi 1.6.0, however in NiFI 1.9.0 they go inside NAR_INF.
>>>>> Why does this happen?
>>>>> It seems the custom processor that we have uses Springboot, and references applicationcontext file which was inside META_INF when it was built. However I cant see that file anymore in the unpacked nar.
>>>>>
>>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>> Does that custom processor type show as an option if you try to add a new processor to the canvas?
>>>>>>
>>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Mike and Bryan,
>>>>>>>
>>>>>>> One of my custom processors appears as inactive in NiFi with a yellow triangle error.
>>>>>>> When I hover over it I see a message saying 'Missing Processor' validated against 'Any Property' is invalid. This is not a valid processor.
>>>>>>> In the log it seems to invoke GhostProcessor.java which is giving the above error when restarting nifi.
>>>>>>> This custom processor sits (with my other processors) in my custom_lib folder and I have provided that path in the nifi properties file as
>>>>>>>
>>>>>>> nifi.nar.library.directory.custom=/opt/nifi/custom_lib
>>>>>>>
>>>>>>>
>>>>>>> Not sure what I missed?
>>>>>>>
>>>>>>> Do I need to make entry of this custom processor somewhere?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Thanks Mike and Bryan.
>>>>>>>> Yes it seems my template was still referring the old version.
>>>>>>>> I will have it updated now and will reimport.
>>>>>>>> Also the version of NiFi we are using is the one that comes with CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera and see if we can get it upgraded.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> What is in the template for the bundle coordinates of your processor?
>>>>>>>>> and does that match the coordinates of the NAR that is deployed?
>>>>>>>>>
>>>>>>>>> Example:
>>>>>>>>>
>>>>>>>>> <bundle>
>>>>>>>>> <group>org.apache.nifi</group>
>>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>>>>>>>>> </bundle>
>>>>>>>>>
>>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>>>>>>> references 1.0.0, then you'll need to update your template.
>>>>>>>>>
>>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > If it's happening immediately upon trying to import the template, I believe that's the error message saying that the 1.9 instance cannot find the NAR file which provided the processor. Also, if you're referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because there are a few critical bugs fixed in 1.9.2.
>>>>>>>>> >
>>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Thanks Bryan.
>>>>>>>>> >> My custom processors are part of a template. However when I try to import my template in NiFi 1.9, I get an error message saying
>>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update all the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera distributed version of NiFi 1.9.
>>>>>>>>> >> Any idea why is this happening?
>>>>>>>>> >>
>>>>>>>>> >> Thanks
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Hello,
>>>>>>>>> >>>
>>>>>>>>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>>>>>>>>> >>> but to make sure you just need to update any nifi dependencies in your
>>>>>>>>> >>> poms to 1.9.2.
>>>>>>>>> >>>
>>>>>>>>> >>> If you created your project from the archetype and didn't change
>>>>>>>>> >>> anything, then this should just be changing the parent in the root pom
>>>>>>>>> >>> to the new version of nifi-nar-bundles.
>>>>>>>>> >>>
>>>>>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>>>>>>>>> >>> need to change.
>>>>>>>>> >>>
>>>>>>>>> >>> -Bryan
>>>>>>>>> >>>
>>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>>>>>>> >>> >
>>>>>>>>> >>> > Hi,
>>>>>>>>> >>> >
>>>>>>>>> >>> > If we have a custom processor that was created with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>>>>>>>>> >>> > Is there some sort of steps that explains the jar and pom updates we need to do for making it work in 1.9?
>>>>>>
>>>>>> --
>>>>>> Sent from Gmail Mobile
>>>>
>>>> --
>>>> Sent from Gmail Mobile
>>
>> --
>> Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
For the custom processor we have, we are extending the
AbstractNifiProcessor.java.
The processor is used to scan metadata of an incoming flow file.
The error we get in logs is as below.
2019-08-13 23:21:21,529 ERROR [main] o.a.nifi.controller.ExtensionBuilder
Could not create Processor of type org.apache.nifi.init.InitiateScan for ID
6fbe2407-7799-3908-f4c4-bf2f8940bf1e; creating "Ghost" implementation
org.apache.nifi.controller.exception.ProcessorInstantiationException:
org.apache.nifi.init.InitiateScan
at
org.apache.nifi.controller.ExtensionBuilder.createLoggableProcessor(ExtensionBuilder.java:425)
at
org.apache.nifi.controller.ExtensionBuilder.buildProcessor(ExtensionBuilder.java:191)
at
org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:298)
at
org.apache.nifi.controller.flow.StandardFlowManager.createProcessor(StandardFlowManager.java:274)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1262)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
at
org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1389)
at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:362)
at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1296)
at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:88)
at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:812)
at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:557)
at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:403)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:167)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:419)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:386)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
On Wed, Aug 14, 2019 at 7:42 AM Bryan Bende <bb...@gmail.com> wrote:
> Without access to the code for your NAR I can only really guess, but it
> sounds like an exception is happening when trying to call the constructor
> of your processor and then it bounces into creating a ghost processor.
>
> What is in the logs at the time you get the ghost processor?
>
> On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>
>> Does that mean I need to recreate the processor? Or there is some
>> workaround?
>>
>> The processor gets unpacked and its bundled dependencies go in NAR_INF.
>> However when I drag the processor on the canvas, it comes with a yellow
>> triangle (and gives the error message I stated above) and properties are
>> missing as well.
>>
>>
>> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
>>
>>> I don’t remember all the reasoning behind the change, but it had to do
>>> with an issue when we upgraded Jetty...
>>>
>>> https://issues.apache.org/jira/browse/NIFI-5479
>>>
>>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>
>>>> Yes it does show as an option.
>>>> One thing I noticed is that the when the nar is unpacked, the bundled
>>>> dependencies are inside META_INF in the work folder in NiFi 1.6.0, however
>>>> in NiFI 1.9.0 they go inside NAR_INF.
>>>> Why does this happen?
>>>> It seems the custom processor that we have uses Springboot, and
>>>> references applicationcontext file which was inside META_INF when it was
>>>> built. However I cant see that file anymore in the unpacked nar.
>>>>
>>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>>> Does that custom processor type show as an option if you try to add a
>>>>> new processor to the canvas?
>>>>>
>>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Mike and Bryan,
>>>>>>
>>>>>> One of my custom processors appears as inactive in NiFi with a yellow
>>>>>> triangle error.
>>>>>> When I hover over it I see a message saying 'Missing Processor'
>>>>>> validated against 'Any Property' is invalid. This is not a valid processor.
>>>>>> In the log it seems to invoke GhostProcessor.java which is giving the
>>>>>> above error when restarting nifi.
>>>>>> This custom processor sits (with my other processors) in my
>>>>>> custom_lib folder and I have provided that path in the nifi properties
>>>>>> file as
>>>>>>
>>>>>> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>>>>>>
>>>>>>
>>>>>> Not sure what I missed?
>>>>>>
>>>>>> Do I need to make entry of this custom processor somewhere?
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Mike and Bryan.
>>>>>>> Yes it seems my template was still referring the old version.
>>>>>>> I will have it updated now and will reimport.
>>>>>>> Also the version of NiFi we are using is the one that comes with
>>>>>>> CDF. I am not sure if CDF supports 1.9.2 yet or not. I will reach out to
>>>>>>> Cloudera and see if we can get it upgraded.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>>>>>
>>>>>>>> What is in the template for the bundle coordinates of your
>>>>>>>> processor?
>>>>>>>> and does that match the coordinates of the NAR that is deployed?
>>>>>>>>
>>>>>>>> Example:
>>>>>>>>
>>>>>>>> <bundle>
>>>>>>>> <group>org.apache.nifi</group>
>>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>>>>>> <version>1.10.0-SNAPSHOT</version>
>>>>>>>> </bundle>
>>>>>>>>
>>>>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>>>>>> references 1.0.0, then you'll need to update your template.
>>>>>>>>
>>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <
>>>>>>>> mikerthomsen@gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > If it's happening immediately upon trying to import the template,
>>>>>>>> I believe that's the error message saying that the 1.9 instance cannot find
>>>>>>>> the NAR file which provided the processor. Also, if you're referring to
>>>>>>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>>>>>>>> there are a few critical bugs fixed in 1.9.2.
>>>>>>>> >
>>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Thanks Bryan.
>>>>>>>> >> My custom processors are part of a template. However when I try
>>>>>>>> to import my template in NiFi 1.9, I get an error message saying
>>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update
>>>>>>>> all the dependencies to NiFi 1.9 and even the plugins. We are using a
>>>>>>>> Cloudera distributed version of NiFi 1.9.
>>>>>>>> >> Any idea why is this happening?
>>>>>>>> >>
>>>>>>>> >> Thanks
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Hello,
>>>>>>>> >>>
>>>>>>>> >>> Most likely your processor built against 1.6 would run fine in
>>>>>>>> 1.9,
>>>>>>>> >>> but to make sure you just need to update any nifi dependencies
>>>>>>>> in your
>>>>>>>> >>> poms to 1.9.2.
>>>>>>>> >>>
>>>>>>>> >>> If you created your project from the archetype and didn't change
>>>>>>>> >>> anything, then this should just be changing the parent in the
>>>>>>>> root pom
>>>>>>>> >>> to the new version of nifi-nar-bundles.
>>>>>>>> >>>
>>>>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api
>>>>>>>> you
>>>>>>>> >>> need to change.
>>>>>>>> >>>
>>>>>>>> >>> -Bryan
>>>>>>>> >>>
>>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>> >
>>>>>>>> >>> > Hi,
>>>>>>>> >>> >
>>>>>>>> >>> > If we have a custom processor that was created with NiFi 1.6,
>>>>>>>> what are the steps we need to follow to make it work in 1.9?
>>>>>>>> >>> > Is there some sort of steps that explains the jar and pom
>>>>>>>> updates we need to do for making it work in 1.9?
>>>>>>>>
>>>>>>> --
>>>>> Sent from Gmail Mobile
>>>>>
>>>> --
>>> Sent from Gmail Mobile
>>>
>> --
> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
Without access to the code for your NAR I can only really guess, but it
sounds like an exception is happening when trying to call the constructor
of your processor and then it bounces into creating a ghost processor.
What is in the logs at the time you get the ghost processor?
On Tue, Aug 13, 2019 at 10:54 PM Bimal Mehta <bi...@gmail.com> wrote:
> Does that mean I need to recreate the processor? Or there is some
> workaround?
>
> The processor gets unpacked and its bundled dependencies go in NAR_INF.
> However when I drag the processor on the canvas, it comes with a yellow
> triangle (and gives the error message I stated above) and properties are
> missing as well.
>
>
> On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
>
>> I don’t remember all the reasoning behind the change, but it had to do
>> with an issue when we upgraded Jetty...
>>
>> https://issues.apache.org/jira/browse/NIFI-5479
>>
>> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
>>
>>> Yes it does show as an option.
>>> One thing I noticed is that the when the nar is unpacked, the bundled
>>> dependencies are inside META_INF in the work folder in NiFi 1.6.0, however
>>> in NiFI 1.9.0 they go inside NAR_INF.
>>> Why does this happen?
>>> It seems the custom processor that we have uses Springboot, and
>>> references applicationcontext file which was inside META_INF when it was
>>> built. However I cant see that file anymore in the unpacked nar.
>>>
>>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>>>
>>>> Does that custom processor type show as an option if you try to add a
>>>> new processor to the canvas?
>>>>
>>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>>
>>>>> Hi Mike and Bryan,
>>>>>
>>>>> One of my custom processors appears as inactive in NiFi with a yellow
>>>>> triangle error.
>>>>> When I hover over it I see a message saying 'Missing Processor'
>>>>> validated against 'Any Property' is invalid. This is not a valid processor.
>>>>> In the log it seems to invoke GhostProcessor.java which is giving the
>>>>> above error when restarting nifi.
>>>>> This custom processor sits (with my other processors) in my
>>>>> custom_lib folder and I have provided that path in the nifi properties
>>>>> file as
>>>>>
>>>>> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>>>>>
>>>>>
>>>>> Not sure what I missed?
>>>>>
>>>>> Do I need to make entry of this custom processor somewhere?
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Mike and Bryan.
>>>>>> Yes it seems my template was still referring the old version.
>>>>>> I will have it updated now and will reimport.
>>>>>> Also the version of NiFi we are using is the one that comes with CDF.
>>>>>> I am not sure if CDF supports 1.9.2 yet or not. I will reach out to
>>>>>> Cloudera and see if we can get it upgraded.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>>>>
>>>>>>> What is in the template for the bundle coordinates of your processor?
>>>>>>> and does that match the coordinates of the NAR that is deployed?
>>>>>>>
>>>>>>> Example:
>>>>>>>
>>>>>>> <bundle>
>>>>>>> <group>org.apache.nifi</group>
>>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>>>>> <version>1.10.0-SNAPSHOT</version>
>>>>>>> </bundle>
>>>>>>>
>>>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>>>>> references 1.0.0, then you'll need to update your template.
>>>>>>>
>>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > If it's happening immediately upon trying to import the template,
>>>>>>> I believe that's the error message saying that the 1.9 instance cannot find
>>>>>>> the NAR file which provided the processor. Also, if you're referring to
>>>>>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>>>>>>> there are a few critical bugs fixed in 1.9.2.
>>>>>>> >
>>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Thanks Bryan.
>>>>>>> >> My custom processors are part of a template. However when I try
>>>>>>> to import my template in NiFi 1.9, I get an error message saying
>>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update
>>>>>>> all the dependencies to NiFi 1.9 and even the plugins. We are using a
>>>>>>> Cloudera distributed version of NiFi 1.9.
>>>>>>> >> Any idea why is this happening?
>>>>>>> >>
>>>>>>> >> Thanks
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Hello,
>>>>>>> >>>
>>>>>>> >>> Most likely your processor built against 1.6 would run fine in
>>>>>>> 1.9,
>>>>>>> >>> but to make sure you just need to update any nifi dependencies
>>>>>>> in your
>>>>>>> >>> poms to 1.9.2.
>>>>>>> >>>
>>>>>>> >>> If you created your project from the archetype and didn't change
>>>>>>> >>> anything, then this should just be changing the parent in the
>>>>>>> root pom
>>>>>>> >>> to the new version of nifi-nar-bundles.
>>>>>>> >>>
>>>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api
>>>>>>> you
>>>>>>> >>> need to change.
>>>>>>> >>>
>>>>>>> >>> -Bryan
>>>>>>> >>>
>>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>>>>>>> wrote:
>>>>>>> >>> >
>>>>>>> >>> > Hi,
>>>>>>> >>> >
>>>>>>> >>> > If we have a custom processor that was created with NiFi 1.6,
>>>>>>> what are the steps we need to follow to make it work in 1.9?
>>>>>>> >>> > Is there some sort of steps that explains the jar and pom
>>>>>>> updates we need to do for making it work in 1.9?
>>>>>>>
>>>>>> --
>>>> Sent from Gmail Mobile
>>>>
>>> --
>> Sent from Gmail Mobile
>>
> --
Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Does that mean I need to recreate the processor? Or there is some
workaround?
The processor gets unpacked and its bundled dependencies go in NAR_INF.
However when I drag the processor on the canvas, it comes with a yellow
triangle (and gives the error message I stated above) and properties are
missing as well.
On Tue, Aug 13, 2019 at 10:47 PM Bryan Bende <bb...@gmail.com> wrote:
> I don’t remember all the reasoning behind the change, but it had to do
> with an issue when we upgraded Jetty...
>
> https://issues.apache.org/jira/browse/NIFI-5479
>
> On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
>
>> Yes it does show as an option.
>> One thing I noticed is that the when the nar is unpacked, the bundled
>> dependencies are inside META_INF in the work folder in NiFi 1.6.0, however
>> in NiFI 1.9.0 they go inside NAR_INF.
>> Why does this happen?
>> It seems the custom processor that we have uses Springboot, and
>> references applicationcontext file which was inside META_INF when it was
>> built. However I cant see that file anymore in the unpacked nar.
>>
>> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>>
>>> Does that custom processor type show as an option if you try to add a
>>> new processor to the canvas?
>>>
>>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>>
>>>> Hi Mike and Bryan,
>>>>
>>>> One of my custom processors appears as inactive in NiFi with a yellow
>>>> triangle error.
>>>> When I hover over it I see a message saying 'Missing Processor'
>>>> validated against 'Any Property' is invalid. This is not a valid processor.
>>>> In the log it seems to invoke GhostProcessor.java which is giving the
>>>> above error when restarting nifi.
>>>> This custom processor sits (with my other processors) in my custom_lib
>>>> folder and I have provided that path in the nifi properties file as
>>>>
>>>> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>>>>
>>>>
>>>> Not sure what I missed?
>>>>
>>>> Do I need to make entry of this custom processor somewhere?
>>>>
>>>>
>>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>>>>
>>>>> Thanks Mike and Bryan.
>>>>> Yes it seems my template was still referring the old version.
>>>>> I will have it updated now and will reimport.
>>>>> Also the version of NiFi we are using is the one that comes with CDF.
>>>>> I am not sure if CDF supports 1.9.2 yet or not. I will reach out to
>>>>> Cloudera and see if we can get it upgraded.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>>>
>>>>>> What is in the template for the bundle coordinates of your processor?
>>>>>> and does that match the coordinates of the NAR that is deployed?
>>>>>>
>>>>>> Example:
>>>>>>
>>>>>> <bundle>
>>>>>> <group>org.apache.nifi</group>
>>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>>>> <version>1.10.0-SNAPSHOT</version>
>>>>>> </bundle>
>>>>>>
>>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>>>> references 1.0.0, then you'll need to update your template.
>>>>>>
>>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > If it's happening immediately upon trying to import the template, I
>>>>>> believe that's the error message saying that the 1.9 instance cannot find
>>>>>> the NAR file which provided the processor. Also, if you're referring to
>>>>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>>>>>> there are a few critical bugs fixed in 1.9.2.
>>>>>> >
>>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Thanks Bryan.
>>>>>> >> My custom processors are part of a template. However when I try to
>>>>>> import my template in NiFi 1.9, I get an error message saying
>>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update
>>>>>> all the dependencies to NiFi 1.9 and even the plugins. We are using a
>>>>>> Cloudera distributed version of NiFi 1.9.
>>>>>> >> Any idea why is this happening?
>>>>>> >>
>>>>>> >> Thanks
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hello,
>>>>>> >>>
>>>>>> >>> Most likely your processor built against 1.6 would run fine in
>>>>>> 1.9,
>>>>>> >>> but to make sure you just need to update any nifi dependencies in
>>>>>> your
>>>>>> >>> poms to 1.9.2.
>>>>>> >>>
>>>>>> >>> If you created your project from the archetype and didn't change
>>>>>> >>> anything, then this should just be changing the parent in the
>>>>>> root pom
>>>>>> >>> to the new version of nifi-nar-bundles.
>>>>>> >>>
>>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api
>>>>>> you
>>>>>> >>> need to change.
>>>>>> >>>
>>>>>> >>> -Bryan
>>>>>> >>>
>>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>>>>>> wrote:
>>>>>> >>> >
>>>>>> >>> > Hi,
>>>>>> >>> >
>>>>>> >>> > If we have a custom processor that was created with NiFi 1.6,
>>>>>> what are the steps we need to follow to make it work in 1.9?
>>>>>> >>> > Is there some sort of steps that explains the jar and pom
>>>>>> updates we need to do for making it work in 1.9?
>>>>>>
>>>>> --
>>> Sent from Gmail Mobile
>>>
>> --
> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
I don’t remember all the reasoning behind the change, but it had to do with
an issue when we upgraded Jetty...
https://issues.apache.org/jira/browse/NIFI-5479
On Tue, Aug 13, 2019 at 9:47 PM Bimal Mehta <bi...@gmail.com> wrote:
> Yes it does show as an option.
> One thing I noticed is that the when the nar is unpacked, the bundled
> dependencies are inside META_INF in the work folder in NiFi 1.6.0, however
> in NiFI 1.9.0 they go inside NAR_INF.
> Why does this happen?
> It seems the custom processor that we have uses Springboot, and references
> applicationcontext file which was inside META_INF when it was built.
> However I cant see that file anymore in the unpacked nar.
>
> On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
>
>> Does that custom processor type show as an option if you try to add a new
>> processor to the canvas?
>>
>> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>>
>>> Hi Mike and Bryan,
>>>
>>> One of my custom processors appears as inactive in NiFi with a yellow
>>> triangle error.
>>> When I hover over it I see a message saying 'Missing Processor'
>>> validated against 'Any Property' is invalid. This is not a valid processor.
>>> In the log it seems to invoke GhostProcessor.java which is giving the
>>> above error when restarting nifi.
>>> This custom processor sits (with my other processors) in my custom_lib
>>> folder and I have provided that path in the nifi properties file as
>>>
>>> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>>>
>>>
>>> Not sure what I missed?
>>>
>>> Do I need to make entry of this custom processor somewhere?
>>>
>>>
>>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>>>
>>>> Thanks Mike and Bryan.
>>>> Yes it seems my template was still referring the old version.
>>>> I will have it updated now and will reimport.
>>>> Also the version of NiFi we are using is the one that comes with CDF. I
>>>> am not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera
>>>> and see if we can get it upgraded.
>>>>
>>>>
>>>>
>>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>>
>>>>> What is in the template for the bundle coordinates of your processor?
>>>>> and does that match the coordinates of the NAR that is deployed?
>>>>>
>>>>> Example:
>>>>>
>>>>> <bundle>
>>>>> <group>org.apache.nifi</group>
>>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>>> <version>1.10.0-SNAPSHOT</version>
>>>>> </bundle>
>>>>>
>>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>>> references 1.0.0, then you'll need to update your template.
>>>>>
>>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > If it's happening immediately upon trying to import the template, I
>>>>> believe that's the error message saying that the 1.9 instance cannot find
>>>>> the NAR file which provided the processor. Also, if you're referring to
>>>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>>>>> there are a few critical bugs fixed in 1.9.2.
>>>>> >
>>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Thanks Bryan.
>>>>> >> My custom processors are part of a template. However when I try to
>>>>> import my template in NiFi 1.9, I get an error message saying
>>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update
>>>>> all the dependencies to NiFi 1.9 and even the plugins. We are using a
>>>>> Cloudera distributed version of NiFi 1.9.
>>>>> >> Any idea why is this happening?
>>>>> >>
>>>>> >> Thanks
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hello,
>>>>> >>>
>>>>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>>>>> >>> but to make sure you just need to update any nifi dependencies in
>>>>> your
>>>>> >>> poms to 1.9.2.
>>>>> >>>
>>>>> >>> If you created your project from the archetype and didn't change
>>>>> >>> anything, then this should just be changing the parent in the root
>>>>> pom
>>>>> >>> to the new version of nifi-nar-bundles.
>>>>> >>>
>>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>>>>> >>> need to change.
>>>>> >>>
>>>>> >>> -Bryan
>>>>> >>>
>>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>>>>> wrote:
>>>>> >>> >
>>>>> >>> > Hi,
>>>>> >>> >
>>>>> >>> > If we have a custom processor that was created with NiFi 1.6,
>>>>> what are the steps we need to follow to make it work in 1.9?
>>>>> >>> > Is there some sort of steps that explains the jar and pom
>>>>> updates we need to do for making it work in 1.9?
>>>>>
>>>> --
>> Sent from Gmail Mobile
>>
> --
Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Yes it does show as an option.
One thing I noticed is that the when the nar is unpacked, the bundled
dependencies are inside META_INF in the work folder in NiFi 1.6.0, however
in NiFI 1.9.0 they go inside NAR_INF.
Why does this happen?
It seems the custom processor that we have uses Springboot, and references
applicationcontext file which was inside META_INF when it was built.
However I cant see that file anymore in the unpacked nar.
On Tue, Aug 13, 2019 at 8:57 PM Bryan Bende <bb...@gmail.com> wrote:
> Does that custom processor type show as an option if you try to add a new
> processor to the canvas?
>
> On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
>
>> Hi Mike and Bryan,
>>
>> One of my custom processors appears as inactive in NiFi with a yellow
>> triangle error.
>> When I hover over it I see a message saying 'Missing Processor' validated
>> against 'Any Property' is invalid. This is not a valid processor.
>> In the log it seems to invoke GhostProcessor.java which is giving the
>> above error when restarting nifi.
>> This custom processor sits (with my other processors) in my custom_lib
>> folder and I have provided that path in the nifi properties file as
>>
>> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>>
>>
>> Not sure what I missed?
>>
>> Do I need to make entry of this custom processor somewhere?
>>
>>
>> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>>
>>> Thanks Mike and Bryan.
>>> Yes it seems my template was still referring the old version.
>>> I will have it updated now and will reimport.
>>> Also the version of NiFi we are using is the one that comes with CDF. I
>>> am not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera
>>> and see if we can get it upgraded.
>>>
>>>
>>>
>>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>>
>>>> What is in the template for the bundle coordinates of your processor?
>>>> and does that match the coordinates of the NAR that is deployed?
>>>>
>>>> Example:
>>>>
>>>> <bundle>
>>>> <group>org.apache.nifi</group>
>>>> <artifact>nifi-update-attribute-nar</artifact>
>>>> <version>1.10.0-SNAPSHOT</version>
>>>> </bundle>
>>>>
>>>> If you made a new version of your NAR, say 2.0.0 and your template
>>>> references 1.0.0, then you'll need to update your template.
>>>>
>>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
>>>> wrote:
>>>> >
>>>> > If it's happening immediately upon trying to import the template, I
>>>> believe that's the error message saying that the 1.9 instance cannot find
>>>> the NAR file which provided the processor. Also, if you're referring to
>>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>>>> there are a few critical bugs fixed in 1.9.2.
>>>> >
>>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Thanks Bryan.
>>>> >> My custom processors are part of a template. However when I try to
>>>> import my template in NiFi 1.9, I get an error message saying
>>>> >> PutFeedMetadata is not known to this NiFi instance. I did update all
>>>> the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera
>>>> distributed version of NiFi 1.9.
>>>> >> Any idea why is this happening?
>>>> >>
>>>> >> Thanks
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>>>> >>>
>>>> >>> Hello,
>>>> >>>
>>>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>>>> >>> but to make sure you just need to update any nifi dependencies in
>>>> your
>>>> >>> poms to 1.9.2.
>>>> >>>
>>>> >>> If you created your project from the archetype and didn't change
>>>> >>> anything, then this should just be changing the parent in the root
>>>> pom
>>>> >>> to the new version of nifi-nar-bundles.
>>>> >>>
>>>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>>>> >>> need to change.
>>>> >>>
>>>> >>> -Bryan
>>>> >>>
>>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>>>> wrote:
>>>> >>> >
>>>> >>> > Hi,
>>>> >>> >
>>>> >>> > If we have a custom processor that was created with NiFi 1.6,
>>>> what are the steps we need to follow to make it work in 1.9?
>>>> >>> > Is there some sort of steps that explains the jar and pom updates
>>>> we need to do for making it work in 1.9?
>>>>
>>> --
> Sent from Gmail Mobile
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
Does that custom processor type show as an option if you try to add a new
processor to the canvas?
On Tue, Aug 13, 2019 at 4:54 PM Bimal Mehta <bi...@gmail.com> wrote:
> Hi Mike and Bryan,
>
> One of my custom processors appears as inactive in NiFi with a yellow
> triangle error.
> When I hover over it I see a message saying 'Missing Processor' validated
> against 'Any Property' is invalid. This is not a valid processor.
> In the log it seems to invoke GhostProcessor.java which is giving the
> above error when restarting nifi.
> This custom processor sits (with my other processors) in my custom_lib
> folder and I have provided that path in the nifi properties file as
>
> *nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
>
>
> Not sure what I missed?
>
> Do I need to make entry of this custom processor somewhere?
>
>
> On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
>
>> Thanks Mike and Bryan.
>> Yes it seems my template was still referring the old version.
>> I will have it updated now and will reimport.
>> Also the version of NiFi we are using is the one that comes with CDF. I
>> am not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera
>> and see if we can get it upgraded.
>>
>>
>>
>> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>>
>>> What is in the template for the bundle coordinates of your processor?
>>> and does that match the coordinates of the NAR that is deployed?
>>>
>>> Example:
>>>
>>> <bundle>
>>> <group>org.apache.nifi</group>
>>> <artifact>nifi-update-attribute-nar</artifact>
>>> <version>1.10.0-SNAPSHOT</version>
>>> </bundle>
>>>
>>> If you made a new version of your NAR, say 2.0.0 and your template
>>> references 1.0.0, then you'll need to update your template.
>>>
>>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
>>> wrote:
>>> >
>>> > If it's happening immediately upon trying to import the template, I
>>> believe that's the error message saying that the 1.9 instance cannot find
>>> the NAR file which provided the processor. Also, if you're referring to
>>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>>> there are a few critical bugs fixed in 1.9.2.
>>> >
>>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
>>> >>
>>> >> Thanks Bryan.
>>> >> My custom processors are part of a template. However when I try to
>>> import my template in NiFi 1.9, I get an error message saying
>>> >> PutFeedMetadata is not known to this NiFi instance. I did update all
>>> the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera
>>> distributed version of NiFi 1.9.
>>> >> Any idea why is this happening?
>>> >>
>>> >> Thanks
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>>> >>>
>>> >>> Hello,
>>> >>>
>>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>>> >>> but to make sure you just need to update any nifi dependencies in
>>> your
>>> >>> poms to 1.9.2.
>>> >>>
>>> >>> If you created your project from the archetype and didn't change
>>> >>> anything, then this should just be changing the parent in the root
>>> pom
>>> >>> to the new version of nifi-nar-bundles.
>>> >>>
>>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>>> >>> need to change.
>>> >>>
>>> >>> -Bryan
>>> >>>
>>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>>> wrote:
>>> >>> >
>>> >>> > Hi,
>>> >>> >
>>> >>> > If we have a custom processor that was created with NiFi 1.6, what
>>> are the steps we need to follow to make it work in 1.9?
>>> >>> > Is there some sort of steps that explains the jar and pom updates
>>> we need to do for making it work in 1.9?
>>>
>> --
Sent from Gmail Mobile
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Hi Mike and Bryan,
One of my custom processors appears as inactive in NiFi with a yellow
triangle error.
When I hover over it I see a message saying 'Missing Processor' validated
against 'Any Property' is invalid. This is not a valid processor.
In the log it seems to invoke GhostProcessor.java which is giving the above
error when restarting nifi.
This custom processor sits (with my other processors) in my custom_lib
folder and I have provided that path in the nifi properties file as
*nifi.nar.library.directory.custom=/opt/nifi/custom_lib*
Not sure what I missed?
Do I need to make entry of this custom processor somewhere?
On Thu, Aug 8, 2019 at 9:14 AM Bimal Mehta <bi...@gmail.com> wrote:
> Thanks Mike and Bryan.
> Yes it seems my template was still referring the old version.
> I will have it updated now and will reimport.
> Also the version of NiFi we are using is the one that comes with CDF. I am
> not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera and
> see if we can get it upgraded.
>
>
>
> On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
>
>> What is in the template for the bundle coordinates of your processor?
>> and does that match the coordinates of the NAR that is deployed?
>>
>> Example:
>>
>> <bundle>
>> <group>org.apache.nifi</group>
>> <artifact>nifi-update-attribute-nar</artifact>
>> <version>1.10.0-SNAPSHOT</version>
>> </bundle>
>>
>> If you made a new version of your NAR, say 2.0.0 and your template
>> references 1.0.0, then you'll need to update your template.
>>
>> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
>> wrote:
>> >
>> > If it's happening immediately upon trying to import the template, I
>> believe that's the error message saying that the 1.9 instance cannot find
>> the NAR file which provided the processor. Also, if you're referring to
>> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
>> there are a few critical bugs fixed in 1.9.2.
>> >
>> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >>
>> >> Thanks Bryan.
>> >> My custom processors are part of a template. However when I try to
>> import my template in NiFi 1.9, I get an error message saying
>> >> PutFeedMetadata is not known to this NiFi instance. I did update all
>> the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera
>> distributed version of NiFi 1.9.
>> >> Any idea why is this happening?
>> >>
>> >> Thanks
>> >>
>> >>
>> >>
>> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>> >>>
>> >>> Hello,
>> >>>
>> >>> Most likely your processor built against 1.6 would run fine in 1.9,
>> >>> but to make sure you just need to update any nifi dependencies in your
>> >>> poms to 1.9.2.
>> >>>
>> >>> If you created your project from the archetype and didn't change
>> >>> anything, then this should just be changing the parent in the root pom
>> >>> to the new version of nifi-nar-bundles.
>> >>>
>> >>> If you set it up yourself, then anywhere you depend on nifi-api you
>> >>> need to change.
>> >>>
>> >>> -Bryan
>> >>>
>> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com>
>> wrote:
>> >>> >
>> >>> > Hi,
>> >>> >
>> >>> > If we have a custom processor that was created with NiFi 1.6, what
>> are the steps we need to follow to make it work in 1.9?
>> >>> > Is there some sort of steps that explains the jar and pom updates
>> we need to do for making it work in 1.9?
>>
>
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Thanks Mike and Bryan.
Yes it seems my template was still referring the old version.
I will have it updated now and will reimport.
Also the version of NiFi we are using is the one that comes with CDF. I am
not sure if CDF supports 1.9.2 yet or not. I will reach out to Cloudera and
see if we can get it upgraded.
On Thu, Aug 8, 2019, 8:51 AM Bryan Bende <bb...@gmail.com> wrote:
> What is in the template for the bundle coordinates of your processor?
> and does that match the coordinates of the NAR that is deployed?
>
> Example:
>
> <bundle>
> <group>org.apache.nifi</group>
> <artifact>nifi-update-attribute-nar</artifact>
> <version>1.10.0-SNAPSHOT</version>
> </bundle>
>
> If you made a new version of your NAR, say 2.0.0 and your template
> references 1.0.0, then you'll need to update your template.
>
> On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > If it's happening immediately upon trying to import the template, I
> believe that's the error message saying that the 1.9 instance cannot find
> the NAR file which provided the processor. Also, if you're referring to
> 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because
> there are a few critical bugs fixed in 1.9.2.
> >
> > On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
> >>
> >> Thanks Bryan.
> >> My custom processors are part of a template. However when I try to
> import my template in NiFi 1.9, I get an error message saying
> >> PutFeedMetadata is not known to this NiFi instance. I did update all
> the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera
> distributed version of NiFi 1.9.
> >> Any idea why is this happening?
> >>
> >> Thanks
> >>
> >>
> >>
> >> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Most likely your processor built against 1.6 would run fine in 1.9,
> >>> but to make sure you just need to update any nifi dependencies in your
> >>> poms to 1.9.2.
> >>>
> >>> If you created your project from the archetype and didn't change
> >>> anything, then this should just be changing the parent in the root pom
> >>> to the new version of nifi-nar-bundles.
> >>>
> >>> If you set it up yourself, then anywhere you depend on nifi-api you
> >>> need to change.
> >>>
> >>> -Bryan
> >>>
> >>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
> >>> >
> >>> > Hi,
> >>> >
> >>> > If we have a custom processor that was created with NiFi 1.6, what
> are the steps we need to follow to make it work in 1.9?
> >>> > Is there some sort of steps that explains the jar and pom updates we
> need to do for making it work in 1.9?
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
What is in the template for the bundle coordinates of your processor?
and does that match the coordinates of the NAR that is deployed?
Example:
<bundle>
<group>org.apache.nifi</group>
<artifact>nifi-update-attribute-nar</artifact>
<version>1.10.0-SNAPSHOT</version>
</bundle>
If you made a new version of your NAR, say 2.0.0 and your template
references 1.0.0, then you'll need to update your template.
On Wed, Aug 7, 2019 at 10:05 PM Mike Thomsen <mi...@gmail.com> wrote:
>
> If it's happening immediately upon trying to import the template, I believe that's the error message saying that the 1.9 instance cannot find the NAR file which provided the processor. Also, if you're referring to 1.9.0 and not 1.9.2 you're going to want to upgrade to the latter because there are a few critical bugs fixed in 1.9.2.
>
> On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
>>
>> Thanks Bryan.
>> My custom processors are part of a template. However when I try to import my template in NiFi 1.9, I get an error message saying
>> PutFeedMetadata is not known to this NiFi instance. I did update all the dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera distributed version of NiFi 1.9.
>> Any idea why is this happening?
>>
>> Thanks
>>
>>
>>
>> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Most likely your processor built against 1.6 would run fine in 1.9,
>>> but to make sure you just need to update any nifi dependencies in your
>>> poms to 1.9.2.
>>>
>>> If you created your project from the archetype and didn't change
>>> anything, then this should just be changing the parent in the root pom
>>> to the new version of nifi-nar-bundles.
>>>
>>> If you set it up yourself, then anywhere you depend on nifi-api you
>>> need to change.
>>>
>>> -Bryan
>>>
>>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > If we have a custom processor that was created with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
>>> > Is there some sort of steps that explains the jar and pom updates we need to do for making it work in 1.9?
Re: Custom Processor Upgrade
Posted by Mike Thomsen <mi...@gmail.com>.
If it's happening immediately upon trying to import the template, I believe
that's the error message saying that the 1.9 instance cannot find the NAR
file which provided the processor. Also, if you're referring to 1.9.0 and
not 1.9.2 you're going to want to upgrade to the latter because there are a
few critical bugs fixed in 1.9.2.
On Wed, Aug 7, 2019 at 9:19 PM Bimal Mehta <bi...@gmail.com> wrote:
> Thanks Bryan.
> My custom processors are part of a template. However when I try to import
> my template in NiFi 1.9, I get an error message saying
> PutFeedMetadata is not known to this NiFi instance. I did update all the
> dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera
> distributed version of NiFi 1.9.
> Any idea why is this happening?
>
> Thanks
>
>
>
> On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
>
>> Hello,
>>
>> Most likely your processor built against 1.6 would run fine in 1.9,
>> but to make sure you just need to update any nifi dependencies in your
>> poms to 1.9.2.
>>
>> If you created your project from the archetype and didn't change
>> anything, then this should just be changing the parent in the root pom
>> to the new version of nifi-nar-bundles.
>>
>> If you set it up yourself, then anywhere you depend on nifi-api you
>> need to change.
>>
>> -Bryan
>>
>> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > If we have a custom processor that was created with NiFi 1.6, what are
>> the steps we need to follow to make it work in 1.9?
>> > Is there some sort of steps that explains the jar and pom updates we
>> need to do for making it work in 1.9?
>>
>
Re: Custom Processor Upgrade
Posted by Bimal Mehta <bi...@gmail.com>.
Thanks Bryan.
My custom processors are part of a template. However when I try to import
my template in NiFi 1.9, I get an error message saying
PutFeedMetadata is not known to this NiFi instance. I did update all the
dependencies to NiFi 1.9 and even the plugins. We are using a Cloudera
distributed version of NiFi 1.9.
Any idea why is this happening?
Thanks
On Wed, Aug 7, 2019 at 3:46 PM Bryan Bende <bb...@gmail.com> wrote:
> Hello,
>
> Most likely your processor built against 1.6 would run fine in 1.9,
> but to make sure you just need to update any nifi dependencies in your
> poms to 1.9.2.
>
> If you created your project from the archetype and didn't change
> anything, then this should just be changing the parent in the root pom
> to the new version of nifi-nar-bundles.
>
> If you set it up yourself, then anywhere you depend on nifi-api you
> need to change.
>
> -Bryan
>
> On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
> >
> > Hi,
> >
> > If we have a custom processor that was created with NiFi 1.6, what are
> the steps we need to follow to make it work in 1.9?
> > Is there some sort of steps that explains the jar and pom updates we
> need to do for making it work in 1.9?
>
Re: Custom Processor Upgrade
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
Most likely your processor built against 1.6 would run fine in 1.9,
but to make sure you just need to update any nifi dependencies in your
poms to 1.9.2.
If you created your project from the archetype and didn't change
anything, then this should just be changing the parent in the root pom
to the new version of nifi-nar-bundles.
If you set it up yourself, then anywhere you depend on nifi-api you
need to change.
-Bryan
On Wed, Aug 7, 2019 at 3:18 PM Bimal Mehta <bi...@gmail.com> wrote:
>
> Hi,
>
> If we have a custom processor that was created with NiFi 1.6, what are the steps we need to follow to make it work in 1.9?
> Is there some sort of steps that explains the jar and pom updates we need to do for making it work in 1.9?