You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Mark Petronic <ma...@gmail.com> on 2015/11/20 01:53:48 UTC
Nifi startup error after I install my first custom processor
Well, I finally built my first processor. Was a nice experience. Nice APIs!
This was my first work with Maven and Nifi so, if things look
wrong/strange, please go easy on me. :) The issue is when I copy the NAR
file in the lib directory and restart Nifi, it does not start up. I tried
removing the work directory for a clean start, no joy. If I remove my NAR,
works fine. Wondering if one of you experts would not mind helping me over
this hump because I have no clue what is happening based on the error
message in the log. Seems like some other processor, DetectDuplicate, is
failing to startup when mine is in there?? I pasted the startup log traces
here: http://pastebin.com/raw.php?i=UzraAE53. My processor code is here:
https://bitbucket.org/mpetronic/nificustomprocessors. I would be very
grateful for any guidance. It seems like a pretty simple processor so I was
surprised this happened. The code is commented so hope that helps you
easily understand what is going on there. Oh, I started from this very
helpful post (thanks Bryan):
http://bryanbende.com/development/2015/02/04/custom-processors-for-apache-nifi/.
I am running a build of Nifi from the GitHub mirror,
commit 90f6830003b76204e25dec9fafec2488c8bda550.
Thanks,
Mark
Re: Nifi startup error after I install my first custom processor
Posted by Mark Petronic <ma...@gmail.com>.
Bryan, thanks so much for the pointers. I got is all working now and
ready to start on the next processor with a whole lot more confidence.
:) FWIW, maybe for someone else who needs this information, I just
added a dependency for logback-classic to get more than the default
slf4f-simple logging (which is standard error console logging only, I
believe)
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</dependency>
And then I just used the typical LoggerFactory.getLogger() to get a
logger in my helper classes and that's it.
I still get this warning in eclipse when I run unit tests but it just
happens to pick logback-classic and same when I run in production so I
do see my log messages going to the app log. I could also play around
with logback.xml now and direct my stuff where ever I want. But, at
least the traces are showing up now so that works. I did a lot of
research into how to exclude slf4j-simple from the classpath in Maven
to avoid the contention warning but nothing I read or tried works.
Anyway, it does pick the logger I wanted so I will ignore for now.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/mpetronic/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/mpetronic/.m2/repository/org/slf4j/slf4j-simple/1.7.12/slf4j-simple-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type
[ch.qos.logback.classic.util.ContextSelectorStaticBinder]
mvn dependency:tree gives this and it appears to me that that the
slf4j-simple dependency is coming from the root of that tree (test) so
maybe that is an internal dependency defined by the maven test runner
or something? Very new to maven so just guessing.
--- maven-dependency-plugin:2.9:tree (default-cli) @ hughes-bigdata-bundle ---
[INFO] com.hughes:hughes-bigdata-bundle:pom:1.0
[INFO] +- junit:junit:jar:4.12:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.mockito:mockito-core:jar:1.10.19:test
[INFO] | \- org.objenesis:objenesis:jar:2.1:test
[INFO] \- org.slf4j:slf4j-simple:jar:1.7.12:test
[INFO] \- org.slf4j:slf4j-api:jar:1.7.12:provided
On Fri, Nov 20, 2015 at 8:31 AM, Bryan Bende <bb...@gmail.com> wrote:
> The dependencies can definitely be confusing to wrap your head around. A
> good rule of thumb is that you would generally not have a direct jar
> dependency on anything under nifi-nar-bundles in the source tree. This
> would mean no jar dependencies on processors, controller services,
> reporting tasks etc. Now there are cases where a NAR can depend on another
> NAR, this is common when a processor needs to use a controller service, a
> good description of that is here [1], another example is how we have the
> hadoop-libraries-nar so multiple NARs can depend on the same set of hadoop
> libraries and not have to duplicate all those jars. Any other dependencies
> like things from nifi-commons, such as processor utils and others, are fair
> game to include in your NAR. If you look at the nifi lib directory you can
> see the following jars:
>
> jcl-over-slf4j-1.7.12.jar
> jul-to-slf4j-1.7.12.jar
> log4j-over-slf4j-1.7.12.jar
> logback-classic-1.1.3.jar
> logback-core-1.1.3.jar
> nifi-api-0.4.0-SNAPSHOT.jar
> nifi-documentation-0.4.0-SNAPSHOT.jar
> nifi-nar-utils-0.4.0-SNAPSHOT.jar
> nifi-properties-0.4.0-SNAPSHOT.jar
> nifi-runtime-0.4.0-SNAPSHOT.jar
> slf4j-api-1.7.12.jar
>
> These are automatically available to every nar, anything else needs to be
> brought in by bundling the jars in your NAR, or by depending on another NAR.
>
> As for logging, nifi itself uses slf4j and logback, so the logger you get
> from getLogger() would be controlled from the logback.xml in the conf
> directory. I think for non-processor classes if you use the slf4j api that
> will be your best bet as I believe it will automatically use the same
> logback configuration, but others could correct me if I am wrong here. I
> believe it is also possible to use log4j directly with in your NAR, for
> example I know some third party client libraries use log4j and by having a
> dependency on log4j-over-slf4j it can somehow route all of the log4j calls
> back through the main slf4j configuration. Hope I didn't confuse things
> more here, let us know.
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
>
>
>
> On Fri, Nov 20, 2015 at 1:26 AM, Mark Petronic <ma...@gmail.com>
> wrote:
>
>> Thanks so much Bryan. It is running now. It is starting to come together
>> but I'm still a little unclear on when to include nifi files to pick up
>> dependencies and when to directly include the upstream dependencies. Like
>> say some nifi nar like nifi-processor-utils already has a dependency on
>> commons-lang3 and my custom processor needs commons-lang3, should I satisfy
>> that by depending on nifi-processor-utils (which in know will already be in
>> the install) or should I add a dependency on commons-lang3 in my nar and
>> duplicate the libraries? I realize that there are version issues to be
>> concerned with which I believe is the genesis for nars to start with but
>> assume I need the same version, too? What't the general best practice
>> there?
>>
>> Also, now that I was able to run my processor, I have a question about
>> logging. I see that the processors use getLogger() to get a logger. But is
>> see in other places a more traditional use of LoggerFactory.getLogger().
>> What is the best approach to logging from helper classes that I use in my
>> processor that are not inner classes of the processor class by defined
>> outside in separate class files because I intend to reuse them across
>> multiple processor types? I need to debug an error that I am seeing in one
>> of those helper classes and need to add some logging.
>>
>> Mark
>>
>> On Thu, Nov 19, 2015 at 8:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>>
>> > Hi Mark,
>> >
>> > Glad to hear you were able to get started building processors, and glad
>> > that blog post helped!
>> >
>> > I pulled down your code and built and deployed it. It looks like the
>> issue
>> > is that your processors pom has a dependency on:
>> > <dependency>
>> > <groupId>org.apache.nifi</groupId>
>> > <artifactId>nifi-standard-processors</artifactId>
>> > <version>0.4.0-SNAPSHOT</version>
>> > </dependency>
>> >
>> > Which means your NAR ends up having the standard processors jar in it,
>> but
>> > they are also deployed in nifi-standard-nar which causes some problems.
>> You
>> > can see the jar is there by looking
>> > in
>> >
>> work/nar/extensions/nifi-bigdata-nar-1.0.nar-unpacked/META-INF/bundled-dependencies/.
>> >
>> > I removed that dependency from the pom and it looked like the only
>> > compilation errors were on some missing json related libraries (jackson
>> and
>> > json path). I added these to your processor pom and it seems to deploy
>> now:
>> >
>> > <dependency>
>> > <groupId>com.jayway.jsonpath</groupId>
>> > <artifactId>json-path</artifactId>
>> > <version>2.0.0</version>
>> > </dependency>
>> > <dependency>
>> > <groupId>org.codehaus.jackson</groupId>
>> > <artifactId>jackson-mapper-asl</artifactId>
>> > <version>1.9.13</version>
>> > </dependency>
>> >
>> > Hope that helps. Let us know if you have any other questions!
>> >
>> > -Bryan
>> >
>> >
>> > On Thu, Nov 19, 2015 at 7:53 PM, Mark Petronic <ma...@gmail.com>
>> > wrote:
>> >
>> > > Well, I finally built my first processor. Was a nice experience. Nice
>> > APIs!
>> > > This was my first work with Maven and Nifi so, if things look
>> > > wrong/strange, please go easy on me. :) The issue is when I copy the
>> NAR
>> > > file in the lib directory and restart Nifi, it does not start up. I
>> tried
>> > > removing the work directory for a clean start, no joy. If I remove my
>> > NAR,
>> > > works fine. Wondering if one of you experts would not mind helping me
>> > over
>> > > this hump because I have no clue what is happening based on the error
>> > > message in the log. Seems like some other processor, DetectDuplicate,
>> is
>> > > failing to startup when mine is in there?? I pasted the startup log
>> > traces
>> > > here: http://pastebin.com/raw.php?i=UzraAE53. My processor code is
>> here:
>> > > https://bitbucket.org/mpetronic/nificustomprocessors. I would be very
>> > > grateful for any guidance. It seems like a pretty simple processor so I
>> > was
>> > > surprised this happened. The code is commented so hope that helps you
>> > > easily understand what is going on there. Oh, I started from this very
>> > > helpful post (thanks Bryan):
>> > >
>> > >
>> >
>> http://bryanbende.com/development/2015/02/04/custom-processors-for-apache-nifi/
>> > > .
>> > > I am running a build of Nifi from the GitHub mirror,
>> > > commit 90f6830003b76204e25dec9fafec2488c8bda550.
>> > >
>> > > Thanks,
>> > > Mark
>> > >
>> >
>>
Re: Nifi startup error after I install my first custom processor
Posted by Bryan Bende <bb...@gmail.com>.
The dependencies can definitely be confusing to wrap your head around. A
good rule of thumb is that you would generally not have a direct jar
dependency on anything under nifi-nar-bundles in the source tree. This
would mean no jar dependencies on processors, controller services,
reporting tasks etc. Now there are cases where a NAR can depend on another
NAR, this is common when a processor needs to use a controller service, a
good description of that is here [1], another example is how we have the
hadoop-libraries-nar so multiple NARs can depend on the same set of hadoop
libraries and not have to duplicate all those jars. Any other dependencies
like things from nifi-commons, such as processor utils and others, are fair
game to include in your NAR. If you look at the nifi lib directory you can
see the following jars:
jcl-over-slf4j-1.7.12.jar
jul-to-slf4j-1.7.12.jar
log4j-over-slf4j-1.7.12.jar
logback-classic-1.1.3.jar
logback-core-1.1.3.jar
nifi-api-0.4.0-SNAPSHOT.jar
nifi-documentation-0.4.0-SNAPSHOT.jar
nifi-nar-utils-0.4.0-SNAPSHOT.jar
nifi-properties-0.4.0-SNAPSHOT.jar
nifi-runtime-0.4.0-SNAPSHOT.jar
slf4j-api-1.7.12.jar
These are automatically available to every nar, anything else needs to be
brought in by bundling the jars in your NAR, or by depending on another NAR.
As for logging, nifi itself uses slf4j and logback, so the logger you get
from getLogger() would be controlled from the logback.xml in the conf
directory. I think for non-processor classes if you use the slf4j api that
will be your best bet as I believe it will automatically use the same
logback configuration, but others could correct me if I am wrong here. I
believe it is also possible to use log4j directly with in your NAR, for
example I know some third party client libraries use log4j and by having a
dependency on log4j-over-slf4j it can somehow route all of the log4j calls
back through the main slf4j configuration. Hope I didn't confuse things
more here, let us know.
[1]
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
On Fri, Nov 20, 2015 at 1:26 AM, Mark Petronic <ma...@gmail.com>
wrote:
> Thanks so much Bryan. It is running now. It is starting to come together
> but I'm still a little unclear on when to include nifi files to pick up
> dependencies and when to directly include the upstream dependencies. Like
> say some nifi nar like nifi-processor-utils already has a dependency on
> commons-lang3 and my custom processor needs commons-lang3, should I satisfy
> that by depending on nifi-processor-utils (which in know will already be in
> the install) or should I add a dependency on commons-lang3 in my nar and
> duplicate the libraries? I realize that there are version issues to be
> concerned with which I believe is the genesis for nars to start with but
> assume I need the same version, too? What't the general best practice
> there?
>
> Also, now that I was able to run my processor, I have a question about
> logging. I see that the processors use getLogger() to get a logger. But is
> see in other places a more traditional use of LoggerFactory.getLogger().
> What is the best approach to logging from helper classes that I use in my
> processor that are not inner classes of the processor class by defined
> outside in separate class files because I intend to reuse them across
> multiple processor types? I need to debug an error that I am seeing in one
> of those helper classes and need to add some logging.
>
> Mark
>
> On Thu, Nov 19, 2015 at 8:23 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> > Hi Mark,
> >
> > Glad to hear you were able to get started building processors, and glad
> > that blog post helped!
> >
> > I pulled down your code and built and deployed it. It looks like the
> issue
> > is that your processors pom has a dependency on:
> > <dependency>
> > <groupId>org.apache.nifi</groupId>
> > <artifactId>nifi-standard-processors</artifactId>
> > <version>0.4.0-SNAPSHOT</version>
> > </dependency>
> >
> > Which means your NAR ends up having the standard processors jar in it,
> but
> > they are also deployed in nifi-standard-nar which causes some problems.
> You
> > can see the jar is there by looking
> > in
> >
> work/nar/extensions/nifi-bigdata-nar-1.0.nar-unpacked/META-INF/bundled-dependencies/.
> >
> > I removed that dependency from the pom and it looked like the only
> > compilation errors were on some missing json related libraries (jackson
> and
> > json path). I added these to your processor pom and it seems to deploy
> now:
> >
> > <dependency>
> > <groupId>com.jayway.jsonpath</groupId>
> > <artifactId>json-path</artifactId>
> > <version>2.0.0</version>
> > </dependency>
> > <dependency>
> > <groupId>org.codehaus.jackson</groupId>
> > <artifactId>jackson-mapper-asl</artifactId>
> > <version>1.9.13</version>
> > </dependency>
> >
> > Hope that helps. Let us know if you have any other questions!
> >
> > -Bryan
> >
> >
> > On Thu, Nov 19, 2015 at 7:53 PM, Mark Petronic <ma...@gmail.com>
> > wrote:
> >
> > > Well, I finally built my first processor. Was a nice experience. Nice
> > APIs!
> > > This was my first work with Maven and Nifi so, if things look
> > > wrong/strange, please go easy on me. :) The issue is when I copy the
> NAR
> > > file in the lib directory and restart Nifi, it does not start up. I
> tried
> > > removing the work directory for a clean start, no joy. If I remove my
> > NAR,
> > > works fine. Wondering if one of you experts would not mind helping me
> > over
> > > this hump because I have no clue what is happening based on the error
> > > message in the log. Seems like some other processor, DetectDuplicate,
> is
> > > failing to startup when mine is in there?? I pasted the startup log
> > traces
> > > here: http://pastebin.com/raw.php?i=UzraAE53. My processor code is
> here:
> > > https://bitbucket.org/mpetronic/nificustomprocessors. I would be very
> > > grateful for any guidance. It seems like a pretty simple processor so I
> > was
> > > surprised this happened. The code is commented so hope that helps you
> > > easily understand what is going on there. Oh, I started from this very
> > > helpful post (thanks Bryan):
> > >
> > >
> >
> http://bryanbende.com/development/2015/02/04/custom-processors-for-apache-nifi/
> > > .
> > > I am running a build of Nifi from the GitHub mirror,
> > > commit 90f6830003b76204e25dec9fafec2488c8bda550.
> > >
> > > Thanks,
> > > Mark
> > >
> >
>
Re: Nifi startup error after I install my first custom processor
Posted by Mark Petronic <ma...@gmail.com>.
Thanks so much Bryan. It is running now. It is starting to come together
but I'm still a little unclear on when to include nifi files to pick up
dependencies and when to directly include the upstream dependencies. Like
say some nifi nar like nifi-processor-utils already has a dependency on
commons-lang3 and my custom processor needs commons-lang3, should I satisfy
that by depending on nifi-processor-utils (which in know will already be in
the install) or should I add a dependency on commons-lang3 in my nar and
duplicate the libraries? I realize that there are version issues to be
concerned with which I believe is the genesis for nars to start with but
assume I need the same version, too? What't the general best practice there?
Also, now that I was able to run my processor, I have a question about
logging. I see that the processors use getLogger() to get a logger. But is
see in other places a more traditional use of LoggerFactory.getLogger().
What is the best approach to logging from helper classes that I use in my
processor that are not inner classes of the processor class by defined
outside in separate class files because I intend to reuse them across
multiple processor types? I need to debug an error that I am seeing in one
of those helper classes and need to add some logging.
Mark
On Thu, Nov 19, 2015 at 8:23 PM, Bryan Bende <bb...@gmail.com> wrote:
> Hi Mark,
>
> Glad to hear you were able to get started building processors, and glad
> that blog post helped!
>
> I pulled down your code and built and deployed it. It looks like the issue
> is that your processors pom has a dependency on:
> <dependency>
> <groupId>org.apache.nifi</groupId>
> <artifactId>nifi-standard-processors</artifactId>
> <version>0.4.0-SNAPSHOT</version>
> </dependency>
>
> Which means your NAR ends up having the standard processors jar in it, but
> they are also deployed in nifi-standard-nar which causes some problems. You
> can see the jar is there by looking
> in
> work/nar/extensions/nifi-bigdata-nar-1.0.nar-unpacked/META-INF/bundled-dependencies/.
>
> I removed that dependency from the pom and it looked like the only
> compilation errors were on some missing json related libraries (jackson and
> json path). I added these to your processor pom and it seems to deploy now:
>
> <dependency>
> <groupId>com.jayway.jsonpath</groupId>
> <artifactId>json-path</artifactId>
> <version>2.0.0</version>
> </dependency>
> <dependency>
> <groupId>org.codehaus.jackson</groupId>
> <artifactId>jackson-mapper-asl</artifactId>
> <version>1.9.13</version>
> </dependency>
>
> Hope that helps. Let us know if you have any other questions!
>
> -Bryan
>
>
> On Thu, Nov 19, 2015 at 7:53 PM, Mark Petronic <ma...@gmail.com>
> wrote:
>
> > Well, I finally built my first processor. Was a nice experience. Nice
> APIs!
> > This was my first work with Maven and Nifi so, if things look
> > wrong/strange, please go easy on me. :) The issue is when I copy the NAR
> > file in the lib directory and restart Nifi, it does not start up. I tried
> > removing the work directory for a clean start, no joy. If I remove my
> NAR,
> > works fine. Wondering if one of you experts would not mind helping me
> over
> > this hump because I have no clue what is happening based on the error
> > message in the log. Seems like some other processor, DetectDuplicate, is
> > failing to startup when mine is in there?? I pasted the startup log
> traces
> > here: http://pastebin.com/raw.php?i=UzraAE53. My processor code is here:
> > https://bitbucket.org/mpetronic/nificustomprocessors. I would be very
> > grateful for any guidance. It seems like a pretty simple processor so I
> was
> > surprised this happened. The code is commented so hope that helps you
> > easily understand what is going on there. Oh, I started from this very
> > helpful post (thanks Bryan):
> >
> >
> http://bryanbende.com/development/2015/02/04/custom-processors-for-apache-nifi/
> > .
> > I am running a build of Nifi from the GitHub mirror,
> > commit 90f6830003b76204e25dec9fafec2488c8bda550.
> >
> > Thanks,
> > Mark
> >
>
Re: Nifi startup error after I install my first custom processor
Posted by Bryan Bende <bb...@gmail.com>.
Hi Mark,
Glad to hear you were able to get started building processors, and glad
that blog post helped!
I pulled down your code and built and deployed it. It looks like the issue
is that your processors pom has a dependency on:
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-standard-processors</artifactId>
<version>0.4.0-SNAPSHOT</version>
</dependency>
Which means your NAR ends up having the standard processors jar in it, but
they are also deployed in nifi-standard-nar which causes some problems. You
can see the jar is there by looking
in work/nar/extensions/nifi-bigdata-nar-1.0.nar-unpacked/META-INF/bundled-dependencies/.
I removed that dependency from the pom and it looked like the only
compilation errors were on some missing json related libraries (jackson and
json path). I added these to your processor pom and it seems to deploy now:
<dependency>
<groupId>com.jayway.jsonpath</groupId>
<artifactId>json-path</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
Hope that helps. Let us know if you have any other questions!
-Bryan
On Thu, Nov 19, 2015 at 7:53 PM, Mark Petronic <ma...@gmail.com>
wrote:
> Well, I finally built my first processor. Was a nice experience. Nice APIs!
> This was my first work with Maven and Nifi so, if things look
> wrong/strange, please go easy on me. :) The issue is when I copy the NAR
> file in the lib directory and restart Nifi, it does not start up. I tried
> removing the work directory for a clean start, no joy. If I remove my NAR,
> works fine. Wondering if one of you experts would not mind helping me over
> this hump because I have no clue what is happening based on the error
> message in the log. Seems like some other processor, DetectDuplicate, is
> failing to startup when mine is in there?? I pasted the startup log traces
> here: http://pastebin.com/raw.php?i=UzraAE53. My processor code is here:
> https://bitbucket.org/mpetronic/nificustomprocessors. I would be very
> grateful for any guidance. It seems like a pretty simple processor so I was
> surprised this happened. The code is commented so hope that helps you
> easily understand what is going on there. Oh, I started from this very
> helpful post (thanks Bryan):
>
> http://bryanbende.com/development/2015/02/04/custom-processors-for-apache-nifi/
> .
> I am running a build of Nifi from the GitHub mirror,
> commit 90f6830003b76204e25dec9fafec2488c8bda550.
>
> Thanks,
> Mark
>