You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Chandrashekhar Kotekar <sh...@gmail.com> on 2019/09/25 18:09:16 UTC

How to create custom processor which needs 2 or more NiFi bundles?

Hi,

I have developed some custom processors within single Maven project (lets
call this project custom-processors). These processors need Hadoop
dependencies (nifi-hadoop-nar), record-serde related dependencies
(nifi-record-serialization-service-api) and Kerberos dependencies apart
from usual NiFI dependencies.

In pom.xml file of NAR, when I mentioned "nifi-standard-services-api-nar"
as shown below

<dependency>
            <groupId>org.apache.nifi</groupId>
            <artifactId>nifi-standard-services-api-nar</artifactId>
            <version>1.9.2</version>
            <type>nar</type>
</dependency>

NAR loading failed in NiFi with ClassNotFoundException. NiFi was not able
to find Hadoop related dependencies.

So to solve this problem I found 2 options:
1. either remove *provided* scope from all Hadoop dependencies and let
those jars get bundled in NAR but then I have to add all the dependencies
which can make NAR bulky
2. Mention "nifi-hadoop-nar" in NAR's pom.xml but with this approach NiFi
complains about NiFi related ClassNotFound and I have to add all the NiFI
dependencies in NAR

Now I understand that we can give only one NAR as a dependency in our
custom processor.

So I would like to know

1) what is the best way to specifiy multiple NiFi bundles in the processor?

2) Is it a good idea to put all the custom processors in single Maven
project or should I create separate Maven project for each custom processor?

Regards,
Chandra

Re: How to create custom processor which needs 2 or more NiFi bundles?

Posted by Chandrashekhar Kotekar <sh...@gmail.com>.

Thanks Bryan, will try it out. 

Sent from my iPhone

> On 25 Sep 2019, at 7:15 pm, Bryan Bende <bb...@gmail.com> wrote:
> 
> 1) You can only have one parent NAR, and it creates a hierarchy, so if
> you need hadoop libraries you would want your parent NAR to be
> nifi-hadoop-libraries-nar, which already has a parent of
> nifi-standard-services-api-nar, so then you have both in your parent
> hierarchy.
> 
> 2) This is really up to you. If you have just a few processors then
> probably put them in a single NAR, if you have lots then maybe break
> them up by functionality.
> 
> On Wed, Sep 25, 2019 at 2:09 PM Chandrashekhar Kotekar
> <sh...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> I have developed some custom processors within single Maven project (lets call this project custom-processors). These processors need Hadoop dependencies (nifi-hadoop-nar), record-serde related dependencies (nifi-record-serialization-service-api) and Kerberos dependencies apart from usual NiFI dependencies.
>> 
>> In pom.xml file of NAR, when I mentioned "nifi-standard-services-api-nar" as shown below
>> 
>> <dependency>
>>            <groupId>org.apache.nifi</groupId>
>>            <artifactId>nifi-standard-services-api-nar</artifactId>
>>            <version>1.9.2</version>
>>            <type>nar</type>
>> </dependency>
>> 
>> NAR loading failed in NiFi with ClassNotFoundException. NiFi was not able to find Hadoop related dependencies.
>> 
>> So to solve this problem I found 2 options:
>> 1. either remove *provided* scope from all Hadoop dependencies and let those jars get bundled in NAR but then I have to add all the dependencies which can make NAR bulky
>> 2. Mention "nifi-hadoop-nar" in NAR's pom.xml but with this approach NiFi complains about NiFi related ClassNotFound and I have to add all the NiFI dependencies in NAR
>> 
>> Now I understand that we can give only one NAR as a dependency in our custom processor.
>> 
>> So I would like to know
>> 
>> 1) what is the best way to specifiy multiple NiFi bundles in the processor?
>> 
>> 2) Is it a good idea to put all the custom processors in single Maven project or should I create separate Maven project for each custom processor?
>> 
>> Regards,
>> Chandra

Re: How to create custom processor which needs 2 or more NiFi bundles?

Posted by Bryan Bende <bb...@gmail.com>.

1) You can only have one parent NAR, and it creates a hierarchy, so if
you need hadoop libraries you would want your parent NAR to be
nifi-hadoop-libraries-nar, which already has a parent of
nifi-standard-services-api-nar, so then you have both in your parent
hierarchy.

2) This is really up to you. If you have just a few processors then
probably put them in a single NAR, if you have lots then maybe break
them up by functionality.

On Wed, Sep 25, 2019 at 2:09 PM Chandrashekhar Kotekar
<sh...@gmail.com> wrote:
>
> Hi,
>
> I have developed some custom processors within single Maven project (lets call this project custom-processors). These processors need Hadoop dependencies (nifi-hadoop-nar), record-serde related dependencies (nifi-record-serialization-service-api) and Kerberos dependencies apart from usual NiFI dependencies.
>
> In pom.xml file of NAR, when I mentioned "nifi-standard-services-api-nar" as shown below
>
> <dependency>
>             <groupId>org.apache.nifi</groupId>
>             <artifactId>nifi-standard-services-api-nar</artifactId>
>             <version>1.9.2</version>
>             <type>nar</type>
> </dependency>
>
> NAR loading failed in NiFi with ClassNotFoundException. NiFi was not able to find Hadoop related dependencies.
>
> So to solve this problem I found 2 options:
> 1. either remove *provided* scope from all Hadoop dependencies and let those jars get bundled in NAR but then I have to add all the dependencies which can make NAR bulky
> 2. Mention "nifi-hadoop-nar" in NAR's pom.xml but with this approach NiFi complains about NiFi related ClassNotFound and I have to add all the NiFI dependencies in NAR
>
> Now I understand that we can give only one NAR as a dependency in our custom processor.
>
> So I would like to know
>
> 1) what is the best way to specifiy multiple NiFi bundles in the processor?
>
> 2) Is it a good idea to put all the custom processors in single Maven project or should I create separate Maven project for each custom processor?
>
> Regards,
> Chandra