You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Keith Lim <Ke...@ds-iq.com> on 2016/04/21 00:42:34 UTC
Where can I find documentation of implementing custom processor?
Is there any good example of implementing a custom Processor?
Thanks,
Keith
Re: Where can I find documentation of implementing custom processor?
Posted by Joe Witt <jo...@gmail.com>.
Hello Keith,
We do have a solid developer guide here [1] that goes into the key
concepts and processor development a bit. The next best thing is to
use the supplied maven archetype to lay down the fundamental pieces.
You can read a bit more about that here [2]. Finally something that
is also quite helpful is looking at the standard processors to see if
anything follows the pattern/problem you had in mind and which can
serve as a good basis [3]
That said it is pretty wild that we do not have a good canonical
'Hello World Processor' tutorial created already. So created a JIRA
to do that.
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
[2] https://richardstechnotes.wordpress.com/2015/12/06/setting-up-for-custom-nifi-processor-development/
[3] https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
[4] https://issues.apache.org/jira/browse/NIFI-1796
Thanks
Joe
On Wed, Apr 20, 2016 at 6:42 PM, Keith Lim <Ke...@ds-iq.com> wrote:
> Is there any good example of implementing a custom Processor?
>
>
> Thanks,
> Keith
Re: Where can I find documentation of implementing custom processor?
Posted by Andy LoPresto <al...@apache.org>.
Ashraf,
Are you looking for a step-by-step tutorial? The Developer Guide [1] does discuss Controller Services to some extent. You can also take a look at an example like SSLContextService and the Maven directory structure here [2].
[1] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#controller-services <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#controller-services>
[2] https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle <https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-services/nifi-ssl-context-bundle>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 21, 2016, at 12:47 PM, Ashraf Hasson <AH...@WindMobile.ca> wrote:
>
> Hi Andy,
>
> Can you complement your list with guides or how-tos on building a custom controller service and tie that into a processor that leverages the service. The only blog I found on this was Phillip Grenier's article (http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/ <http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/>) but it doesn't detail how the bundle directory structure and dependencies were built with maven.
> Developing a Custom Apache Nifi Controller Service - NiFi ... <http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/>
> www.nifi.rocks <http://www.nifi.rocks/>
> How to develop a custom Apache Nifi controller service. The example in this post will utilize Java Properties files.
>
> Thanks,
>
>
> From: Andy LoPresto <al...@apache.org>
> Sent: April 20, 2016 6:59 PM
> To: dev@nifi.apache.org
> Subject: Re: Where can I find documentation of implementing custom processor?
>
> Hi Keith,
>
> I’ve moved this conversation to the dev list as that is where coding is more frequently discussed as opposed to the users list. If you’re not already subscribed, you can do so following the instructions here [1] to ensure you see any replies.
>
> The best approach is probably to examine the standard processors in [2]. Here you will be able to see how other developers have extended the AbstractProcessor class with domain-specific implementations. One example is EncryptContent [3] — you can see that while a lot of the encryption logic is delegated to the various *Encryptor classes, the code includes processor-specific tasks like configuring PropertyDescriptors (how users enter properties for the processor) and validating those values, configuring relationships like success and failure, and using the session and flowfile with StreamCallbacks and reporting provenance events.
>
> For extensive documentation on writing a processor, the Developer Guide [4] is the most comprehensive resource and describes the processor API, documenting your processor, common patterns, and error handling & testing. If you have specific questions during the development process, please feel free to ask on the dev list. The community has a broad and deep knowledge and is excited to help new developers with general NiFi development or very domain-specific questions as well.
>
> [1] https://nifi.apache.org/mailing_lists.html <https://nifi.apache.org/mailing_lists.html>
> [2] https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard <https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard>
> [3] https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java <https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java>
> [4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html>
>
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Apr 20, 2016, at 3:42 PM, Keith Lim <Keith.Lim@ds-iq.com <ma...@ds-iq.com>> wrote:
>>
>> Is there any good example of implementing a custom Processor?
>>
>> Thanks,
>> Keith
Re: Where can I find documentation of implementing custom processor?
Posted by Ashraf Hasson <AH...@WindMobile.ca>.
Hi Andy,
Can you complement your list with guides or how-tos on building a custom controller service and tie that into a processor that leverages the service. The only blog I found on this was Phillip Grenier's article (http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/) but it doesn't detail how the bundle directory structure and dependencies were built with maven.
Developing a Custom Apache Nifi Controller Service - NiFi ...<http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/>
www.nifi.rocks
How to develop a custom Apache Nifi controller service. The example in this post will utilize Java Properties files.
Thanks,
________________________________
From: Andy LoPresto <al...@apache.org>
Sent: April 20, 2016 6:59 PM
To: dev@nifi.apache.org
Subject: Re: Where can I find documentation of implementing custom processor?
Hi Keith,
I've moved this conversation to the dev list as that is where coding is more frequently discussed as opposed to the users list. If you're not already subscribed, you can do so following the instructions here [1] to ensure you see any replies.
The best approach is probably to examine the standard processors in [2]. Here you will be able to see how other developers have extended the AbstractProcessor class with domain-specific implementations. One example is EncryptContent [3] - you can see that while a lot of the encryption logic is delegated to the various *Encryptor classes, the code includes processor-specific tasks like configuring PropertyDescriptors (how users enter properties for the processor) and validating those values, configuring relationships like success and failure, and using the session and flowfile with StreamCallbacks and reporting provenance events.
For extensive documentation on writing a processor, the Developer Guide [4] is the most comprehensive resource and describes the processor API, documenting your processor, common patterns, and error handling & testing. If you have specific questions during the development process, please feel free to ask on the dev list. The community has a broad and deep knowledge and is excited to help new developers with general NiFi development or very domain-specific questions as well.
[1] https://nifi.apache.org/mailing_lists.html
[2] https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
[3] https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java
[4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Apr 20, 2016, at 3:42 PM, Keith Lim <Ke...@ds-iq.com>> wrote:
Is there any good example of implementing a custom Processor?
Thanks,
Keith
Re: Where can I find documentation of implementing custom processor?
Posted by Ashraf Hasson <AH...@WindMobile.ca>.
Hi Andy,
Can you complement your list with guides or how-tos on building a custom controller service and tie that into a processor that leverages the service. The only blog I found on this was Phillip Grenier's article (http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/) but it doesn't detail how the bundle directory structure and dependencies were built with maven.
Developing a Custom Apache Nifi Controller Service - NiFi ...<http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/>
www.nifi.rocks
How to develop a custom Apache Nifi controller service. The example in this post will utilize Java Properties files.
Thanks,
________________________________
From: Andy LoPresto <al...@apache.org>
Sent: April 20, 2016 6:59 PM
To: dev@nifi.apache.org
Subject: Re: Where can I find documentation of implementing custom processor?
Hi Keith,
I've moved this conversation to the dev list as that is where coding is more frequently discussed as opposed to the users list. If you're not already subscribed, you can do so following the instructions here [1] to ensure you see any replies.
The best approach is probably to examine the standard processors in [2]. Here you will be able to see how other developers have extended the AbstractProcessor class with domain-specific implementations. One example is EncryptContent [3] - you can see that while a lot of the encryption logic is delegated to the various *Encryptor classes, the code includes processor-specific tasks like configuring PropertyDescriptors (how users enter properties for the processor) and validating those values, configuring relationships like success and failure, and using the session and flowfile with StreamCallbacks and reporting provenance events.
For extensive documentation on writing a processor, the Developer Guide [4] is the most comprehensive resource and describes the processor API, documenting your processor, common patterns, and error handling & testing. If you have specific questions during the development process, please feel free to ask on the dev list. The community has a broad and deep knowledge and is excited to help new developers with general NiFi development or very domain-specific questions as well.
[1] https://nifi.apache.org/mailing_lists.html
[2] https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
[3] https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java
[4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
Andy LoPresto
alopresto@apache.org<ma...@apache.org>
alopresto.apache@gmail.com<ma...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
On Apr 20, 2016, at 3:42 PM, Keith Lim <Ke...@ds-iq.com>> wrote:
Is there any good example of implementing a custom Processor?
Thanks,
Keith
Re: Where can I find documentation of implementing custom processor?
Posted by Andy LoPresto <al...@apache.org>.
Hi Keith,
I’ve moved this conversation to the dev list as that is where coding is more frequently discussed as opposed to the users list. If you’re not already subscribed, you can do so following the instructions here [1] to ensure you see any replies.
The best approach is probably to examine the standard processors in [2]. Here you will be able to see how other developers have extended the AbstractProcessor class with domain-specific implementations. One example is EncryptContent [3] — you can see that while a lot of the encryption logic is delegated to the various *Encryptor classes, the code includes processor-specific tasks like configuring PropertyDescriptors (how users enter properties for the processor) and validating those values, configuring relationships like success and failure, and using the session and flowfile with StreamCallbacks and reporting provenance events.
For extensive documentation on writing a processor, the Developer Guide [4] is the most comprehensive resource and describes the processor API, documenting your processor, common patterns, and error handling & testing. If you have specific questions during the development process, please feel free to ask on the dev list. The community has a broad and deep knowledge and is excited to help new developers with general NiFi development or very domain-specific questions as well.
[1] https://nifi.apache.org/mailing_lists.html <https://nifi.apache.org/mailing_lists.html>
[2] https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard <https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard>
[3] https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java <https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java>
[4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 20, 2016, at 3:42 PM, Keith Lim <Ke...@ds-iq.com> wrote:
>
> Is there any good example of implementing a custom Processor?
>
> Thanks,
> Keith
Re: Where can I find documentation of implementing custom processor?
Posted by Andy LoPresto <al...@apache.org>.
Hi Keith,
I’ve moved this conversation to the dev list as that is where coding is more frequently discussed as opposed to the users list. If you’re not already subscribed, you can do so following the instructions here [1] to ensure you see any replies.
The best approach is probably to examine the standard processors in [2]. Here you will be able to see how other developers have extended the AbstractProcessor class with domain-specific implementations. One example is EncryptContent [3] — you can see that while a lot of the encryption logic is delegated to the various *Encryptor classes, the code includes processor-specific tasks like configuring PropertyDescriptors (how users enter properties for the processor) and validating those values, configuring relationships like success and failure, and using the session and flowfile with StreamCallbacks and reporting provenance events.
For extensive documentation on writing a processor, the Developer Guide [4] is the most comprehensive resource and describes the processor API, documenting your processor, common patterns, and error handling & testing. If you have specific questions during the development process, please feel free to ask on the dev list. The community has a broad and deep knowledge and is excited to help new developers with general NiFi development or very domain-specific questions as well.
[1] https://nifi.apache.org/mailing_lists.html <https://nifi.apache.org/mailing_lists.html>
[2] https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard <https://github.com/apache/nifi/tree/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard>
[3] https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java <https://github.com/apache/nifi/blob/e4b7e47836edf47042973e604005058c28eed23b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java>
[4] https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html <https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Apr 20, 2016, at 3:42 PM, Keith Lim <Ke...@ds-iq.com> wrote:
>
> Is there any good example of implementing a custom Processor?
>
> Thanks,
> Keith