You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andre <an...@fucs.org> on 2015/11/30 03:01:42 UTC
Should we have an easier way to implement ListenBlah processors?
Hi,
I am trying to give it a go on NIFI-856 (I'm not a coder either but
decided to take the challenge).
As I try to get my head around it I've noticed that coding "Listeners"
for NiFi is sort of a nightmare. (please take no offense, it is just a
sincere unskilled code statement).
I've looked at ListenHTTP, ListenUDP and ListenSyslog for inspiration,
I got the impression they all seem to have a need to tackle the
transmission of messages between the Listener "daemon" and the
onTrigger method. (e.g. syslogEvents within ListenSyslog).
Is this understanding correct?
If so, Shouldn't the framework contain something that makes it easier
to write this sort of processor instead of having to recreate them?
I say that because nearly every Listen processors will:
1. Bind to a socket;
2. Receive messages;
3. Read messages;
4. Add them flowfile;
5. Acknowledge receipt;
Where 4 and 5 are usually applicable to some protocols (like Logstash,
RELP, RLTP, etc).
Keen to hear your thoughts
Andre
Re: Should we have an easier way to implement ListenBlah processors?
Posted by Joe Witt <jo...@gmail.com>.
Hello Andre,
Wanted to follow up with you and see how things are developing here
and offer any assistance.
Thanks
Joe
On Mon, Nov 30, 2015 at 11:09 AM, Andre <an...@fucs.org> wrote:
> Hi Joe,
>
> Damn! I should have suspected it... :-(
>
> Anyhow, the code is starting to take shape here :
>
> https://github.com/trixpan/nifi-lumberjack-bundle
> (I will send a PR when I get closer to complete)
>
> I believe I've cleaned most of the netty dependencies, except for the
> issues I faced trying to get TLS to work (netty is so much easier for
> this sort of things! ), so meanwhile I am simply using SOCAT to play
> the TLS termination point until I can get my head around.
>
> The code heavily follows the ListenSyslog pattern (don't pay attention
> to the mess, I will clean it later) but still doesn't inject
> flowfiles. So far I have been able to receive the initial window size
> packets, however, no traffic is processed yet (I suspect it is a
> problem with the framedecoder switch statement but will have to
> confirm tomorrow morning.
>
> Cheers
>
>
> On Mon, Nov 30, 2015 at 1:45 PM, Joe Witt <jo...@gmail.com> wrote:
>> Hello
>>
>> I think it is fair to say that building listeners, where NiFi is
>> acting as a recipient of data being pushed to it, is among the harder
>> extension points to build. I also think that the outline of the
>> general pattern is fair at a high level but that a good API to make
>> that more repeatable generically is pretty hard for the reasons Tony
>> mentions.
>>
>> That said there are things we can do and should do to just keep making
>> such things a little bit easier over time. We're happy to help you
>> walk through NIFI-856 and if we find things to chip away at
>> generically then let's identify them and work them off.
>>
>> Thanks
>> Joe
>>
>> On Sun, Nov 29, 2015 at 9:37 PM, Tony Kurc <tr...@gmail.com> wrote:
>>> Many of these use blockingqueues to pass between the "listener" and the
>>> "workers". I do think some library support for this would be a good idea,
>>> to help provide thread safe publication and make exception handling a
>>> little easier. I think providing any of the networking support may be
>>> tricky because of the diversity of client apis and libraries. But a nice
>>> interprocessor pub/sub api seems like a good move. I noticed Joe Witt
>>> mentioned adding a little delay on poll in his listensyslog, I mentioned
>>> how coverity and findbugs complain about ignoring the return code of
>>> offer... Can you put a ticket in for this? I wouldn't mind working it.
>>> On Nov 29, 2015 9:01 PM, "Andre" <an...@fucs.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to give it a go on NIFI-856 (I'm not a coder either but
>>>> decided to take the challenge).
>>>>
>>>> As I try to get my head around it I've noticed that coding "Listeners"
>>>> for NiFi is sort of a nightmare. (please take no offense, it is just a
>>>> sincere unskilled code statement).
>>>>
>>>> I've looked at ListenHTTP, ListenUDP and ListenSyslog for inspiration,
>>>> I got the impression they all seem to have a need to tackle the
>>>> transmission of messages between the Listener "daemon" and the
>>>> onTrigger method. (e.g. syslogEvents within ListenSyslog).
>>>>
>>>> Is this understanding correct?
>>>>
>>>> If so, Shouldn't the framework contain something that makes it easier
>>>> to write this sort of processor instead of having to recreate them?
>>>>
>>>> I say that because nearly every Listen processors will:
>>>>
>>>> 1. Bind to a socket;
>>>> 2. Receive messages;
>>>> 3. Read messages;
>>>> 4. Add them flowfile;
>>>> 5. Acknowledge receipt;
>>>>
>>>> Where 4 and 5 are usually applicable to some protocols (like Logstash,
>>>> RELP, RLTP, etc).
>>>>
>>>> Keen to hear your thoughts
>>>>
>>>> Andre
>>>>
Re: Should we have an easier way to implement ListenBlah processors?
Posted by Andre <an...@fucs.org>.
Hi Joe,
Damn! I should have suspected it... :-(
Anyhow, the code is starting to take shape here :
https://github.com/trixpan/nifi-lumberjack-bundle
(I will send a PR when I get closer to complete)
I believe I've cleaned most of the netty dependencies, except for the
issues I faced trying to get TLS to work (netty is so much easier for
this sort of things! ), so meanwhile I am simply using SOCAT to play
the TLS termination point until I can get my head around.
The code heavily follows the ListenSyslog pattern (don't pay attention
to the mess, I will clean it later) but still doesn't inject
flowfiles. So far I have been able to receive the initial window size
packets, however, no traffic is processed yet (I suspect it is a
problem with the framedecoder switch statement but will have to
confirm tomorrow morning.
Cheers
On Mon, Nov 30, 2015 at 1:45 PM, Joe Witt <jo...@gmail.com> wrote:
> Hello
>
> I think it is fair to say that building listeners, where NiFi is
> acting as a recipient of data being pushed to it, is among the harder
> extension points to build. I also think that the outline of the
> general pattern is fair at a high level but that a good API to make
> that more repeatable generically is pretty hard for the reasons Tony
> mentions.
>
> That said there are things we can do and should do to just keep making
> such things a little bit easier over time. We're happy to help you
> walk through NIFI-856 and if we find things to chip away at
> generically then let's identify them and work them off.
>
> Thanks
> Joe
>
> On Sun, Nov 29, 2015 at 9:37 PM, Tony Kurc <tr...@gmail.com> wrote:
>> Many of these use blockingqueues to pass between the "listener" and the
>> "workers". I do think some library support for this would be a good idea,
>> to help provide thread safe publication and make exception handling a
>> little easier. I think providing any of the networking support may be
>> tricky because of the diversity of client apis and libraries. But a nice
>> interprocessor pub/sub api seems like a good move. I noticed Joe Witt
>> mentioned adding a little delay on poll in his listensyslog, I mentioned
>> how coverity and findbugs complain about ignoring the return code of
>> offer... Can you put a ticket in for this? I wouldn't mind working it.
>> On Nov 29, 2015 9:01 PM, "Andre" <an...@fucs.org> wrote:
>>
>>> Hi,
>>>
>>> I am trying to give it a go on NIFI-856 (I'm not a coder either but
>>> decided to take the challenge).
>>>
>>> As I try to get my head around it I've noticed that coding "Listeners"
>>> for NiFi is sort of a nightmare. (please take no offense, it is just a
>>> sincere unskilled code statement).
>>>
>>> I've looked at ListenHTTP, ListenUDP and ListenSyslog for inspiration,
>>> I got the impression they all seem to have a need to tackle the
>>> transmission of messages between the Listener "daemon" and the
>>> onTrigger method. (e.g. syslogEvents within ListenSyslog).
>>>
>>> Is this understanding correct?
>>>
>>> If so, Shouldn't the framework contain something that makes it easier
>>> to write this sort of processor instead of having to recreate them?
>>>
>>> I say that because nearly every Listen processors will:
>>>
>>> 1. Bind to a socket;
>>> 2. Receive messages;
>>> 3. Read messages;
>>> 4. Add them flowfile;
>>> 5. Acknowledge receipt;
>>>
>>> Where 4 and 5 are usually applicable to some protocols (like Logstash,
>>> RELP, RLTP, etc).
>>>
>>> Keen to hear your thoughts
>>>
>>> Andre
>>>
Re: Should we have an easier way to implement ListenBlah processors?
Posted by Joe Witt <jo...@gmail.com>.
Hello
I think it is fair to say that building listeners, where NiFi is
acting as a recipient of data being pushed to it, is among the harder
extension points to build. I also think that the outline of the
general pattern is fair at a high level but that a good API to make
that more repeatable generically is pretty hard for the reasons Tony
mentions.
That said there are things we can do and should do to just keep making
such things a little bit easier over time. We're happy to help you
walk through NIFI-856 and if we find things to chip away at
generically then let's identify them and work them off.
Thanks
Joe
On Sun, Nov 29, 2015 at 9:37 PM, Tony Kurc <tr...@gmail.com> wrote:
> Many of these use blockingqueues to pass between the "listener" and the
> "workers". I do think some library support for this would be a good idea,
> to help provide thread safe publication and make exception handling a
> little easier. I think providing any of the networking support may be
> tricky because of the diversity of client apis and libraries. But a nice
> interprocessor pub/sub api seems like a good move. I noticed Joe Witt
> mentioned adding a little delay on poll in his listensyslog, I mentioned
> how coverity and findbugs complain about ignoring the return code of
> offer... Can you put a ticket in for this? I wouldn't mind working it.
> On Nov 29, 2015 9:01 PM, "Andre" <an...@fucs.org> wrote:
>
>> Hi,
>>
>> I am trying to give it a go on NIFI-856 (I'm not a coder either but
>> decided to take the challenge).
>>
>> As I try to get my head around it I've noticed that coding "Listeners"
>> for NiFi is sort of a nightmare. (please take no offense, it is just a
>> sincere unskilled code statement).
>>
>> I've looked at ListenHTTP, ListenUDP and ListenSyslog for inspiration,
>> I got the impression they all seem to have a need to tackle the
>> transmission of messages between the Listener "daemon" and the
>> onTrigger method. (e.g. syslogEvents within ListenSyslog).
>>
>> Is this understanding correct?
>>
>> If so, Shouldn't the framework contain something that makes it easier
>> to write this sort of processor instead of having to recreate them?
>>
>> I say that because nearly every Listen processors will:
>>
>> 1. Bind to a socket;
>> 2. Receive messages;
>> 3. Read messages;
>> 4. Add them flowfile;
>> 5. Acknowledge receipt;
>>
>> Where 4 and 5 are usually applicable to some protocols (like Logstash,
>> RELP, RLTP, etc).
>>
>> Keen to hear your thoughts
>>
>> Andre
>>
Re: Should we have an easier way to implement ListenBlah processors?
Posted by Tony Kurc <tr...@gmail.com>.
Many of these use blockingqueues to pass between the "listener" and the
"workers". I do think some library support for this would be a good idea,
to help provide thread safe publication and make exception handling a
little easier. I think providing any of the networking support may be
tricky because of the diversity of client apis and libraries. But a nice
interprocessor pub/sub api seems like a good move. I noticed Joe Witt
mentioned adding a little delay on poll in his listensyslog, I mentioned
how coverity and findbugs complain about ignoring the return code of
offer... Can you put a ticket in for this? I wouldn't mind working it.
On Nov 29, 2015 9:01 PM, "Andre" <an...@fucs.org> wrote:
> Hi,
>
> I am trying to give it a go on NIFI-856 (I'm not a coder either but
> decided to take the challenge).
>
> As I try to get my head around it I've noticed that coding "Listeners"
> for NiFi is sort of a nightmare. (please take no offense, it is just a
> sincere unskilled code statement).
>
> I've looked at ListenHTTP, ListenUDP and ListenSyslog for inspiration,
> I got the impression they all seem to have a need to tackle the
> transmission of messages between the Listener "daemon" and the
> onTrigger method. (e.g. syslogEvents within ListenSyslog).
>
> Is this understanding correct?
>
> If so, Shouldn't the framework contain something that makes it easier
> to write this sort of processor instead of having to recreate them?
>
> I say that because nearly every Listen processors will:
>
> 1. Bind to a socket;
> 2. Receive messages;
> 3. Read messages;
> 4. Add them flowfile;
> 5. Acknowledge receipt;
>
> Where 4 and 5 are usually applicable to some protocols (like Logstash,
> RELP, RLTP, etc).
>
> Keen to hear your thoughts
>
> Andre
>